Ssl error certificate verify failed

SSL certificate_verify_failed errors typically occur as a result of outdated Python default certificates or invalid root certificates. We will cover how to fix this issue in 4 ways in this article.

Why certificate_verify_failed  happen?

The SSL connection will be established based on the following process.   We will get errors if any of these steps does not go well.

For this error certificate_verify_failed, it usually happens during step 2 and step 3.

• The client sends a request to the server for a secure session. The server responds by sending its X.509 digital certificate to the client.
• The client receives the server’s X.509 digital certificate.
• The client authenticates the server, using a list of known certificate authorities.
• The client generates a random symmetric key and encrypts it using server’s public key.
• The client and server now both know the symmetric key and can use the SSL encryption process to encrypt and decrypt the information contained in the client request and the server response.

When the client receives the server’s certificate, it begins chaining that certificate back to its root. It will begin by following the chain to the intermediate that has been installed, from there it continues tracing backwards until it arrives at a trusted root certificate.

If the certificate is valid and can be chained back to a trusted root, it will be trusted. If it can’t be chained back to a trusted root, the browser will issue a warning about the certificate.

Related: Check SSL Certificate Chain with OpenSSL Examples

Error info about certificate_verify_failed

We will see the following error.

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)>

What is SSL certificate

Server certificates are the most popular type of X.509 certificate. SSL/TLS certificates are issued to hostnames (machine names like ‘ABC-SERVER-02’ or domain names like google.com).

A server certificate is a file installed on a website’s origin server. It’s simply a data file containing the public key and the identity of the website owner, along with other information. Without a server certificate, a website’s traffic can’t be encrypted with TLS.

Technically, any website owner can create their own server certificate, and such certificates are called self-signed certificates. However, browsers do not consider self-signed certificates to be as trustworthy as SSL certificates issued by a certificate authority.

Related: 2 Ways to Create self signed certificate with Openssl Command

How to fix certificate_verify_failed?

If you receive the “certificate_verify_failed” error when trying to connect to a website, it means that the certificate on the website is not trusted. There are a few different ways to fix this error.

We will skip the SSL certificate check in the first three solutions.  For the fourth solution, we are going to install the latest CA certificate from certifi.

Create unverified context in SSL

import ssl
context = ssl._create_unverified_context()
urllib.request.urlopen(req,context=context)

Create unverified https context in SSL

import ssl
ssl._create_default_https_context = ssl._create_unverified_context
urllib2.urlopen(“https://google.com”).read()

Use requests module and set ssl verify to false

requests.get(url, headers=Hostreferer,verify=False)

Update SSL certificate with PIP

we can also update our SSL certificate With PIP.  All we would have to do is  to update our SSL certificate directory with the following piece of code: pip install –upgrade certifi

What this command does is update our system’s SSL certificate directory.

Reference:

Understanding SSL certificates

Check SSL Certificate Chain with OpenSSL Examples

5 ways to check SSL Certificate

We know setting up SSL certificates with Python can be confusing — that’s
why we’re here to help

When you’re dealing with Python or any programming language at all, there’s plenty of room for mistakes to be made or technical errors to occur. Among these potential errors is the Python SSL “certificate_verify_failed” error. Getting this error can be frustrating, especially if you’ve done your best to ensure that everything is done right.

When dealing with this error, it’s
important to know that it isn’t hard to solve — but it does require patience.

That being said, before you can
fix the issue, you need to understand why it occurs in the first place. Let’s
dive into the reasons this error occurs, as well as what you can do to address
it.

SSL certificate_verify_failed errors typically occur as a result of outdated Python default certificates or invalid root certificates. If you’re a website owner and you’re receiving this error, it could be because you’re not using a valid SSL certificate. Here’s where you can get one:

Since this error is usually paired
to web page scrapers in Python, let’s assume that this is a typical scenario
where the error happens:

First, imagine you’re trying to
scrape a page. You fire the scraper up, only to be met with an error page.

Don’t worry, though. This issue
can be resolved with a simple command, which we’ll get to shortly.

But what causes the error? The
issue comes from your web browser attempting to download a program that it will
not let it download because of the expired SSL certificates that came
with your version of Python. (Since that version of SSL is no longer deemed
“safe” by Python, your end users receive the warning message.)

How Can I Fix the SSL Certificate_verify_failed Error?

Some people might suggest that you
simply disable the certificate verification function. But this tactic not only
fails to resolve the issue, but also means that you’re no longer verifying the
certificate, which can lead to a variety of other issues.

To fix this this problem, you may
need to upgrade your SSL certificate directory. The most common way to do so is
to use the following PIP code.

PIP,
which stands for “Python Package Installer,” is exactly how it sounds — it’s a
package installer for Python. This command allows for easy installation of
packages — or, in this case, our updated SSL certificates. With PIP, all you
would have to do to update your SSL certificate directory is input the
following piece of code:

pip install --upgrade certifi

What
this command does is update your system’s SSL certificate directory. This allows
you to download the files that were previously being denied as a result of the
lack of an SSL certificate (which, in this case, was the page scraper).

After
executing the code, the error should be gone. 
That wasn’t so hard, was it?

Troubleshooting Guides

• How to Resolve SSL_ERROR_RX_RECORD_TOO_LONG as a Site Visitor
• How to Fix the ERR_SSL_PROTOCOL_ERROR in 8 Easy Steps (2020 Edition)

When working on your Rails app or when installing gems, you might get this Ruby SSL error:

SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

This post explains what the error is and provides some possible solutions.

This error, as you can see on the message, has something to do with SSL and certificates. First, let’s discuss why you need to use SSL. When your app connects to a 3rd-party API or when the gem command connects to rubygems.org, you need to use the HTTPS version e.g. https://api.example.com or https://rubygems.org.

If you don’t use the HTTPS version and use the non-secure HTTP version, the data you receive can be altered by anyone on the path from the user to your server. And you wouldn’t know it was altered. If you request a gem from http://rubygems.org from your laptop, and say you’re using public WiFi, the WiFi provider can give you a modified gem.

In short, you should always use HTTPS, which is secured using an SSL certificate.

Secure Sockets Layer or SSL is the security protocol that provides secure communication between two machines. In the cases above, between the machines of your app and the 3rd-party API or between your machine and the rubygems.org machine.

How does SSL work? Machine 1 initiates the connection and looks at the SSL certificate of machine 2. This certificate contains numbers that machine 1 will use to encrypt their communication. This is a simple and incomplete description of the whole process. We’ll not go into details on how the encryption or the SSL handshake works.

The encrypted communication makes sure that what you send to the server and vice versa isn’t altered. However, the initial connection is done in clear text since it comes before encrypted communication is set up. So, while you’re sure that communication is secure, how do you know you’re talking to the correct server?

SSL certificates are signed by a Certificate Authority. The signature tells you that the certificates are real. The Certificate Authorities are also signed for the same reason. This can go on until you reach a root certificate. This setup creates a chain of trust. As long as you trust the root certificate, you can trust the certificate of the website you’re communicating with.

The root certificates are installed on your machine and there’s an implied trust that the existing certificates are valid.

When machine 1 connects to machine 2, machine 1 has to verify that the certificate is real. If this verification fails, you’ll get the error certificate verify failed.

When verification fails, it can mean one of two things. Either your machine doesn’t have the correct root certificates or you’re connecting to a URL that has a problem with the certificates.

If it’s the latter, the best (if not only) option is to ask the 3rd-party site to fix their certificates. The solutions mentioned below are for the first scenario where the problem is on your machine.

Scale performance. Not price. Try Engine Yard today and enjoy our great support and huge scaling potential for 14 days.
Deploy your app for free with Engine Yard.

The Problem

The error certificate verify failed happens when a machine can’t verify the certificate of the machine it’s connecting to. What does this mean in practice?

When developing Rails applications, this can happen when

• you run gem install or bundle install. Your machine needs to connect to the gem sources like https://rubygems.org
• your app connects to a 3rd-party API like GitHub API
• you use a gem like ActiveMerchant to connect to payment sites like PayPal

Solutions

Update CA certificates

The correct solution depends on which code connects to an HTTPS URL. The first thing you can try is to update the root certificates on your machine.

If you’re using Linux, you can use your package manager to update the CA certificates.

apt-get update ca-certificates
yum update ca-certificates

On RVM on OSX, you can run

rvm osx-ssl-certs update all

If you don’t use RVM, you can extract the certificates from Apple’s Keychain yourself.

cert_file='$( openssl version -d | awk -F''' '{print $2}' )/cert.pem'
mkdir -p '${cert_file%/*}'
security find-certificate -a -p /Library/Keychains/System.keychain > '$cert_file'
security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain >> '$cert_file'

For more information, check out the SSL documentation.

Update Gems

In some cases, updating the system CA certificates doesn’t work because some gems specify their own CA certificates. ActiveMerchant provides its own cacert.pem at https://github.com/activemerchant/active_merchant/tree/master/lib/certs. If your errors come from ActiveMerchant, try updating the gem to the latest version.

rubygems.org in 2014 had to update their SSL certificate. RubyGems also provides CA certificates and a newer RubyGems version had to be manually installed to get it working again. You can read more about this issue in the Ruby Gems guides here. This is unlikely to happen again but if you’re having issues with RubyGems, check your system certificates first then the RubyGems issues.

Bad Solutions

There are other solutions to this error but are not considered best practices.

Turn off verification

You can turn off verification when using net-http. However, this isn’t recommended.

http.verify_mode = OpenSSL::SSL::VERIFY_NONE

Use http version

When you encounter the certificate verify failed error when installing gems, some suggest using http://rubygems.org as the source instead of https://rubygems.org. This is bad as you’re installing gems in clear text. Turning off verification is better than using clear text because even without verification, you’re still using encrypted communication. But make no mistake, both of these solutions aren’t recommended.

Set SSL_CERT_FILE

This isn’t a bad solution if you know what you’re doing. However, there are a lot of solutions out there that suggest downloading CA certificates to your machine and setting the SSL_CERT_FILE environment variable to its location.

The problem with this approach is you don’t know if you can trust the CA certificates you’re downloading. In some cases, the CA certificates are even downloaded in clear text. Double trouble if you ask me.

Summary

The Ruby OpenSSL error certificate verify failed means your code can’t verify that the SSL certificate of the website or API you’re connecting to is the real one. It’s important to solve this issue correctly to keep your communication secure.

If you are interested on security issues, make sure to check out this Engine Yard’s blog section

Overview of the problem

When using Python to connect to z/OSMF, you might see the following errors:

   «certificate verify failed: self signed certificate in certificate chain»
OR
   «certificate verify failed: unable to get local issuer certificate»

This might be caused either by server configuration or Python configuration. In this article, we assume you use a self-signed CA certificate in z/OSMF. We will guide you step by step to workaround the certification error. You can choose either workaround if you wish.

• Workaround 1: verify = False
• Workaround 2: verify = CAfile (Specify a certificate in the PARM)
• Workaround 3: verify = True (Update key store in Python)

self signed certificate

If your z/OSMF was configured with self signed certificate, the python3 output error is:
certificate verify failed: self signed certificate in certificate chain

• Workaround 1: verify = False

          Setting verify = False will skip SSL certificate verification.

• Workaround 2: verify = CAfile (Specify a certificate in the PARM)

          The CAfile must be set to the CA certificate Bundle, if you set it as the server certificate, you will get the above error.

• Workaround 3: Verify = True (Update key store in Python)

The default value for parameter verify is True. Python 3.6+ installer has its own default certificate store for verifying SSL connections. The system certificate store will  not be used any more. To use default certificate store, python library certifi must be installed in advance, you can use command «pip3 install certifi» to install it. Python default certificate store was in cacert.pem file, you can get it by

       >>> import certifi
       >>> certifi.where()
        ‘/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/certifi/cacert.pem’

If you get the above error, it means that your CA certificate was not included in cacert.pem, please use below command to add it:
     For Mac or Linux:
          $ cat [full path of your-cacert.pem] >> [full path of cacert.pem]
     For Windows:
          C:type [full path of your-cacert.pem] >> [full path of cacert.pem]

CA signed certificate

If your z/OSMF was configured with CA signed certificate, the python3 output error is:
certificate verify failed: unable to get local issuer certificate

• Workaround 1: verify = False

          Setting verify = False will skip SSL certificate verification.

• Workaround 2: verify = CAfile (Specify a certificate in the PARM)

          The CAfile is a CA certificate Bundle, it must be the Root CA certificate. If it is not a Root CA certificate, the above error will be showed.

• Workaround 3: Verify = True (Update key store in Python)

The default value for parameter verify is True. Python 3.6+ installer has its own default certificate store for verifying SSL connections. The system certificate store will not be used any more. To use default certificate store, python library certifi must be installed in advance, you can use command “pip3 install certifi”to install it. Python default certificate store was in cacert.pem file, you can get it by

        >>> import certifi
        >>> certifi.where()
        ‘/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/certifi/cacert.pem’

If you get the above error, it means that your Root CA certificate was not included in cacert.pem, please use below command to add it:
     For Mac or Linux:
           $ cat [full path of your-Root-cacert.pem] >> [full path of cacert.pem]
     For Windows:
           C:type [full path of your-Root-cacert.pem] >> [full path of cacert.pem]

How to export z/OSMF CA certificate

• Export CA certificate from z/OS

      1. RACDCERT EXPORT(LABEL(‘zOSMFCA’)) DSN(‘IBMUSER.CACERT.IBMUSER.CRT’) FORMAT(CERTDER) CERTAUTH

      Where:

• zOSMF is the label of the CA certificate.
• IBMUSER.CACERT.IBMUSER.CRT is the data set that will contain the client certificate. RACF command will auto-create this dataset
• CERTDER indicates that certificate is in DER format

     2. Download the CA certificate from z/OS
     3. Convert to PEM format
          openssl x509 -in zOSMFCA.crt -inform der -outform pem -out zOSMFCA.pem

Today when I using pytube python package to download the YouTube video, I got the following error message:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>

I never got this error in the past, and it seems not a problem with YouTube page version. After I check on the Internet, I find maybe the version of urllib module is different from the previous version.

Simply put, the error seems an error in verifying the SSL certificate. Maybe the target website uses a certificate signed by itself, I’m not really sure.

Sample Program With Error

Let’s take an example:

import urllib.request
r = urllib.request.urlopen('https://google.com')

Output:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>

Solution

However, if we choose not to authenticate SSL certificate, we can make urllib work.

# coding: utf-8
import urllib.request
import ssl


def main():
    ssl._create_default_https_context = ssl._create_unverified_context
    r = urllib.request.urlopen('https://google.com')
    print(r.status)
    print(r)


if __name__ == '__main__':
    main()

Output:

200
<http.client.HTTPResponse object at 0x7fc1af051ef0>

In addition to this method, if you just want a specific connection not to be verified, and the rest of the connection is to verify the SSL certificate, you can write:

# coding: utf-8
import urllib.request
import ssl


def main():
    context = ssl._create_unverified_context()
    r = urllib.request.urlopen('https://google.com', context=context)
    print(r.status)
    print(r)


if __name__ == '__main__':
    main()

Output:

200
<http.client.HTTPResponse object at 0x7fc1af051ef0>

References

• https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error
• https://moreless.medium.com/how-to-fix-python-ssl-certificate-verify-failed-97772d9dd14c

Read More

• [Python] Using package pytube to download YouTube videos
• [Python] Using «GoogleNews» package to get the Google News