Certbot renew error

Make sure your Certbot installation is up to date

As of late 2020 Certbot is no longer maintained as an apt package and you should make sure it is running through Snap on Ubuntu. This procedure is totally safe and easy to do.

Please read our full article on this topic in order to get up to speed.

Checking if your Let’s Encrypt Certificate is working

Typically Certbot runs fine in Webdock the first time you run it and problems crop up over time or when you do changes to your configuration. Often the first indication you get of a problem is when you get an email from Let’s Encrypt with a title similar to «Let’s Encrypt certificate expiration notice for domain xxx»

When you generate certificates with Webdock we automatically add a Cron job which keeps your certificate up to date. If you get this email then this means your certificate may be about to expire and you need to check if renewals are working.

In Webdock you can simply run the «Test Certbot renewal» script on your server on the Server Scripts page. 

What this script essentially does is runs the following command on your server and returns the output so you can inspect what is happening

certbot renew --dry-run

In the following sections we detail various error messages which may show up here and what to do about them.

Invalid response / The client lacks sufficient authorization

If you get a message similar to the following

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for mydomain.com
http-01 challenge for www.mydomain.com
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mydomain.com/.well-known/acme-challenge/pYpAC6kT25C0itcTNKd8hwb_0VaoPxJVIkVg5_xn-N4 [77.111.240.95]: 403
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   unauthorized
   Detail: Invalid response from
   http://mydomain.com/.well-known/acme-challenge/pYpAC6kT25C0itcTNKd8hwb_0VaoPxJVIkVg5_xn-N4
   [77.111.240.95]: 403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Try the suggested fix: Check if your DNS records are OK

First thing to check are your DNS records and whether your A and AAAA records are OK and pointing to your server. However, if your certificate has worked before and you reach your website just fine, then it is unlikely this is the culprit.

However, you should  keep an eye on whether there are any web forwards configured (some DNS providers allow this) e.g. if you forward www to non-www or vice-versa, this may trip up Certbot. In which case remove the domain you are forwarding using DNS from your certificate. This should resolve the issue.

To remove a domain in Webdock you simply run Server Identity with the domains you want your webserver to actually respond to and then run the SSL Certificates tool once more.

On Apache: Make sure you exclude Certbot from any .htaccess rewrites

If you are on Apache, check if you have a .htaccess file with rewrites in your web root. If you have such a file you should place a rule to exclude Let’s Encrypt Certbot from any rewrites by placing the exception before any RewriteCond statements. For example: 

RewriteEngine on
RewriteRule ^.well-known/acme-challenge/ - [L]
RewriteCond %{SERVER_NAME} =mydomain.demo.com [OR]
RewriteCond %{SERVER_NAME} =www.mydomain.com [OR]
RewriteCond %{SERVER_NAME} =livesite.mydomain.com [OR]
RewriteCond %{SERVER_NAME} =mydomain.com [OR]
RewriteCond %{SERVER_NAME} =mydomain.dk [OR]
RewriteCond %{SERVER_NAME} =www.mydomain.dk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

Once you’ve placed the acme-challenge RewriteRule in there, try running Certbot again.

On Apache: Try rolling back completely and nuking any Certbot config

If your DNS records and rewrites are ok and Certbot renew still fails, you should try and issue the certbot rollback command:

certbot rollback

If this gives you errors, try removing the Let’s Encrypt SSL configuration file located at (in default Webdock stacks):

/etc/apache2/sites-enabled/webdock-le-ssl.conf

Next run Certbot rollback — and if it succeeds keep running it untill it says it hasn’t touched your configuration. You should now be back to a stock configuration and you should be able to run the SSL Certificates tool once more to generate a new certificate.

If you are not using Webdock and following this guide, you should make sure you back up your configuration before doing this and make sure your config is as you want it and working for vanilla http before running Certbot.

On Nginx: Make sure you exclude Certbot from any deny rules

Similar to Apache, if you have custom configuration which excludes access to hidden directories, such as the following:

# Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
# Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
location ~ /. {
    deny all;
}

You need to add the following rule beofre this in order to allow Certbot to validate using the web root method:

# Necessary for Let's Encrypt Domain Name ownership validation
location ~ /.well-known {
  allow all;
}

Error while running apache2ctl configtest

This error is related to the last fix detailed above and can be solved in the same manner, if you encounter a message like the following:

Error while running apache2ctl configtest.
Action 'configtest' failed.
The Apache error log may have more information.

AH00526: Syntax error on line 30 of /etc/apache2/sites-enabled/000-default-le-ssl.conf:
SSLCertificateFile: file '/etc/letsencrypt/live/mydomain.com/fullchain.pem' does not exist or is empty

Simply delete the old config and try again

During the Webdock Let’s Encrypt SSL installation, a Certbot rollback command hasn’t completely cleared an old default config file. You can resolve this by removing the file /etc/apache2/sites-enabled/000-default-le-ssl.conf and run Certbot once more.

No such file or directory error from Nginx

Sometimes during a Certbot rollback operation or when Certbot tries to renew/install a certificate, Nginx cannot start the webserver as there is till old Certbot configuration hanging around in your Nginx vhost config file. You may see an error like the following:

Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: [emerg] BIO_new_file("/etc/letsencrypt/live/mydomain.com-0001/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/mydomain.com-0001/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

To solve this issue, remove any references to SSL certificates from your Nginx configuration file. The lines typically look like this:

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

Once you have removed these lines run Certbot rollback untill it says nothing was modified. If you are using Webdock you would then run the Server Identity tool and finally the SSL Certificates tool and you will now have a shiny new (and working) certificate installed on your server.

If you are not using Webdock, you would likely issue another certbot rollback command and see if it succeeds, next review your config and make sure it is as you want it and that it works for vanilla http, finally you would run Certbot again to generate your certifcate.

Timeout Errors

If you get an error similar to the following:

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   connection
   Detail: Fetching
   https://mydomain.com/.well-known/acme-challenge/m5JIZwyOioenm7XraPru2FV-8VPtCXF1YpoHmctfVDc:
   Timeout during connect (likely firewall problem)

You need to check whether your DNS records are OK and whether your webserver is listening on your server on port 80 and 443. Make sure you check whether you can ping and ping6 your domain. Typically we see this error happening is if AAAA records for a domain have not been updated, or there are old records pointing to a wrong IP hanging around — but a timeout might also indicate a networking issue (for example a firewall) as stated in the message, or that your server is actively refusing connections for some reason.

Another cause for this issue is that Nginx and Certbot are not aligned as to where your web root is located and Certbot is placing its verification files for web root authentication in the wrong place. A quick way to fix this in Webdock is to simply set the Web Root once more (the small pencil icon next to the web root in the Webdock dashboard) and then run renewal again.

The error may also look something like this:

Waiting for verification...
Cleaning up challenges
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)

In our experience, this specific message is due to Let’s Encrypt not being able to connect via. IPv6 — so check that IPv6 works if you have AAAA records for your domain, otherwise all the advice above is also applicable, i.e. check your DNS and connection issues in general. On some older Ubuntu systems (Xenial especially) a general package upgrade with apt update and apt upgrade has been seen by us as an easy resolution to some ipv6 connection issues.

Connection Refused Error

If you get a Connection Refused error when doing renewals, this may be due to old-style Certbot configuration not having written the appropriate configuration to your Nginx config in order to allow Nginx to listen on your IPv6 address. You would typically see an error like the following:

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   connection
   Detail: Fetching
   https://mydomain.com/.well-known/acme-challenge/ewpBCX7N0nzDyBZZILYP-y9sKHI4seFGac4Se7TpwfA:
   Connection refused

What this error really is saying is: Certbot could not connect to your domain on port 443. Typically what we see is that if you now check what is listening on which ports by running netstat you will get something like the following:

root@myserver:~# netstat -tapn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      441/mongod
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      772/mysqld
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      440/memcached
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      716522/nginx -g dae
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      614876/pure-ftpd (S
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      449/sshd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      716522/nginx -g dae
tcp6       0      0 :::80                   :::*                    LISTEN      716522/nginx -g dae
tcp6       0      0 :::21                   :::*                    LISTEN      614876/pure-ftpd (S
tcp6       0      0 :::22                   :::*                    LISTEN      449/sshd

Notice how nginx is listening on tcp (ipv4) port 443 but not tcp6 (ipv6) ? This is the problem as the Let’s Encrypt certificate servers prefer to connect through ipv6 if there are AAAA records set up for your domain. 

In order to solve this, check that you have the following line in your Nginx config (on Webdock servers the vhostconfig file is typically found at /etc/nginx/sites-enabled/webdock) , and if not, add it:

listen [::]:443 ssl ipv6only=on; # managed by Certbot

Next, restart Nginx:

systemctl restart nginx

Check that nginx is listening on tcp6 port 443 and try your renewal again. It should work this time — but if not, then make sure nginx is listening on the appropriate interfaces and on both ports 80 and 443 and is reachable from the outside, because if it is not you will get this error message.

Account creation on ACMEv1 is disabled

If you get an error message similar to the following

Certbot failed in schonherr Certbot message: Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
The client lacks sufficient authorization :: Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.
Please see the logfiles in /var/log/letsencrypt for more details.

This simply means that the installed version of Certbot is too old and doesn’t support newer authentication methods. Simply upgrade your system with:

apt-get update -y
apt-get upgrade -y

If it prompts you what to do about existing config files just choose the defaults which is to keep the existing configuration. Once the system is upgraded it is a good idea to reboot the system, or at the very least your webserver and php-fpm before trying again.

Please note: During the system upgrade your webserver will likely go down temporarily. Plan accordingly.

The request message was malformed

Similarily an indication your Certbot is too old may be seen from the following Certbot output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: urn:ietf:params:acme:error:malformed :: The request message was malformed :: Method not allowed. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

The solution here is, as with the error above, to upgrade your system. The same warning applies: upgrading may bring down your webserver temporarily.

Test Certbot Renewal script hangs for a long time, prompts for a new webroot in output

Sometimes Webdock users experience that the Test Certbot renewal scripts hangs for a long time and then outputs something similar to the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Non-interactive renewal: random delay of 362 seconds
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain.com
Cleaning up challenges
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: Missing command line flag or config entry for this setting:
Select the webroot for mydomain.com:
Choices: [Enter a new webroot, /var/www/html]

(You can set this with the --webroot-path flag). Skipping.
All renewal attempts failed. The following certs could not be renewed:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

This is due to Certbot having «forgotten» what the appropriate web root is for your domain. We use web root authentication in order to avoid certain other issues with Cloudflare and Cloudflare caching. In any case, the solution here is to set the web root once more on your server, after which you should run the SSL Certificates tool once more. If you still get errors, try running the Server Identity tool first.

My operating system is (include version):

MacOS Catalina

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

homebrew

I ran this command and it produced this output:

sudo certbot renew --force-renew

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/domain.dev.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')
Attempting to renew cert (domain.dev) from /etc/letsencrypt/renewal/domain.dev.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.'). Skipping.

Certbot’s behavior differed from what I expected because:

I thought this command will renew the certs.

Here is a Certbot log showing the issue (if available):

Logs are stored in /var/log/letsencrypt by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

2019-11-01 14:08:00,323:DEBUG:certbot.main:certbot version: 0.39.0
2019-11-01 14:08:00,324:DEBUG:certbot.main:Arguments: ['--force-renew']
2019-11-01 14:08:00,324:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-11-01 14:08:00,363:DEBUG:certbot.log:Root logging level set at 20
2019-11-01 14:08:00,364:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-11-01 14:08:00,402:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x10ac66a50> and installer <certbot.cli._Default object at 0x10ac66a50>
2019-11-01 14:08:00,423:DEBUG:certbot.renewal:Auto-renewal forced with --force-renewal...
2019-11-01 14:08:00,423:DEBUG:certbot.plugins.selection:Requested authenticator manual and installer None
2019-11-01 14:08:00,427:DEBUG:certbot.plugins.disco:Other error:(PluginEntryPoint#manual): An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.
Traceback (most recent call last):
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/plugins/disco.py", line 130, in prepare
    self._initialized.prepare()
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/plugins/manual.py", line 90, in prepare
    self.option_name('auth-hook')))
certbot.errors.PluginError: An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.
2019-11-01 14:08:00,428:DEBUG:certbot.plugins.selection:No candidate plugin
2019-11-01 14:08:00,428:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2019-11-01 14:08:00,428:INFO:certbot.main:Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')
2019-11-01 14:08:00,429:WARNING:certbot.renewal:Attempting to renew cert (domain.dev) from /etc/letsencrypt/renewal/domain.dev.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.'). Skipping.
2019-11-01 14:08:00,431:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/renewal.py", line 449, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/main.py", line 1202, in renew_cert
    installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/plugins/selection.py", line 235, in choose_configurator_plugins
    diagnose_configurator_problem("authenticator", req_auth, plugins)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/plugins/selection.py", line 339, in diagnose_configurator_problem
    raise errors.PluginSelectionError(msg)
certbot.errors.PluginSelectionError: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')

Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

Not using soomething like this.

[root@post ~]# certbot renew

Saving debug log to /var/log/letsencrypt/letsencrypt.log
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Processing /etc/letsencrypt/renewal/post.atr-contact.ru.conf
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Cert is due for renewal, auto-renewing…

Plugins selected: Authenticator apache, Installer apache

Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate

Performing the following challenges:

http-01 challenge for post.x

http-01 challenge for post.x

Waiting for verification…

Cleaning up challenges

Attempting to renew cert (post.x) from /etc/letsencrypt/renewal/post.x.conf produced an unexpected error: Failed authorization procedure. post.x.ru (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://x.ru/.well-known/acme-c
hallenge/wrf0oG19dQsjGBtoLqVwUjlObtAvgC6nuHHwiBFYGe4 [195.x]: «<html>rn<head><title>404 Not Found</title></head>rn<body>rn<center><h1>404 Not Found</h1></cen
ter>rn<hr><center>nginx/1.15.7</ce», post.x (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid respo
nse from http://post.x/.well-known/acme-challenge/Rzqu9QzH-HjB1SP1ZXmdpZl_bX5wMtugv-vf… [195.x]: «<html>rn<head><title>404 Not Found</title></
head>rn<body>rn<center><h1>404 Not Found</h1></center>rn<hr><center>nginx/1.15.7</ce». Skipping.
All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/post.x/fullchain.pem (failure)
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/post.x/fullchain.pem (failure)
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

— The following errors were reported by the server:

Domain: post.x

Type: unauthorized

Detail: Invalid response from

http://post.x/.well-known/acme-challenge/wrf0oG19dQsjGBtoLqVwUjlObtAvgC6nuHHw…
[195.x]: «<html>rn<head><title>404
Not
Found</title></head>rn<body>rn<center><h1>404 Not

Found</h1></center>rn<hr><center>nginx/1.15.7</ce»

Domain: post.x

Type: unauthorized

Detail: Invalid response from
http://post.x.ru/.well-known/acme-challenge/Rzqu9QzH-HjB1SP1ZXmdpZl_bX5wMtugv…

[195.x]: «<html>rn<head><title>404 Not
Found</title></head>rn<body>rn<center><h1>404 Not

Found</h1></center>rn<hr><center>nginx/1.15.7</ce»

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

Toxahh

(21.05.19 13:38:54 MSK)

SSL certificate renewals can be messy.

The PEM files won’t work, Nginx cannot find the cert, and the green light won’t turn on no matter what you do.

You are not alone.

Hundreds of Nginx owners face SSL certificate renewal issues everyday, and here at Bobcares, our Support Engineers help them fix it in a jiffy.

We provide 100% transparent end-customer support to web hosts, digital marketers and VPS hosting providers.  Their customers raise tech support requests in their help desk, and our experts resolve it for them.

A lot of these customers use Nginx as their web server, and SSL cert renewal is a common issue we troubleshoot.

Today we’ll look at the top 5 issues reported by Nginx users, and how we resolve them:

1. Let’s Encrypt renewal command breaks

Many Nginx server owners these days use Let’s Encrypt as their certificate provider.

LetsEncrypt provides a way to renew SSL certificates using various command line tools, such as CertBot.

We’ve seen these reasons for CertBot based renewal to fail:

– Wrong WebRoot specified in renew command

CertBot uses a command line options called “WebRoot” to know where it should store the new certificates.

In newly migrated websites, or those that were updated recently, we’ve seen this setting to be wrong, causing the renewal to fail.

We adjust the CertBot settings to match the current website configuration to fix this error.

– “.well-known” folder deleted or wrong permissions set

CertBot uses a folder called “.well-known” in the website document root to check the authenticity of a request.

In many websites we’ve seen this folder with wrong permissions or missing altogether (possibly due to a security cleanup or a migration error). Restoring this folder quickly resolve this issue.

– HTTP authorization set for .well-known folder

Some sites setup an additional layer of security by protecting “.well-known” folder with .htaccess authentication.

This will break the command line based renewal process. So, in these sites, we disable HTTP authentication until the renewal is complete.

Of course, there are other errors as well related to CertBot based renewal.

That is because the renewal process goes through a complex sequence of steps, which includes auto-generating a CSR, contacting the LE servers using ACME protocol, passing the auth test using .well-known folder, setting up the certificates in the right folder, and successfully restarting the Nginx servers.

If any step in this sequence fails for some reason, the renewal fails.

So, there isn’t a real one-size-fit-all-solution. Instead, our Support Engineers gather evidence from the LetsEncrypt log file (/var/log/letsencrypt/letsencrypt.log), and systematically rule out each possibility to zero in on the root cause.

2. Old SSL shows up even after an update

Sure, SSL renewals can be tricky, but the most confusing of all is when after a successful renewal, the browser still shows the old expired SSL.

You’d think Nginx is doing that out of spite. But no, there are perfectly logical reasons for that.

Here are a few common causes:

– Cert uploaded to the wrong location

We’ve seen many cases where customers use old or incompatible renewal steps to upload new certificates.

It usually happens after a recent app upgrade, site migration or server upgrade.

In these cases the new certificates get stored in a folder that’s not Nginx’s certificate path.

To resolve this error, we look at the Nginx configuration files for the cert path, and then check if the new files are uploaded to this location.

– New certificate not uploaded in gateway server

In some web configurations, the Nginx server sits behind a load-balancer or a caching server.

So, it is not enough to upload the new certificate in the Nginx server. It should be uploaded to the front end servers as well.

Many site owners miss this step, leading to renewal errors.

– Nginx reload failed

There are times when Nginx won’t completely restart. The old process still keeps serving the old certificate from memory.

If we suspect a runaway Nginx process, we force kill it, and restart the service to fix this issue.

3. Intermediate certificates are built wrong or are corrupted

A pet peeve of Nginx users is Nginx’s inability to recognize intermediate certificates (aka cert bundles, aka certificate chain).

Instead, users are required to put their site certificate and all intermediate certificates into one file in the right order, and then point it to the “ssl_certificate” configuration option.

Yeah, it sounds complicated, and it sure is complicated for someone not used to tinkering with server backend.

That is why many webmasters see the error “Site Not Trusted” even after an apparently successful upgrade.

Almost all of these errors are caused by incorrect copy-paste of certificate data into the files. Some characters would be missing, or the certs will be pasted one on top of another.

All that cause cause the cert file to be corrupted, and connecting browsers will be unable to complete a successful verification.

We fix this by programmatically merging the Site Cert and Chain Cert so that there are not manual errors.

4. Forbidden errors while running cert renew

These days one cannot be too paranoid.

That is why some server administration panels like Plesk provide a way to block all access to hidden directories (ones that start with a period [ . ] ).

It is a measure to protect sensitive information some apps store in hidden directories.

Unfortunately, automatic cert installers such as Let’s Encrypt’s CertBot use hidden folders (/path/to/docroot/.well-known) for user validation.

Such validation measures will fail if hidden folder protection is enabled.

So, when we see a “403 Forbidden” error message in the cert renewal log files, we look for such security restrictions, and remove it until the new certificate is installed.

5. Nginx crashes (does not start) after certificate renewal

The most urgent issues often relate to crashed Nginx servers after a failed upgrade.

Some auto-installers are capable of changing configuration entries, but if the installer fails for some reason, it will break the Nginx configuration file.

In such cases, we restore the Nginx configuration file from backup to quickly get the site back online.

If a backup isn’t available, we get the site online by resetting the SSL configuration section that most automated tools target.

Once the site is back online, we make the changes manually to bring the new certificate online.

Conclusion

Certificate renewals can be a hassle. Today we’ve seen the top 5 issues that are reported by Nginx users while renewing SSL, and how our Support Engineers fix them.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;