State reconnecting tls error tls

I’m having issues with my OpenVPN site configurations after upgrading to 2.5.0 (reproduced on two pfsense boxes). I previously have not had any issues while upgrading with previous pfsense versions. It seems that certificate verification may be broken or working differently in 2.5.0 — I can connect successfully only after disabling certificate verification.

For my setup, I have a self-signed root CA, intermediate CA (signed by the root CA), and server/user certificates (signed by the intermediate CA). The root CA, intermediate CA, and server/user certificates are all imported into pfsense. My certificate depth verification is set to Two (Client+Intermediate+Server). The peer certificate authority is set to the intermediate CA. I get the following logs when attempting to connect from a client (as well as a pfsense to pfsense site-to-site setup):

Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY WARNING: depth=0, unable to get certificate CRL: <user cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY WARNING: depth=1, unable to get certificate CRL: <intermediate cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY WARNING: depth=2, unable to get certificate CRL: <root cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY SCRIPT OK: depth=2, <root cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY OK: depth=2, <root cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY SCRIPT ERROR: depth=1, <intermediate cert>
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 SSL alert (write): fatal: unknown CA
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 TLS_ERROR: BIO read tls_read_plaintext error
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 TLS Error: TLS object -> incoming plaintext read error
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 TLS Error: TLS handshake failed

It seems like there is a verification of the intermediate CA at depth=1 which fails with unknown CA. I’m not sure what is normally done in previous versions of pfsense.

@joshh
I have observed same failure to connect after updating to pfsense 21.02 and OpenVPN 2.5.0. I also use a similar setup where I have a self generated root CA, intermediate CA with server/client certs signed by the intermediate CA.

I don’t have a copy of the previous openvpn server config generated by pfsense, but it appears that pfsense enabled an additional feature tls-verify that wasn’t used in previous version of pfsense. After upgrading pfsense, the «Certificate Depth» option was set to «Two (Client + Intermediate + Server)».

This results in following config line in the generated config file /var/etc/openvpn/server1/config.ovpn:

tls-verify «/usr/local/sbin/ovpn_auth_verify tls ‘myhostname.mydomain.com’ 2»

One would have thought this would work since both client and server are signed by the same intermediate cert. But as with your failure, the script returns a non zero value causing openvpn to fail the connection. The script «/usr/local/sbin/ovpn_auth_verify» appears to be maintained by pfsense.

I’ve not yet spent anytime troubleshooting the script «/usr/local/sbin/ovpn_auth_verify» to determine why it’s failing with my self generated root CA and intermediate cert. As a workaround, disabling «Certificate Depth» by setting to «Do Not Check» allows clients to connect until able to resolve why the tls-verify script is failing.

I think I may have found cause of failure. The tls-verify cmd calls «/usr/local/sbin/ovpn_auth_verify» which in turn calls following «/usr/local/sbin/fcgicli»

The fifth arg ($5) to ovpn_auth_verify looks to be the subject of the root CA (passed by openvpn). I found that if the CN of the this subject contains a space (the last element of subject), the call to fcgicli fails with error message «Something wrong happened while reading request». I suspect that fcgicli fails the parsing of key value pairs if the last element of the subject contains a space. Adding some escapes/quotes to the command in «/usr/local/sbin/ovpn_auth_verify» seems to fix. If I set cert depth to 2 (or more) with fix applied, the RESULT is «OK» and passes. If I set to depth of 1, it fails as expected with «FAILED».

> diff ovpn_auth_verify ovpn_auth_verify.fixed 
27c27
<               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
---
>               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=\"$5\"&serial=$serial&config=$config")

Searching bug reports, seems that problem may be more of string length rather than whitespace in CN (my testing also confirms). The length of my subject name 137 chars which looks to be a little larger than length that causes problem per bug report.

See bug report #10560 and #4521.

@gribnut ‘s fix didn’t work for me, however …

For some reason, OpenVPN confuses things by having TWO clients: «OpenVPN Connect v3», (The one you download from openvpn.net), «OpenVPN GUI v11», which is what PFSense uses when you download the full client installer from «Client Export».

Since some clients have more than one VPN profile they need to connect to, and not everyone is using PFSense with their fancy bundled deploy, I use the OpenVPN Connect client. It’s got a nicer UI that’s easier to use for the end-users plus has a nice traffic graph. It’s also the one readily downloadable from openvpn.net.

PFSense seems to have broken the OpenVPN Connect client. If you use the crappy client PFSense bundles with Client Export it works for me.

@gribnut

Same problem here, same fix as you suggested.
CA CN is «Home CA» (note the space) and client is an old OpenVPN v2.3.4 (with the associated network-manager-openvpn plugin v0.9.10) on a Debian Jessie.
pfSense v2.4.5p1 has no issue, upgraded to pfSense v2.5.0 it is impossible to connect with the «client certificate verification failed» error already reported in the starting post.
Applied your fix (\"$5\" in the verification script) and everything works again as expected for the old Debian Jessie.

Android phone with «OpenVPN for Android» v0.7.21 has no issue whatsoever, before or after the fix, thus the issue seems also related to the OpenVPN client used.
For sure the /usr/local/sbin/ovpn_auth_verify script distributed with pfSense v2.5.0 is buggy: v2.4.5p1 had no problem at all.

@gribnut This fix seems to have solved a similar problem I have been having with TLS/SSL OpenVPN connections.
Converted a stable 2.4.5p1 to 2.4.5p1 pair to 2.5.0 — 2.5.0 and lost the existing S2S OpenVPN link. I rebuilt the CA/Certs for Server and Client under 2.5.0 and got the link back.
Later I realized the Server had a secondary RoadWarrior setup that was also now failing previously stable clients. Logs pointed to failure to find CA.
Tried your fix and Voila it all came back to life.

Looks like a definite bug here, what’s interesting is my CA common name has no spaces as far as I can tell.
It’s «811pow-ovpn-rdwar-ca» although I’m not sure if there’s a leading space buried in there. I’ve used an OpenSSL command to dump the subject, but it’s not clear if the output adds a space to the entries as they are printed.

@fr3ddie
I forgot to mention that anytime you save openvpn config, it will write over any changes made to /usr/local/sbin/ovpn_auth_verify. I’ve not looked to see which file updates /usr/local/sbin/ovpn_auth_verify, so I just disabled cert depth check altogether until long arg to fcgicli is resolved.

Just guessing, but suspect that reason one of your clients might work whereas as others don’t is due to length of cert subject for client. The tls_verify will iterate through entire chain (depending up configured depth to search). The length of string for subject (the entire subject not just CN) could end up causing arg passed to fcgiclie to exceed value that works.

@gribnut

So, does disabling that depth check allow the VPN to work? Currently, I can get it to work on the same LAN, but not from outside the firewall.

@gribnut Hmmmmm, I think there’s a bit more to this one than meets the eye.
On ahunch, I backed out the code change to ovpn_auth_verify back to factory.

Roadwarrior links still good

Restarted the RoadWarror server process

Roadwarrior links still good

Rebooted the remote pfSense box

Roadwarrior links still good

So now I ‘m at the point that I can’t make it fail anymore after backing all my changes out <sigh>

One thing to note in all this — none of my OpenVPN Servers do more than a depth check of 1.

@gribnut
thank you for your information.
But there is still something that I can’t understand: how a length issue on the certificate subject can be fixed by just enclosing the subject in apexes?

What I believe is that, probably, there is more than one issue here in fcgicli: an issue linked to the subject length and maybe another issue linked to «special» characters (spaces? slashes?) included in the certificate subject that is fixed by simply enclosing the subject between apexes.

@nycspud Do you know what error messages OpenVPN was logging while it was failing?
I’d be curious what happens if you set Certificate Depth back to Level one, does OpenVPN start failing again or is it «magically» OK?

As noted below, there seems to be an internal interaction were not catching that trips the Certficate checks and causes OpenVPN to fail. My experiences so far have been odd in that I’ve seen hard fails right after an upgrade but once I managed to massage the certificate depth or the ovpn_auth_verify «fix», I could no longer replicate the issue by backing out the fix.

@divsys Yes, setting Certificate Depth = 1 or higher causes it to fail.
With Certificate Depth = 1 or higher
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 WARNING: Failed running command (—tls-verify script): external program exited with error status: 1
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 VERIFY SCRIPT ERROR: depth=1, C=US, ST=XX, L=XX, O=XX, emailAddress=XX@XX.XX, CN=ovpn
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS_ERROR: BIO read tls_read_plaintext error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS object -> incoming plaintext read error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS handshake failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 SIGUSR1[soft,tls-error] received, client-instance restarting

On the client side when Certificate Depth = 1 or higher
2021-02-23 07:11:48 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2021-02-23 07:11:48 TLS Error: TLS handshake failed
2021-02-23 07:11:48 SIGUSR1[soft,tls-error] received, process restarting
2021-02-23 07:11:48 MANAGEMENT: >STATE:1614093108,RECONNECTING,tls-error,,,,,
2021-02-23 07:11:48 Restart pause, 5 second(s)

I only have a self signed root CA, and no intermediate CA.

I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.
Feb 24 13:45:36 openvpn 31850 Restart pause, 5 second(s)
Feb 24 13:45:36 openvpn 31850 SIGUSR1[soft,tls-error] received, process restarting
Feb 24 13:45:36 openvpn 31850 TCP/UDP: Closing socket
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS handshake failed
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS object -> incoming plaintext read error
Feb 24 13:45:36 openvpn 31850 TLS_ERROR: BIO read tls_read_plaintext error
Feb 24 13:45:36 openvpn 31850 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Feb 24 13:45:36 openvpn 31850 VERIFY ERROR: depth=1, error=self signed certificate in certificate chain: C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=Private Internet Access, name=Private Internet Access, emailAddress=secure@privateinternetaccess.com, serial=11996199170155461251

PIA suggested rolling back OpenVPN from 2.5 to 2.4.
Any suggestions?
Thanks.

@wtw said in OpenVPN 2.5.0 Certificate Verification Fails:

I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.

The problem in this thread is for OpenVPN servers on pfSense, not clients. Your problem is unlikely to be related. Start a new thread for your issue if you haven’t already.

@gribnut said in OpenVPN 2.5.0 Certificate Verification Fails:

As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.

I tried the suggestion you provded «The fifth arg ($5)» above. No change.
I tried the patch #9460; ce76f299853dccb036de229f08a30013593c98fd as suggested. No change.
I started a new topic: pfSense 2.5.0-RELEASE OpenVPN Cert bug
What worked was creating a new self-signed cert using the same data. The difference in the backup XML is provided there.
Is patch #4521 different than #9460?
If the patch in only for an OpenVPN server, then this would be expected.

@jimp said in OpenVPN 2.5.0 Certificate Verification Fails:

If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for ce76f299853dccb036de229f08a30013593c98fd to apply the change. It will use php-cgi instead of fcgicli.

I had the same issue immediately after upgrading from 2.4.5p1 to 2.5.0, and this fix did not work for me either. However, this was because it fixes the wrong thing.

The commit above modifies /usr/local/sbin/ovpn_auth_verify_async, while OpenVPN actually calls /usr/local/sbin/ovpn_auth_verify. I changed this script in the same way the commit does:

• change /usr/local/sbin/fcgicli to /usr/local/bin/php-cgi
• remove the -d in front of the query string

This worked; OpenVPN now again allows incoming connections.

Here is the patch (apply with strip 0 in /usr/local/sbin) for reference:

--- ovpn_auth_verify.orig       2021-03-07 18:20:59.312509000 +0000
+++ ovpn_auth_verify    2021-03-07 18:21:42.060270000 +0000
@@ -24,14 +24,14 @@
        for check_depth in $(/usr/bin/seq ${3} -1 0)
        do
                eval serial="$tls_serial_${check_depth}"
-               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
+               RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.tls-verify.php "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
        done
 else
        # Single quoting $password breaks getting the value from the variable.
        # Base64 and urlEncode usernames and passwords
        password=$(echo -n "${password}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g')
        username=$(echo -n "${username}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g')
-       RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.auth-user.php -d "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5")
+       RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.auth-user.php "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5")
 fi

 if [ "${RESULT}" = "OK" ]; then

—
Christian

I can confirm this.

It is always the same thing: I googled and found do nothing
appropriate.

Than i dived into the things and figured out the length problem myself. Definitely, length is the problem. Reproducable on the commandline.

When googling again with fcgicli in the search, i found this thread.

Manually applied the patch and everthing works fine.

good day to all same problem i have after update pfsense to 2.5.1
this messages i recived in server

Jul 5 13:20:08 openvpn 90254 ip:43573 TLS Error: TLS handshake failed
Jul 5 13:20:08 openvpn 90254 ip:43573 TLS Error: TLS object -> incoming plaintext read error
Jul 5 13:20:08 openvpn 90254 ip:43573 TLS_ERROR: BIO read tls_read_plaintext error
Jul 5 13:20:08 openvpn 90254 ip:43573 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Jul 5 13:20:08 openvpn 90254 ip:43573 VERIFY ERROR: depth=0, error=unsupported certificate purpose: C=RO, ST=HD, L=MT, O=ITL, emailAddress=mail, CN=pfsmtsrv, OU=IT, serial=1
Jul 5 13:19:28 openvpn 90254 Initialization Sequence Completed
Jul 5 13:19:28 openvpn 90254 UDPv4 link remote: [AF_UNSPEC]
Jul 5 13:19:28 openvpn 90254 UDPv4 link local (bound): [AF_INET]ip:44442
Jul 5 13:19:28 openvpn 90254 /usr/local/sbin/ovpn-linkup ovpns4 1500 1622 ip 255.255.255.0 init
Jul 5 13:19:28 openvpn 90254 /sbin/ifconfig ovpns4 10.1.2.1 10.1.2.2 mtu 1500 netmask 255.255.255.0 up
Jul 5 13:19:28 openvpn 90254 TUN/TAP device /dev/tun4 opened
Jul 5 13:19:28 openvpn 90254 TUN/TAP device ovpns4 exists previously, keep at program end
Jul 5 13:19:28 openvpn 90254 WARNING: experimental option —capath /var/etc/openvpn/server4/ca
Jul 5 13:19:28 openvpn 90254 Initializing OpenSSL support for engine ‘devcrypto’
Jul 5 13:19:28 openvpn 90254 NOTE: the current —script-security setting may allow this configuration to call user-defined scripts
Jul 5 13:19:28 openvpn 90243 library versions: OpenSSL 1.1.1k-freebsd 25 Mar 2021, LZO 2.10
Jul 5 13:19:28 openvpn 90243 OpenVPN 2.5.1 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Apr 5 2021
Jul 5 13:19:28 openvpn 90243 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless «allow-compression yes» is also set.

This messages is in client

Jul 5 13:24:23 openvpn 34328 UDPv4 link remote: [AF_INET]ip:44442
Jul 5 13:24:23 openvpn 34328 UDPv4 link local (bound): [AF_INET]ip:0
Jul 5 13:24:23 openvpn 34328 TCP/UDP: Preserving recently used remote address: [AF_INET]ip:44442
Jul 5 13:24:23 openvpn 34328 NOTE: the current —script-security setting may allow this configuration to call user-defined scripts
Jul 5 13:24:23 openvpn 34328 WARNING: No server certificate verification method has been enabled
Jul 5 13:24:18 openvpn 34328 SIGUSR1[soft,ping-restart] received, process restarting
Jul 5 13:24:18 openvpn 34328 [pfsmtsrv] Inactivity timeout (—ping-restart), restarting
Jul 5 13:22:13 openvpn 34328 UDPv4 link remote: [AF_INET]ip:44442
Jul 5 13:22:13 openvpn 34328 UDPv4 link local (bound): [AF_INET]ip:0
Jul 5 13:22:13 openvpn 34328 TCP/UDP: Preserving recently used remote address: [AF_INET]89.121.228.158:44442
Jul 5 13:22:13 openvpn 34328 WARNING: experimental option —capath /var/etc/openvpn/client2/ca
Jul 5 13:22:13 openvpn 34328 Initializing OpenSSL support for engine ‘devcrypto’
Jul 5 13:22:13 openvpn 34328 NOTE: the current —script-security setting may allow this configuration to call user-defined scripts
Jul 5 13:22:13 openvpn 34328 WARNING: No server certificate verification method has been enabled.
Jul 5 13:22:13 openvpn 34328 WARNING: using —pull/—client and —ifconfig together is probably not what you want
Jul 5 13:22:13 openvpn 34039 library versions: OpenSSL 1.1.1k-freebsd 25 Mar 2021, LZO 2.10
Jul 5 13:22:13 openvpn 34039 OpenVPN 2.5.1 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Apr 5 2021
Jul 5 13:22:13 openvpn 34039 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless «allow-compression yes» is also set.