Error unable to create a successful tls connection to the ldap masters

When upgrading to 8.5x, «Unable to create a successful TLS connection to the ldap masters» comes up

Problem

When upgrading from 7.x or 8.0.x to 8.5+, the upgrade stops with the error :

Unable to create a successful TLS connection to the ldap masters.
Fix cert configuration prior to upgrading

Solution

To fix this, any one of the following can be tried :

Method 1: Using a commercial certificate

If there is a valid commercial certificate, like a wild card one, just deploying that on all the nodes will fix the error.

Method 2: Deploying a certificate using the LDAP’s CA

• Copy the CA from the ldap server to the other servers :

rsync -Pa /opt/zimbra/ssl/zimbra/ca/ root@otherserver.example.com:/opt/zimbra/ssl/zimbra/ca/

• Deploy this CA on the other servers :

/opt/zimbra/bin/zmcertmgr deployca

• Recreate and redeploy the self-signed cert on the other servers :

/opt/zimbra/bin/zmcertmgr createcrt -new -days 3650
/opt/zimbra/bin/zmcertmgr deploycrt self

The certificates now will be created using the LDAP’s newly deployed CA, so the ldap should recognize these as valid certificates now.
This would be the best way to do it if self-signed certificates are used.

Method 3: Disabling all TLS connections

Run this on the ldap and proxy servers before the upgrade

su - zimbra 
zmlocalconfig -e ssl_allow_untrusted_certs=true 
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ldap_starttls_required=false
zmlocalconfig -e ldap_common_require_tls=0
zmcontrol restart

After the upgrade is done, a new self-signed CA and certificate can be deployed across the environment, and then TLS can be enabled again by flipping the values for the above.

Submitted by: Shashank Shekhar Tewari

sub1

Posts: 5

Joined: Mon Dec 22, 2014 4:37 am

jorgedlcruz

Zimbra Alumni

Posts: 2782

Joined: Thu May 22, 2014 4:47 pm

dslauter

Posts: 6

Joined: Sat Sep 13, 2014 3:36 am

dlbewley

Advanced member

Posts: 82

Joined: Fri Sep 12, 2014 10:15 pm

sub1

Posts: 5

Joined: Mon Dec 22, 2014 4:37 am

adilm

Posts: 4

Joined: Tue Dec 23, 2014 9:28 am

t.goetten

Posts: 19

Joined: Fri Sep 12, 2014 11:22 pm

jorgedlcruz

Zimbra Alumni

Posts: 2782

Joined: Thu May 22, 2014 4:47 pm

root@zimbra-sn-u14-01:/home/oper# /opt/zimbra/bin/zmcertmgr viewdeployedcrt

t.goetten

Posts: 19

Joined: Fri Sep 12, 2014 11:22 pm

jorgedlcruz

Zimbra Alumni

Posts: 2782

Joined: Thu May 22, 2014 4:47 pm

/opt/zimbra/bin/zmcertmgr createcsr comm -new -subject "/C=GB/ST=London/L=London/O=Zimbra/OU=Zimbra IT/CN=FQDN" -subjectAltNames "FQDN,HOSTNAME"

Who is online

Users browsing this forum: Google [Bot] and 35 guests

Что скажем?

# /opt/zimbra/libexec/zmsetup.pl

Each time zmsetup.pl stops because some module is missing, just install the missing module and start zmsetup.pl again.

Finally, when the installation was finished, I saved freshly installed Zimbra in case I need it later, e.g. when I want to start from this point on. I did this with the following command:

rsync -av /opt/zimbra /opt/zimbra.orig

Note that in the following text I’ll reference from time to time zimbra.orig directory because we’ll need some files from there.

Experimenting with a new installation

Migrating existing Zimbra installation to a new server

I suggest that before you do this, you do some preparation steps:

1. Block outbound connections to port 25, i.e. prevent new instance of Zimbra to send emails until you migration is complete. And then, before going into production, check mail queue for orphaned/duplicate mails.
2. Synchronizations are time consuming on any, even moderately used host. So, do first live migration, and then stop old server and repeat migration.
3. In any case, don’t do migration when the old server is running because you might leave your new server in inconsistent state.

Now, go to the old installation of Zimbra and copy the whole Zimbra tree over the Zimbra installation on the new server. The command should look something like this:

$ rsync -av —delete —exclude zimbramon/lib /opt/zimbra root@newserver:/opt/

Probably this will take some time. Also, don’t forget to exclude zimbramon/lib directory or otherwise you’ll end up with an old version of Perl and when you try to start some tool from Zimbra you’ll get the following error message:

$ zmcontrol stop
Perl lib version (v5.8.8) doesn’t match executable version (v5.16.3) at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/Config.pm line 46.
Compilation failed in require at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
BEGIN failed—compilation aborted at /opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi/lib.pm line 6.
Compilation failed in require at /opt/zimbra/bin/zmcontrol line 25.
BEGIN failed—compilation aborted at /opt/zimbra/bin/zmcontrol line 25.

Anyway, in case you got the previous error just copy again zimbramon/lib directory from Zimbra 8 as explained earlier in the text.

At this point zmcontrol stop/start wan’t work almost certainly. So, I suggest that you try to start service by service and check each one if it works or not. Checks can be performed looking at the process list or using netstat/ss command to see listening ports. In case some service doesn’t work, check log files in /opt/zimbra/log directory and the file /var/log/zimbra.log. Actually, it would be good to check log files no matter if you think a service works or not.

The first service you should start is ldap. That one is easy, just run the following command as a zimbra user:

$ ldap start

then, look if there is slapd process running:

$ ps -ef | grep sldapd
zimbra   30224     1  0 06:56 ?        00:00:00 /opt/zimbra/openldap/sbin/slapd -l LOCAL0 -u zimbra -h ldap://mail.zemris.fer.hr:389 ldapi:/// -F /opt/zimbra/data/ldap/config

and you can also check if ldap port (389) is opened:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 3.1.2.1:389      0.0.0.0:*               LISTEN
tcp        0      0 3.1.2.1:53       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:53     0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:953    0.0.0.0:*               LISTEN
tcp6       0      0 ::1:53           :::*                    LISTEN
tcp6       0      0 :::22            :::*                    LISTEN
tcp6       0      0 ::1:953          :::*                    LISTEN

Logging, by the way, goes to syslog/journalctl because of the option -l, and that means we should look into /var/log/zimbra.log for slapd messages:

Next, I tried to start mailbox service:

$ zmmailboxdctl start

and that didn’t work. Namely, no java process:

$ ps -ef | grep java
zimbra    4717  1470  0 07:19 pts/0    00:00:00 grep —color=auto java

and no listening sockets for imap, pop, etc.:

$ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 3.1.2.1:389      0.0.0.0:*   LISTEN
tcp        0      0 3.1.2.1:53       0.0.0.0:*   LISTEN
tcp        0      0 127.0.0.1:53     0.0.0.0:*   LISTEN
tcp        0      0 0.0.0.0:22       0.0.0.0:*   LISTEN
tcp        0      0 127.0.0.1:953    0.0.0.0:*   LISTEN
tcp6       0      0 ::1:53           :::*        LISTEN
tcp6       0      0 :::22            :::*        LISTEN
tcp6       0      0 ::1:953          :::*        LISTEN

Looking into log file (/opt/zimbra/log/zmmailboxd.out) reveals the following error messages:

Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        … 18 more
1227 WARN  [main] log — Nested in java.lang.reflect.InvocationTargetException:
java.net.BindException: Cannot assign requested address

After few more minutes of scratching my head of why (while constantly resisting temptation to use strace tool) I finally realized that the problem is related to my customization on the old server. Namely, I changed jetty configuration so that Zimbra binds to a specific IP address which isn’t available on the new server. The solution for this is easy, I saved a copy of Zimbra 7 immediately after installation (zimbra.orig directory, remember) so the following two commands solved this particular problem:

$ cd /opt/zimbra.orig/zimbra/$ rsync -av —delete jetty-6.1.22.z6 /opt/zimbra/

So, again I tried to start mailbox service on Zimbra:

$ zmmailboxdctl startStarting mailboxd…done.

This time, it seems to be good. But checking ports and processes again showed that it doesn’t work! Looking at log file showed the following cryptic error:

Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented
        at javax.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:80)
        at com.zimbra.common.util.NetUtil.newBoundServerSocket(NetUtil.java:95)
        at com.zimbra.common.util.NetUtil.bindServerSocket(NetUtil.java:163)
        at com.zimbra.common.util.NetUtil.bindSslTcpServerSocket(NetUtil.java:54)
        …

That was very cryptic error message. Luckily, Googling for it immediately gave the following thread. Namely, the problem is with SSL/TLS, and the following two commands from the given thread fixed that part of the problem:

This time, it almost worked as now I had two java processes and listening sockets. But, there were a log of messages in log file like the following one:

find /opt/zimbra -uid 500 -exec chown zimbra.zimbra {} ;

Now, both LDAP and mailbox services worked! Time to try zmcontrol start but first I stopped running services. Still didn’t work. This time I received two error messages from Web and mail processes:

Starting apache…/opt/zimbra/httpd-2.2.22/bin/httpd: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory
failed.
/opt/zimbra/postfix/sbin/postalias: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory
/opt/zimbra/postfix/sbin/postfix: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory

But these are easy to solve, just copy missing libraries from CentOS 5 installation. And note that there is a library and symlink to the library! I tried starting Zimbra again using zmcontrol. Again, no luck. The problem this time is a missing Perl subroutine while starting stat module:

Undefined subroutine &main::getZimbraHome called at /opt/zimbra/bin/zmstatctl line 156.

This subroutine was removed in Perl packages distributed with Zimbra 8.6, which we are using here. I found this subroutine to be defined in Zmstat.pl file and first I tried to fix it by just copying this old file, but it turns out that more than one file was changed so I tried, and succeeded, with by copying all non-binary files:

Trying again zmcontrol we are receiving warnings about deprecated use of certain Perl features:

Use of qw(…) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 25.
Use of qw(…) as parentheses is deprecated at /opt/zimbra/zimbramon/lib/Zimbra/Util/Common.pm line 26.

Finally, I got to the step where I tried to upgrade Zimbra to a newer version. Go to the unpacked Zimbra 8 directory and start upgrade process as usual:

./install.sh —platform-override

Validating ldap configuration
Error: Unable to create a successful TLS connection to the ldap masters.
       Fix cert configuration prior to upgrading.

$ zmlocalconfig -e ldap_master_url=ldaps://mymaster.somewhere.com:636
$ zmlocalconfig -e ldap_url=ldaps://myreplica.somewhere.com:636
$ zmlocalconfig -e ldap_starttls_supported=0
$ zmlocalconfig -e ldap_port=636 

# openssl x509 -in /opt/zimbra/conf/ca/ca.pem -noout -text | grep ‘Not After’
            Not After : Apr 11 11:22:21 2015 GMT

ERROR 1133 (28000) at line 2: Can’t find any matching row in the user table
ERROR 1396 (HY000) at line 1: Operation DROP USER failed for »@’mail.somedomain.com’

Also, at one point installation stopped because some errors with LDAP’s password. The configuration menu was presented with lot of marks that things have to be fixed. I selected LDAP password, and when the configuration program assigned new password everything after that went well.

And that’s it! Unfortunately, my old mail has some additional services I had to migrate too, but I won’t write about them in this post.

I have to note several things. First, this process heavily depends on configurations on both new and old server being as close as possible. This means that if you did customizations that are outside of /opt/zimbra tree, then you’ll have to transfer those too. In my case, I integrated mailman with Zimbra, and mailman lives in other directories, so I had problems. Second,

So, here is the recipe on how to move old Zimbra installation to the new server. This is condensed version of the previous section.

Preparation

1. Download ZCS 7.2.7 (for RHEL6) and ZCS 8.6.0 (you can download newer version if you wish, but I didn’t test it so expect there might be some issues). I’ll assume you downloaded them into /tmp directory.
2. Unpack both archives.
3. Enter into ZCS 8.6 directory and then into packages/ subdirectory. Make there temporary directory (e.g. tmp/) and enter into it. Unpack core package (zimbra-core-8.6.0_GA_1153.RHEL7_64-20141215151110.x86_64.rpm) using rpm2cpio and cpio commands.
4. Enter into ZCS 7.2.7 directory and modify file util/utilfunc.sh file. Line 2287 should be PACKAGEINST=’rpm —force -iv’.
5. From the old server copy the following libraries into /lib64 directory on a new server: libssl.so.*, libcrypto.so.*, libpcre.so.*, and libexpat.so.*.

Installation

1. Enter direcotry ZCS 7.2.7 and start installation with the following command:
   ./install.sh -s —platform-override
2. When script finishes, go to /opt/zimbra/zimbramon directory rename lib directory to lib.orig.
3.  Go to the directory where you unpacked zimbra-core package (should be something like /tmp/zcs-8.6.0-something/packages/tmp/) and execute the following command:
   rsync -av opt/zimbra/zimbramon/lib /opt/zimbra/zimbramon/.
4. Run the command /opt/zimbra/libexec/zmsetup.pl. In case the script complains about missing Perl module and stops, install the missing module and run the script again. 
5. Stop Zimbra and disable starting during the boot process:

   # /etc/init.d/zimbra stop
   # chkconfig zimbra off
6. Make a copy of vanilla Zimbra directory tree, i.e. execute the following commands:

   # cd /opt
   # rsync -av zimbra zimbra.orig

Syncing with old server

1. Go to the old server and run the following command:

   rsync -av —delete —exclude zimbramon/lib —exclude jetty* /opt/zimbra newserver:/opt/

2. Again on the new server run the following commands (as a zimbra user):

   $ zmlocalconfig -e zimbra_uid=997
   $ zmlocalconfig -e zimbra_gid=996
   
   Change UID and GID to match your new installation!

3. Turn on LDAP server, i.e. switch to the Zimbra user and execute the following command:

   $ ldap start

4. Run the following commands to generate new CA.
   
   # /opt/zimbra/bin/zmcertmgr createca -new
   # /opt/zimbra/bin/zmcertmgr createcrt -new -days 1825
   # /opt/zimbra/bin/zmcertmgr deploycrt self
   # /opt/zimbra/bin/zmcertmgr deployca

   Might not be necessary, but it doesn’t hurt. And yes, if you have commercial CERT, you’ll have to import it and these steps are not necessary.

5. Turn off LDAP server, i.e. switch to the Zimbra user and execute the following command:

   $ ldap stop

6. Run the following commands to «fix» Perl version mismatch on new and old servers:

   # cd /opt/zimbra.orig/zimbra/zimbramon/lib.orig
   # rsync -av —exclude *.so —exclude x86_64-linux-thread-multi * /opt/zimbra/zimbramon/lib/

Upgrade

1. Go to the directory where you unpacked ZCS 8.6 and run upgrade process as usual:

   ./install.sh —platform-override