Failed to start slapd attempting debug start to determine error zimbra

A week ago, I received a call from one of our clients to whom we provide system and networking support. They had called me to inform that their Zimbra server had stopped working after an unexpected power outage. So, I asked them to provide me a remote access to their server so that I could analyze the cause of the problem. It didn’t strike me as a big deal then because there had been lots of issues with this server previously and most of them were about spam filters, users’ accounts, or some services not working properly. After looking at the problem, I learned that it was due to the failed LDAP server whose database, according to Zimbra, had been corrupted. Well, it was something new. Then, as most of us do when facing problem, I googled the Zimbra’s error message and error code to find information on this issue.

The error message I got looked like this:

[zimbra@mail ~]$ ldap start
Failed to start slapd. Attempting debug start to determine error.
56d2e1e8 mdb_db_open: database "": mdb_dbi_open(/opt/zimbra/data/ldap/mdb/db/ad2i) failed: MDB_CORRUPTED: Located page was wrong type (-30796).
56d2e1e8 backend_startup_one (type=mdb, suffix=""): bi_db_open failed! (-30796)

Zimbra LDAP DB Recovery

The search results on this error message mostly provided two main recovery methods: first, to recover the corrupted database and second, to restore the previously backed up working database. So, I backed up my existing database and tried first method. As a matter of fact, Zimbra itself ships with a LDAP database recovery tool by default. I found numerous articles which taught database recovery for  earlier versions or Network edition of Zimbra and OpenLDAP only. However, I have Community Edition of Zimbra and its version is as follows:

[zimbra@mail ~]$ zmcontrol -v
Release 8.5.0_GA_3042.RHEL6_64_20140828192005 RHEL6_64 FOSS edition.

And the LDAP database can be recovered in this Zimbra version as follows:

[zimbra@mail ~]$ /opt/zimbra/bdb/bin/db_recover -v -h /opt/zimbra/data/ldap/mdb/db
BDB2526 Finding last valid log LSN: file: 1 offset 28

But unfortunately for me, this didn’t help. Instead of going through the recovery process, db_recover exited without making any changes to the database. Perhaps I messed up with my log file or accidentally deleted it before trying recovery. I can’t tell this for sure because I went through lots of articles about this issue and tried few of the suggestions. Later I learned from some OpenLDAP folks that deleting or modifying database log file is very bad and can make database recovery almost impossible. Or, there was another possibility that both my system and database assumed that their information was correct, but had noncomplying entries. According to some MDB or LDAP DB experts, sometimes while writing information into database and there is immediate power cut, the operating system or application assumes that the information has been successfully written into database, but the database doesn’t have any knowledge of it. This mismatch can lead to database corruption and can also create trouble in recovery.

 After failing to recover my corrupted database repeatedly, there was the second option to get my LDAP and Zimbra running i.e. restoring previously backed up database. But since I didn’t have any backup whatsoever of this server and my predecessors also didn’t take any backup, I skipped this option too. I guess you can imagine the pressure and stress one feels under such circumstance. So, I was left with no other choice than to setup a new Zimbra server to at least allow our clients to send and receive new mails.

Restoring Old Mails in New Server

Setting up a new Zimbra server wasn’t much of a big deal. I was able to install a fresh CentOS system and ZCS in a couple of hours. We had a list of mail users  and distribution lists from a year ago, using which I created users and distribution lists in my new server. Then, I asked my network team to change the NAT setting in the Gateway device from old server’s IP to new server’s IP address. After that, the new server was ready to send and receive mails on behalf of our clients. This reduced the stress level a little, but the challenge of restoring older mails and contacts was still there.

Once the new Zimbra server was up and running with basic configuration and tuning, I started looking into restoration of old mails and contacts from crashed server. For that purpose, I researched so extensively that I at least read a hundred articles and forum posts. I looked under every stone I could find and I even turned over some stones time and again to discover any clue for my goal. Even after failing multiple times at database recovery, I kept on trying to recover it because the more I researched about my problem, the more I realized that LDAP database is the key in Zimbra’s recovery or restoration or migration. Therefore, I needed a way to either recover my LDAP database or restore old mails and contacts from older server’s user accounts to new server.

While trying to restore mails from old server to new server, one helpful member from Zimbra community suggested me a way to do it. You can find my question in this link: http://community.zimbra.com/collaboration/f/1886/t/1141281 Following this suggestion, I created a new user account in new server and restored all the older mails to this account. Here’s how I did it:

[root@mail ~]$ screen -S Sajjan
[root@mail ~]$ mkdir /opt/zimbra/store/0/Recovered-mails
[root@mail ~]$ rsync -av root@192.168.2.1:/opt/zimbra/store /opt/zimbra/store/0/Recovered-mails

To elaborate my steps, I first created a screen session because I was working over SSH and I didn’t want the processes I created to die on my disconnection. After creating a session and starting my work inside it, I could easily close and resume my work later by connecting to my screen. Then, I created a folder called Recovered-mails inside store folder and copied the entire store folder from old server to new server using Rsync. Since I had more than 128 GB of mails and I was copying them over network, it was a good thing to run it inside screen. Now you see the benefit of Screen, right?

[root@mail ~]$ su zimbra
[zimbra@mail ~]$ zmprov ca recovery@sajjan.com.np <password> displayName "Recovery Account"
[zimbra@mail ~]$ zmmailbox -z -m recovery@sajjan.com.np cf /Old-Mails
[zimbra@mail ~]$ zmmailbox -z -m "recovery@sajjan.com.np" addMessage /Old-Mails /opt/zimbra/store/0/Recovered-mails/store/0/*/msg/*

Next, I logged into zimbra account to control mail server. First, I added a new user called recovery@sajjan.com.np with zmprov. Then, I created a folder Old-Mails in the mailbox of this user, where I would be restoring all the old mails into. At last, I restored all mails inside Recovered-mails directory into the Old-Mails folder of recovery@sajjan.com.np. I was planning to apply filters for getting mails of specific users and then redirecting the filtered mails to their respective user accounts. But this plan was going cost a lot of time and effort for me because I had about 180 user accounts and the volume of mails was also quite high. And as you may now guess, I wasn’t going down that road for the lazy type of person that I am. So, I started to search for better ways to accomplish this or to automate things. Until I had a better idea, I manually filtered the recovered mails and redirected/shared them to few users based on the criticality and urgency of their mails.

Finally I came to learn that Zimbra creates a directory for each user account inside the store/0 directory of zimbra’s home directory and names these user directories as the respective IDs of the users. For example, if a MySQL ID of a particular user is 10 and zimbra has been installed in /opt directory, then all mails belonging to that user are stored inside /opt/zimbra/store/0/10 directory. And this information is the key for the successful restoration and mapping of all user mails from one to another server. As simple as it sounds, everything I did next is primarily based on this small concept. And now, the fun part begins.

In order to map and restore mails of a user from old server to new server, I needed to know the ID of that user in the old server because that’s the value which was attached to the mail store. Thank goodness, I had MySQL server running in old server. As a matter of fact, Zimbra’s MySQL server isn’t dependent upon its LDAP server/database. MySQL server of old server wasn’t running by default, so I started it manually and ran a select query to fetch the list of user’s IDs and email addresses. Note this is in old server.

[zimbra@old-mail ~]$ mysql.server start
[zimbra@old-mail ~]$ mysql -e "SELECT id,comment FROM zimbra.mailbox;"
1    admin@sajjan.com.np
2    user1@sajjan.com.np
3    user2@sajjan.com.np
[zimbra@old-mail ~]$ mysql -e "SELECT id,comment FROM zimbra.mailbox;" > /tmp/id-comment.list 

Doing this dumps the list of IDs and emails of all users in old server’s database into a file called id-comment.list in /tmp. That’s all I needed to do in my old server. Then, I went back to new server and copied the dump file.

[zimbra@mail ~]$ scp root@192.168.2.1:/tmp/id-comment.list /tmp

Then, I wanted to create a folder called Old-Mails in the mailbox of every user to store their past mails. I could’ve done it from the web panel by logging into every user’s account one-by-one and creating folder there or by entering a folder creation command for every user. But I had over 180 users, so I wasn’t going to do that. Therefore, I created a script to do that for me, which is mentioned below. When running this script, I didn’t use the id-comment.list file that I created earlier because the number of users in old server was greater than that in the new server and also that list contained ID values in it, which was unnecessary in this script. So, I generated a list of user accounts present in the new server and passed this new list to the script.

[zimbra@mail ~]$ mysql -e "SELECT comment FROM zimbra.mailbox;" > /tmp/accounts.list
[zimbra@mail ~]$ vi /tmp/create_folder.sh 

#!/bin/bash
#Author: Sajjan Bhattarai
#Date: March 3, 2016
#Description: This script takes a list of accounts in the new Zimbra server and create a folder named "Old-Mails" in the mailbox of each user, where all the mails from old mail server will be restored.

# Looping through users in the list
while IFS='' read -r user || [[ -n "$user" ]]; do
	echo "$user: Creating Old-Mails folder..."
	# Creating folder inside a user's mailbox
	output=$(zmmailbox -z -m "$user" cf /Old-Mails)
	# Handling error status of the above command
	if [ $? -eq 0 ]
	then
		echo "Success!"
	else
		echo "Failed!"
		echo "$user:" >> create_folder.log
		echo "$output" >> create_folder.log
	fi
done < "$1"

[zimbra@mail ~]$ chmod +x /tmp/create_folder.sh
[zimbra@mail ~]$ /tmp/create_folder.sh /tmp/accounts.list

Running script as above created Old-Mails folder inside mailbox of every user in the new server. If for any reason this folder isn’t created in any user’s account, this failed event will be logged with its error message in the create_folder.log. If you’re wondering why I created new folder to store old mails rather than simply restoring them into existing Inbox folder, I did it that way mainly for safety purpose and simplicity. There’s no any absolute reason for it. Once these folders were created, I wrote another script to restore old mails into the user accounts to which they belonged to. This script takes the id-comment.list file that I created earlier.

[zimbra@mail ~]$ vi /tmp/restore_mails.sh

#!/bin/bash
#Author: Sajjan Bhattarai
#Date: March 3, 2016
#Description: This script takes the list of zimbra accounts' ids and email addresses and inputs the mails associated with those users from old mail store to new Zimbra server's mailbox

#Splitting IDs and Email addresses in every line delimited by whitespace
while IFS='' read -r user || [[ -n "$user" ]]; do
	# Creating array containing ID and Email
	user_id=($user)
	echo "${user_id[0]} : ${user_id[1]}"
	# Restoring mails from user's mail directory to new mailbox inside folder named "Old-Mails
	zmmailbox -z -m "${user_id[1]}" addMessage /Old-Mails /opt/zimbra/store/0/Recovered-mails/store/0/"${user_id[0]}"/msg/*	
done < "$1"

[zimbra@mail ~]$ chmod +x /tmp/restore_mails.sh
[zimbra@mail ~]$ /tmp/restore_mails.sh /tmp/id-comment.list

Well, this is it. I successfully restored past mails of all users into their respective mailbox. But my work didn’t end there because I didn’t have as much user accounts in new server as in old server. That means, I was missing some users and along with that, mails associated with those users. To be exact, I lacked 40 user accounts and it wasn’t forgivable if I were to let it go.

Completing the Users’ List in New Server

To complete the users’ list, I first needed to identify the users missing in my new server. To do this, I once again utilized the previously used id-comments.list and accounts.list file. First, I needed to extract only user accounts from id-comments.list so that I could perform comparison between them and accounts in accounts.list to determine lacking accounts. For this purpose, I wrote another little script which created accounts.old.list containing email addresses only. Then, I compared the account lists from old and new servers and put the missing accounts in a new file accounts.missing.

[zimbra@mail ~]$ vi /tmp/extract_accounts.sh 

#!/bin/bash
#
while IFS='' read -r user || [[ -n "$user" ]]; do
  user_id=($user)
  echo "${user_id[1]}" >> /tmp/accounts.old.list
done < "$1" 

[zimbra@mail ~]$ chmod +x /tmp/extract_accounts.sh
[zimbra@mail ~]$ /tmp/extract_accounts.sh /tmp/id-comments.list
[zimbra@mail ~]$ comm -2 -3 < (sort /tmp/accounts.old.list) < (sort /tmp/accounts.list) > /tmp/accounts.missing

After I had a list of missing accounts, I passed this list to my another script which adds users into Zimbra server.

[zimbra@mail ~]$ vi /tmp/add_users.sh 

#!/bin/bash
# Author: Sajjan Bhattarai
# Date: March 5, 2016
# Description: This bash script takes the list of user accounts from old zimbra server as an argument and creates the corresponding email accounts in the new Zimbra server.

domain="sajjan.com.np"
default_password="sajjan@12345#"

# This script creates user accounts with default CoS, so user specific CoS hasn't been defined here. You may add CoS settings accordingly.

# Splitting email address into user and domain name
while IFS='@' read -a user || [[ -n "$user" ]]; do
	# Splitting user account into firstname and lastname
	IFS='.' read -a names <<< "$user"
	# Capitalizing first letter of names
	firstname="$(tr '[:lower:]' '[:upper:]' <<< ${names[0]:0:1})${names[0]:1}"
	lastname="$(tr '[:lower:]' '[:upper:]' <<< ${names[1]:0:1})${names[1]:1}"
	echo "Creating $user@$domain..."
	# Creating user account in Zimbra server
	#output=$(zmprov ca "$user"@"$domain" "$default_password" displayName "$firstname $lastname")
	# Checking the exit status of zmprov function
	if [ $? -eq 0 ]
	then
		echo "Success!"
		echo "$user@$domain" >> new_users.list
	else
		echo "Failed!"
		echo "$user@$domain:" >> generate_users.log
		echo "$output" >> generate_users.log
		echo "===================================================" >> generate_users.log
	fi
done < "$1"

[zimbra@mail ~]$ chmod +x /tmp/add_users.sh
[zimbra@mail ~]$ /tmp/add_users.sh /tmp/accounts.missing

On completion of this script, all missing accounts were successfully added into Zimbra. If for some reason an account wasn’t added, that failing account would be logged into /tmp/generate_users.log file along with the error message. I then created Old-Mails folder for these users as well.

[zimbra@mail ~]$ /tmp/create_folder.sh /tmp/accounts.missing

Then, to restore past mails of these newly added users, I had to know the user ID of each of these accounts in old server. I could’ve known ID of each user by making a query to MySQL server of old server and then used that ID to restore mails to that user’s mailbox as follows:

[zimbra@old-mail ~]$ mysql -e "SELECT id,comment from zimbra.mailbox WHERE comment LIKE 'missing-user@sajjan.com.np';"
168
[zimbra@mail ~]$ zmmailbox -z -m missing-user@sajjan.com.np addMessage /Old-Mails /opt/zimbra/store/0/Recovered-mails/store/0/168/msg/*

Or, I could’ve also opened up /tmp/id-comment.list file and searched for missing-user@sajjan.com.np to find its ID. Then, I could’ve used above zmmailbox command to restore mails. But I had to do it for 40 accounts, which would’ve taken quite a long time and effort. So, I created a script for it as well.

[zimbra@mail ~]$ vi /tmp/generate_remaining_id_comment_list.sh 

#!/bin/bash

# Author: Sajjan Bhattarai
# Date: March 6, 2016
# Description: This script takes the list of remaining zimbra accounts and fetches their respective IDs from the list containing both ID and Email address. After fetching the matching ID and Email, it stores them into remaining-id-comment.list file.

# Looping through each missing user account
while IFS='' read -r user || [[ -n "$user" ]]; do
  # Creating array containing ID and Email
  output=$(cat /tmp/id-comment.list | grep -w "$user")
  echo "$output" >> /tmp/remaining-id-comment.list
done < "$1" 

[zimbra@mail ~]$ chmod +x /tmp/generate_remaining_id_comment_list.sh
[zimbra@mail ~]$ /tmp/generate_remaining_id_comment_list.sh /tmp/accounts.missing

Lastly, I put this remaining-id-comment.list file into the restore_mails.sh script, which restored all the past mails of remaining users into their mailbox.

[zimbra@mail ~]$ /tmp/restore_mails.sh /tmp/remaining-id-comment.list

Finally, this is it. My Zimbra mail server has been fully restored. I never thought of it, but now I’m truly thankful toward the problem I faced. Because without it, I wouldn’t have gone any deeper into Zimbra and wouldn’t have stepped any further from my comfort zone. Yes, it nearly brought me down, but I raised myself and solved it. Moral of the story: Don’t stop when you’re tired and cannot find a way to solve a problem, rather stop when you’ve fully solved it. Thank you for reading!