Psql error fatal peer authentication failed for user zabbix

Zabbix version: 3.4
Operating System: Red Hat Enterprise Linux 7.4
Database: PostgreSQL 9.2.23
Platform: AWS EC2

Zabbix is an open-source network monitoring system that collects and stores metrics from networks, servers, and more. Released in March 2004, it been utilized in enterprises environment due to its high performance and scalability. See the Zabbix wikipedia page and the Comparison of Network Monitoring Systems page for more information.

Zabbix has 4 major components:

1. Server: the main component. Collects data from the clients.
2. Database: a database used by Zabbix to store metrics that are collected. This can be MySQL, PostgreSQL, SQLite, Oracle, or IBM DB2.
3. Web Client: a web UI that enables configurations and allows data visualizations
4. Client: an agent that is installed on servers that you want to monitor

My specific use case: to collect metrics on my homelab and network in order to keep track of the system and network usage. I want to then visualize these metrics on a dashboard, with a visualization platform like Grafana. The long term goal is to learn how to collect and visualize metrics – then bridge this to an automation platform to automatically kick of scripts in response to some alert given by the metrics.

So let’s get to it: installing Zabbix 3.4 on a RHEL 7.4 machine with PostgreSQL. I will be following the official Zabbix documentation here:

1. First, find the rpm from Zabbix from the Zabbix repository for RHEL 7 here. Note this package contains configuration files for yum (a package manager), and is not the full installation.
2. Install the rpm with the rpm command, plus the -i (install) -v (verbose) and -h (human readable) flags, plus the full url to the rpm. Like so:

   sudo rpm -ivh http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm

3. Now that the rpm is installed, you can see that it added a zabbix.repo file in /etc/yum.repos.d/ and this allows you to use yum to install Zabbix. To install the main server, and web frontend, you can do this with the following command:

   sudo yum install zabbix-server-pgsql zabbix-web-pgsql

   Note that you may want to separate the zabbix server and the zabbix web to different servers. This will allow one to run without impacting the performance of the other. But for the purpose of this tutorial I will install both on the same server.

4. Next, if you haven’t already, you also need to install the postgreSQL client and server.
   1. Install the postgreSQL client with:

      sudo yum install postgresql

   2. Install the postgreSQL server with:

      sudo yum install postgresql-server postgresql-contrib

   3. Now you need to initialize the postgreSQL server. Do this by:

      sudo postgresql-setup initdb

   4. Now start the server, and the enable it to automatically start every time you boot.
      
sudo systemctl start postgresql
sudo systemctl enable postgresql

5. Now that the postgreSQL database is ready, you need to create the schema (set of tables and relationships between them).
   1. First you need to create a Zabbix user in the postgreSQL database. You can create one by running the command below, and create a password for the user when prompted (remember this password for later!):

      sudo -u postgres createuser --pwprompt zabbix

   2. Next, create a database belonging on the newly created Zabbix user:

      sudo -u postgres createdb -O zabbix zabbix

      This command runs “createdb -O zabbix zabbix” as the user postgres. The ‘-O zabbix’ switch makes the Zabbix user the owner of the database. The second ‘zabbix’ specifies the name of the database (which is zabbix).

   3. Now we need to create the schema. Luckily, this part is largely automated. You will just need to run the command below. Be warned, this will print out a lot of lines!

      zcat /usr/share/doc/zabbix-server-pgsql-3.4.5/create.sql.gz | sudo -u zabbix psql -d zabbix

      Note: for the sake of understanding commands before you run them, let’s break this command down.

      1. We have a file from zabbix that contains all the postgreSQL commands to create the database schema. This file is located in /usr/share/doc/zabbix-server-pgsql-3.4.5/create.sql.gz but notice the ending “.gz” which is a compressed file in gzip format.
      2. zcat is like the command cat. cat prints out the entire content of a file all at once, but cat does not work for compressed files. Instead, zcat is used to cat compressed files. So “zcat /usr/share/doc/zabbix-server-pgsql-3.4.5/create.sql.gz” will print out the contents of the compressed file create.sql.gz
      3. Next, the | is called a pipe, which passes the results of the left command, to the command on the right. In this case, this passes the contents of create.sql.gz over to the next command “sudo -u postgres psql -d zabbix”
      4. This sudo command uses the -u switch to run the command “psql -d zabbix” as the zabbix user. “psql -d zabbix” will run the psql client to connect to the zabbix database.
   4. If you are interested in seeing what the schema looks like, you can connect to the postgreSQL database. You can run the postgreSQL client with psql and the switches -U (user) and -d (database). We will first try with the zabbix user.

      psql -U zabbix -d zabbix

      This should prompt you for a password. However, postgreSQL has some extra security that prevents you from connecting as Zabbix, giving you the error:

      psql: FATAL: Peer authentication failed for user "zabbix"

      So let’s think. There are two ways to work around this:

      1. The first way: Connect instead as the postgres user. You can switch to the postgres account and then connect to the database like so:
         
sudo su - postgres
psql -d postgres


         Once you successfully connect, you will see the following terminal prompt:

         postgres=#

         Now that you are connected to the postgres database, you can type l to list all databases, and dt or dt+ to list all tables in the zabbix database. When ready, type q to quit back to your bash shell.

      2. The second way: You can modify the security of your postgreSQL server. The postgreSQL server is configured through the pg_hba.conf file located in /var/lib/pgsql/data/pg_hba.conf (Note: you can type find / -name pg_hba.conf if is in a different location). To modify this file:

         sudo vi /var/lib/pgsql/data/pg_hba.conf

         Find the following section near the bottom:
         
# TYPE DATABASE USER ADDRESS METHOD

         


         # "local" is for Unix domain socket connections only
local all all peer


         Then add the new line as shown. Press i to edit.
         
# TYPE DATABASE USER ADDRESS METHOD

         


         # "local" is for Unix domain socket connections only
local all all peer
local all zabbix password


         Afterwards, type :wq to save and quit. You will need to restart the postgreSQL server for the changes to take effect. Do this by

         sudo systemctl restart postgresql

         Now try to connect again, and it should ask you for your password this time.

         psql -U zabbix -d zabbix

         Now that you are connected to the postgres database, you can type l to list all databases, and dt or dt+ to list all tables in the zabbix database. To see a particular table, type select * from tablename;

         If there are too many columns, you can type x to turn on expanded display to see the table properly. Type x off to turn off.

         When ready, type q to quit back to your bash shell.

   5. Now that the postgreSQL database is all set up, you need to point Zabbix to the server. You can do this by editing zabbix_server.conf and find the 4 separate lines shown below and modify yours to be the same as mine (with the exception of password – please put the password you set up for the Zabbix user). Remember to use i to edit and :wq to save and quit.
      
sudo vi /etc/zabbix/zabbix_server.conf
DBHost=
DBName=zabbix
DBUser=zabbix
DBPassword=password

6. Now with everything configured we’re ready to start Zabbix! Run the command below to start the Zabbix server service.
7. sudo systemctl start zabbix-server

   However, I’m getting an error:

   Job for zabbix-server.service failed because a configured resource limit was exceeded. See "systemctl status zabbix-server.service" and "journalctl -xe" for details.

   After double checking all my work and making sure that it is correct, I’m looking into the error to troubleshoot why:

   1. Run the suggested commands to see the errors in the log:

      
systemctl status zabbix-server.service
Dec 30 09:27:41 ip-172-31-30-81.us-west-2.compute.internal systemd[1]: zabbix-server.service never wrote its PID file. Failing.
Dec 30 09:27:41 ip-172-31-30-81.us-west-2.compute.internal systemd[1]: Failed to start Zabbix Server.
Dec 30 09:27:41 ip-172-31-30-81.us-west-2.compute.internal systemd[1]: Unit zabbix-server.service entered failed state.
Dec 30 09:27:41 ip-172-31-30-81.us-west-2.compute.internal systemd[1]: zabbix-server.service failed.


   2. A look at the zabbix server log:

      
less /var/log/zabbix/zabbix_server.log
9431:20171230:091807.617 cannot set resource limit: [13] Permission denied
9431:20171230:091807.617 cannot disable core dump, exiting...
9434:20171230:091817.866 Starting Zabbix Server. Zabbix 3.4.5 (revision 76340).


   3. Doing a quick google search for “Zabbix disable core dump permission denied” reveals that this is a SELinux issue documented in ZBX-10542. However, this is supposed to be fixed in CentOS 7.4, yet we are having the same issue. A suggested fix was to install the updated selinux-policy package (selinux-policy noarch 3.13.1-166.el7_4.7).

      sudo yum install selinux-policy

      But this did not fix the issue, even after a reboot.

   4. There are now two ways to move forward: SELinux to permissive, or use a workaround suggested in ZBX-10542. I chose to set SELinux to permissive mode. You can do this by editing /etc/sysconfig/selinux and change SELINUX=enforcing to SELINUX=permissive
      
sudo vi /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
   5. You will need to reboot your server for the new SELinux mode to take effect. To reboot:

      sudo shutdown -r now

   6. When the server is back up, start Zabbix server again:

      sudo systemctl start zabbix-server

   7. Now it works! Check the status and you should see “Started Zabbix Server”

      sudo systemctl status zabbix-server

8. Set Zabbix server to start on boot up:

   sudo systemctl enable zabbix-server

9. Congratulations, you have set up Zabbix server! Coming up soon is to finish configuring the Zabbix frontend web page.

WEB FRONT END

1. Install Apache, PHP, and zabbix’s web front end.

   sudo yum install httpd php zabbix-web-pgsql

   Troubleshooting

   Problem
   When installing zabbix-web-pgsql you may encounter the following error:

   
Error: Package: zabbix-web-3.4.6-1.el7.noarch (zabbix)
Requires: php-bcmath
Error: Package: zabbix-web-3.4.6-1.el7.noarch (zabbix)
Requires: php-mbstring


   Workflow:

   • The error indicates that the php-bcmath and php-mbstring packages are required by zabbix-web, but yum cannot find and install these packages.
   • A quick google of these packages brought me to this site that said the php-bcmath and php-mbstring are part of RHEL’s optional packages repo.
   • The rhel-server-optional repo was disabled. Enabling this repo reveals the php-bcmath and php-mbstring packages.
   • If you couldn’t find out that these packages are in the rhel-server-optional repo, you could enable all the repos one by one to see which one it is in.

   Solution
   Enable the rhel/centos optional server packages.

   1. vi /etc/yum.repos.d/redhat-rhui.repo
   2. Look for [rhui-REGION-rhel-server-optional]>
   3. In that section, change “enabled=0” to “enabled=1”. Save and quit.
   4. You can now install the package via yum install zabbix-web-pgsql
2. Edit /etc/httpd/conf.d/zabbix.conf. Uncomment the line below and change the timezone (bolded). Use this site to determine the appropriate location:

   # php_value date.timezone Europe/Riga

3. Start the apache server, and enable it to automatically start on boot:
   
sudo systemctl start httpd
sudo systemctl enable httpd


   In a web browser, go to http://host:80/zabbix

   If you are on the same server as your zabbix installation, go to http://localhost:80/zabbix

   Troubleshooting

   Problem:
   I am unable to connect to my Zabbix web server on AWS through my laptop.

   Workflow:

   • Check to the httpd service to see if it is running: systemctl status httpd
   • httpd service was running, so this seems like a networking issue.
   • I have an active ssh connection to the server, so it cannot be that the internet link is down on either side.
   • Perhaps some traffic is blocked as a security measure. We are trying to connect to port 80. ssh works via port 22. Something else we can try is by doing an ICMP ping, so I pinged the server: ping ip.address
   • Pings were not being answered. As suspected, perhaps there is a firewall/security rules in place.
   • Looking around on AWS EC2 console lead me to the Security Groups page where there was a security group that only allowed TCP traffic through port 22 (for ssh). But httpd uses port 80, so I added a rule to allow TCP traffic on port 80. Now I can load the zabbix front end in my browser.

   Solution:
   Configure the AWS Security Groups to allow for TCP packets in via port 80 (apache/httpd uses this port).
4. If everything has been set up correctly, you will be greeted by the Zabbix setup UI. Click “Next Step”.
5. This next screen checks for the PHP pre-requisites. If you’re running CentOS or RHEL 7, and installed PHP 5.4 then you should meet all the pre-requisites. Note that there are optional requirements, seen by the lack of a required value in the ‘Required’ column. It is okay to continue if you see a fail or a warning for these requirements. Click “Next Step” when ready.
6. Now you will set up Zabbix’s frontend to connect to the zabbix database. Put in the required information. In this case, I will put:
   
Database type: PostgreSQL
Database host: localhost (My frontend is on the same host as the server/database. If yours is not, put the hostname of your zabbix database server)
Database port: 0 (This defaults to default PostgreSQL port, 5432)
Database name: zabbix
Database user: zabbix
Password: (hint: it's also in /etc/zabbix/zabbix_server.conf set as DBPassword)


   Click “Next Step” to continue.

   Troubleshooting

   Problem:
   An error occurred while establishing a connection with the PostgreSQL database.

   Workflow:
   I wondered whether specifying the host ‘localhost’ would mean Zabbix tries to connect using the host ‘localhost’ as if you run the command psql -U zabbix -d zabbix -h localhost. That would fail since that is different than a local connection, such as psql -U zabbix -d zabbix. Try out the two commands, and see which one works. If you get an ‘Ident authentication error’ refer to my Step 4 to allow local password authentication.

   Solution:
   Delete the ‘Database host’ text field, and leave it blank so that no hostname is specified. This allowed Zabbix to establish a local connection to the psql database.

7. For this next step, you can set up a connection with the Zabbix server. Enter the hostname, and the port number. If your Zabbix server and Zabbix web are on the same host, use the hostname ‘localhost’. Otherwise, put in the hostname of your Zabbix server. Click “Next Step” to continue.
8. It will now show you a review. Look at the information provided and check to make sure everything is correct. Go back and correct anything as necessary. Otherwise, click “Next Step” to continue.
9. If everything went well, it will tell you that the configuration file has been placed in /usr/share/zabbix/conf/zabbix.conf.phpIf it could not create the configuration file, you can download it and save it as /usr/share/zabbix/conf/zabbix.conf.php (in the Zabbix web host)
10. Lastly, click “Finish”. You should now see the Zabbix login screen. The default credentials are:
    Username: Admin
    Password: zabbix

That’s it! Congratulations on setting up your Zabbix system!

You have successfully installed Zabbix with a postgreSQL backend and a web frontend!