Rpc error no cluster leader

For a new project Consul was checked if it could be used as a backend system for data. By quick-reading through the documentation, Consul looks promising, so far. Although the installation was quick and painless, a couple of communication errors occurred due to blocked firewall ports. Unfortunately the logged errors were not really helpful nor did they give a hint what exactly could be the problem. Here’s an overview of the Consul installation, configuration and how to solve communication errors in the Consul cluster.

Setting up Consul

The setup is pretty easy as Consul is just a single binary file. And a config file you create which the binary will read. That’s it. This means that installation is equal to a download, unzip and move:

root@consul01:~# wget https://releases.hashicorp.com/consul/1.5.3/consul_1.5.3_linux_amd64.zip
root@consul01:~# unzip consul_1.5.3_linux_amd64.zip
root@consul01:~# mv consul /usr/local/bin/

Create a dedicated user/group for Consul and create the destination directory for its data:

root@consul01:~# useradd -m -d /home/consul -s /sbin/nologin consul
root@consul01:~# mkdir /var/lib/consul
root@consul01:~# chown -R consul:consul /var/lib/consul

Create the config file. This config file can have any name you want and place it in any directory, as long as the «consul» user is able to read it.

Here’s an example config file with placeholders:

root@consul01:~# cat /etc/consul.json
{
  «server»: true,
  «node_name»: «$NODE_NAME»,
  «datacenter»: «dc1»,
  «data_dir»: «$CONSUL_DATA_PATH»,
  «bind_addr»: «0.0.0.0»,
  «client_addr»: «0.0.0.0»,
  «advertise_addr»: «$ADVERTISE_ADDR»,
  «bootstrap_expect»: 3,
  «retry_join»: [«$JOIN1», «$JOIN2», «$JOIN3»],
  «ui»: true,
  «log_level»: «DEBUG»,
  «enable_syslog»: true,
  «acl_enforce_version_8»: false
}

In order to launch Consul as a service, a Systemd unit file should be created:

Make sure, the paths are correct!

Now enable the service:

root@consul01:~# systemctl daemon-reload
root@consul01:~# systemctl enable consul.service

Configure Consul Cluster

A Consul cluster should have a minimum of three cluster nodes. In this example I created three nodes:

• consul01: 192.168.253.8
• consul02: 192.168.253.9
• consul03: 10.10.1.45

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

Whenever you are installing and running Consul from HashiCorp on Oracle Linux you might run into some strange errors. Even though your configuration JSON file passes the configuration validation the log file contains a long repetitive lst of the same error complaining about «failed to sync remote state: No cluster leader» and » coordinate update error: No cluster leader».

Consul is a tool for service discovery and configuration. It provides high level features such as service discovery, health checking and key/value storage. It makes use of a group of strongly consistent servers to manage the datacenter. Consul is developed by HasiCorp and is available from its own website.

It might be that you have the below output when you start consul:

    2016/11/25 21:03:50 [INFO] raft: Initial configuration (index=0): []
    2016/11/25 21:03:50 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
    2016/11/25 21:03:50 [INFO] serf: EventMemberJoin: consul_1 127.0.0.1
    2016/11/25 21:03:50 [INFO] serf: EventMemberJoin: consul_1.private_dc 127.0.0.1
    2016/11/25 21:03:50 [INFO] consul: Adding LAN server consul_1 (Addr: tcp/127.0.0.1:8300) (DC: private_dc)
    2016/11/25 21:03:50 [INFO] consul: Adding WAN server consul_1.private_dc (Addr: tcp/127.0.0.1:8300) (DC: private_dc)
    2016/11/25 21:03:55 [WARN] raft: no known peers, aborting election
    2016/11/25 21:03:57 [ERR] agent: failed to sync remote state: No cluster leader
    2016/11/25 21:04:14 [ERR] agent: coordinate update error: No cluster leader
    2016/11/25 21:04:30 [ERR] agent: failed to sync remote state: No cluster leader
    2016/11/25 21:04:50 [ERR] agent: coordinate update error: No cluster leader
    2016/11/25 21:05:01 [ERR] agent: failed to sync remote state: No cluster leader
    2016/11/25 21:05:26 [ERR] agent: coordinate update error: No cluster leader
    2016/11/25 21:05:34 [ERR] agent: failed to sync remote state: No cluster leader
    2016/11/25 21:06:02 [ERR] agent: coordinate update error: No cluster leader
    2016/11/25 21:06:10 [ERR] agent: failed to sync remote state: No cluster leader
    2016/11/25 21:06:35 [ERR] agent: coordinate update error: No cluster leader

The main reason for the above is that you try to start consul in an environment where there is no cluster available, or it is the first node of the cluster. In case you start it as the first node of the cluster or as the only node of the cluster you have to ensure that you include -bootstrap-expect 1 as a command line option when starting (in case you will only have one node).

You can also include «bootstrap_expect»: 1 in the json configuration file if you use a configuration file to start Consul.

As an example, the below start of Consult will prevent the above errors:

consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul

I have tried to follow this guide and while it doesn’t seem to work I also followed an old guide we had created before for an upgrade from 0.5.2 to 0.7.1. The below method is the guide we created before but with updated versions and slight changes to it.

My reason for not following the guide I linked to is that with chef and our script is sees that one or more servers still exist and will not bootstrap the server.

Do this on all nodes first

wget https:
//releases.hashicorp.com/consul/1.3.1/consul_1.3.1_linux_amd64.zip
unzip consul_1.3.1_linux_amd64.zip # rename to consul_1.3.1 when prompt'd
rm consul_1.3.1_linux_amd64.zip

Do this on all nodes

./consul leave

Then rename everything

mv consul consul_0.7.1
mv consul_1.3.1 consul

Finally re-run chef-client which in turn runs a ruby script that checks for existing nodes and if found will create a new regular server and but if no nodes exist it will use the -bootstrap flag to bootstrap the server.

When I do the above steps I can’t seem to check with ./consul monitor -token=<master acl token> so what I have done is copied the commands from the ruby script and manually did them which allowed me to see the output where it says it can’t select a leader. See below log

2018/12/13 14:57:15 [INFO] consul: New leader elected: consul-testing-node-2
2018/12/13 14:57:16 [ERR] agent: Coordinate update error: No cluster leader
2018/12/13 14:57:17 [WARN] raft: Election timeout reached, restarting election
2018/12/13 14:57:17 [INFO] raft: Node at 10.7.1.5:8300 [Candidate] entering Candidate state in term 175
2018/12/13 14:57:17 [WARN] raft: Unable to get address for server id 10.7.1.6:8300, using fallback address 10.7.1.6:8300: Could not find address for server id 10.7.1.6:8300
2018/12/13 14:57:17 [ERR] agent: failed to sync remote state: No cluster leader

Commands used

/work-disk/consul/consul agent -bootstrap -data-dir /work-disk/consul_data -config-file /work-disk/consul/conf.json -bind 10.7.1.5

/work-disk/consul/consul agent -data-dir /work-disk/consul_data -config-file /work-disk/consul/conf.json -bind 10.7.1.5

Conf.json file

{
"client_addr": "0.0.0.0",
"datacenter": "consul-testing",
"acl_datacenter": "consul-testing",
"acl_default_policy":"deny",
"acl_down_policy": "deny",
"acl_master_token": "<master acl token>",
"acl_agent_token": "<agent acl token>",
"data_dir": "/work-disk/consul_data",
"domain": "consul",
"enable_script_checks": true,
"dns_config": {
"enable_truncate": true,
"only_passing": true
},
"enable_syslog": true,
"encrypt": "<enc key>==",
"leave_on_terminate": true,
"log_level": "INFO",
"rejoin_after_leave": true,
"server": true,
"start_join": ["10.7.1.5", "10.7.1.6"]
}