Channel error on connection rabbitmq

Overview

This guide covers various topics related to channels, an AMQP 0-9-1-specific abstraction.
Channels cannot exist without a connection, so getting familiar with the Connections guide first
is highly recommended.

This guide covers:

• The basics of channels
• Channel lifecycle
• Channel exceptions (errors) and what they mean
• Channel resource usage
• Monitoring and metrics related to channels and how to identify common problems
• Flow control

and other topics related to connections.

The Basics

Some applications need multiple logical connections to the
broker. However, it is undesirable to keep many TCP
connections open at the same time because doing so consumes
system resources and makes it more difficult to configure
firewalls. AMQP 0-9-1 connections are multiplexed with
channels that can be thought of as «lightweight
connections that share a single TCP connection».

Every protocol operation performed by a client happens on a channel.
Communication on a particular channel is completely separate
from communication on another channel, therefore every protocol
method also carries a channel ID (a.k.a. channel number), an integer
that both the broker and clients use to figure out which channel the method is for.

A channel only exists in the context of a connection and never on its own.
When a connection is closed, so are all channels on it.

For applications that use multiple threads/processes for
processing, it is very common to open a new channel per thread/process
and not share channels between them.

Channel Lifecycle

Opening Channels

Applications open a channel right after successfully opening a connection.

Here’s a Java client example that opens a new channel with an automatically allocated channel ID
after opening a new connection:

ConnectionFactory cf = new ConnectionFactory();
Connection conn = cf.createConnection();

Channel ch = conn.createChannel();

// ... use the channel to declare topology, publish, consume

In .NET client channels are represented using the IModel interface, so the names in the API
are different:

var cf = new ConnectionFactory();
var conn = cf.newConnection();

// the .NET client calls channels "models"
var ch = conn.CreateModel();

// ... use the channel to declare topology, publish, consume

Much like connections, channels are meant to be long lived. That is, there is no need to open
a channel per operation and doing so would be very inefficient, since opening a channel is a
network roundtrip.

Closing Channels

When a channel is no longer needed, it should be closed. Closing a channel will render it
unusable and schedule its resources to be reclaimed:

Channel ch = conn.createChannel();

// do some work

// close the channel when it is no longer needed

ch.close();

The same example using the .NET client:

// the .NET client calls channels "models"
var ch = conn.CreateModel();

// do some work

// close the channel when it is no longer needed

ch.Close();

As mentioned above, a closed channel cannot be used. An attempt to perform an operation
on a closed channel will result in an exception that says that the channel has already been
closed.

When a channel’s connection is closed, so is the channel.

Channels and Error Handling

In the section above channels were closed by applications. There is another way a channel can
be closed: due to a protocol exception.

Certain scenarios are assumed to be recoverable («soft») errors in the protocol. They render
the channel closed but applications can open another one and try to recover or retry a number of
times. Most common examples are:

• Redeclaring an existing queue or exchange with non-matching properties
  will fail with a 406 PRECONDITION_FAILED error
• Accessing a resource the user is not allowed to access will fail
  with a 403 ACCESS_REFUSED error
• Binding a non-existing queue or a non-existing exchange will fail with a 404 NOT_FOUND error
• Consuming from a queue that does not exist will fail with a 404 NOT_FOUND error
• Publishing to an exchange that does not exist will fail with a 404 NOT_FOUND error
• Accessing an exclusive queue from a connection other than its declaring one will
  fail with a 405 RESOURCE_LOCKED

Client libraries provide a way to observe and react to channel exceptions. For example, in the Java
client there is a way to register an error handler and access a channel
shutdown (closure) reason.

Any attempted operation on a closed channel will fail with an exception. Note that when RabbitMQ
closes a channel, it notifies the client of that using an asynchronous protocol method. In other words,
an operation that caused a channel exception won’t fail immediately but a channel closure event
handler will fire shortly after.

Some client libraries may use blocking operations that wait for
a response. In this case they may communicate channel exceptions differently, e.g. using
runtime exceptions, an error type, or other means appropriate for the language.

See the AMQP 0-9-1 Reference for a more complete list of
error codes.

Resource Usage

Each channel consumes a relatively small amount of memory on the client. Depending on client library’s
implementation detail it can also use a dedicated thread pool (or similar) where consumer
operations are dispatched, and therefore one or more threads (or similar).

Each channel also consumes a relatively small amount of memory on the node the client is connected to,
plus a few Erlang processes. Since a node usually serves multiple channel connections, the effects
of excessive channel usage or channel leaks will primarily be reflected in RabbitMQ nodes’ metrics
and not those of clients.

Given both of these factors, limiting the number of channels used per connection is highly recommended.
As a guideline, most applications can use a single digit number of channels per connection.
Those with particularly high concurrency rates (usually such applications are consumers)
can start with one channel per thread/process/coroutine and switch to channel pooling
when metrics suggest that the original model is no longer sustainable, e.g. because it consumes
too much memory.

See the Monitoring, Metrics and Diagnostics section to learn about how to inspect
channels, the number of channels on a connection, channel churn rate and so on.

Maximum Number of Channels per Connection

The maximum number of channels that can be open on a connection simultaneously
is negotiated by client and server at connection time. The value is configurable for
both RabbitMQ and client libraries.

On the server side, the limit is controlled using the channel_max:

# no more 100 channels can be opened on a connection at the same time
channel_max = 100

Should the configured limit be exceeded, the connection will be closed with a fatal
error:

2019-02-11 16:04:06.296 [error] <0.887.0> Error on AMQP connection <0.887.0> (127.0.0.1:49956 -> 127.0.0.1:5672, vhost: '/', user: 'guest', state: running), channel 23:
 operation none caused a connection exception not_allowed: "number of channels opened (22) has reached the negotiated channel_max (22)"

Clients can be configured to allow fewer channels per connection. With RabbitMQ Java client,
ConnectionFactory#setRequestedChannelMax is the method that controls the limit:

ConnectionFactory cf = new ConnectionFactory();
// Ask for up to 32 channels per connection. Will have an effect as long as the server is configured
// to use a higher limit, otherwise the server's limit will be used.
cf.setRequestedChannelMax(32);

With RabbitMQ .NET client, use the ConnectionFactory#RequestedChannelMax
property:

var cf = new ConnectionFactory();
// Ask for up to 32 channels per connection. Will have an effect as long as the server is configured
// to use a higher limit, otherwise the server's limit will be used.
cf.RequestedChannelMax = 32;

The lower value of the two is used: the client cannot
be configured to allow for more channels than the server configured maximum.
Clients that attempt that will run into an error that looks like this in the logs:

2019-02-11 16:03:16.543 [error] <0.882.0> closing AMQP connection <0.882.0> (127.0.0.1:49911 -> 127.0.0.1:5672):
failed to negotiate connection parameters: negotiated channel_max = 2047 is higher than the maximum allowed value (32)

Monitoring, Metrics and Diagnostics

Number of currently open channels and channel opening/closure rates are important metrics
of the system that should be monitored. Monitoring them will help detect a number of
common problems:

• Channel leaks
• High channel churn

Both problems eventually lead to node exhaustion of resources.

Individual channel metrics such as the number of unacknowledged messages
or basic.get operation rate can help identify irregularities and inefficiencies
in application behavior.

Memory Use

Monitoring systems and operators alike may need to inspect how much memory
channels consume on a node, the total
number of channels on a node and then identify how many there are on each connection.

The number of channels is displayed in the management UI on the Overview tab,
as is the number of connections.
By dividing the number of channels by the number of connections
the operator can determine an average number of channels per connection.

To find out how much memory on a node is used by channels, use rabbitmq-diagnostics memory_breakdown:

rabbitmq-diagnostics memory_breakdown -q --unit mb
# => [elided for brevity]
# ...
# => connection_channels: 3.596 mb (2.27%)
# ...
# => [elided for brevity]

See the RabbitMQ Memory Use Analysis guide for details.

Channel Leaks

A channel leak is a condition under which an application repeatedly opens channels without closing them,
or at least closing only some of them.

Channel leaks eventually exhaust the node (or multiple target nodes) of RAM and CPU resources.

Relevant Metrics

Management UI’s Overview tab lists a total number of channels in all virtual hosts the current
user has access to:

To inspect the current number of channels on a connection as well as the per-connection channel limit, navigate
to the Connections tab and enable the relevant columns if they are not displayed:

Overview and individual node pages provide a chart of channel churn rate as of RabbitMQ 3.7.9.
If the rate of channel open operations is consistently higher than that of channel close operations,
this is evidence of a channel leak in one of the applications:

To find out what connection leaks channels, inspect per-connection channel count as demonstrated in this guide.

High Channel Churn

A system is said to have high channel churn when its rate of newly opened channels is consistently high and
its rate of closed channels is consistently high. This usually means that an application
uses short lived channels or channels are often closed due to channel-level exceptions.

While with some workloads this is a natural state of the system,
long lived channels should be used instead when possible.

Management UI provides a chart of channel churn rate.
Below is a chart that demonstrates a fairly low channel churn with a virtually identical number of channel open and closed
in the given period of time:

While connection and disconnection rates are system-specific, rates consistently above 100/second likely indicate a suboptimal
connection management by one or more applications and usually are worth investigating.

Note that some clients and runtimes (notably PHP) do not use long-lived connections and high connection
churn rates are expected from them unless a specialized proxy is used.

Inspecting Channels and Their State in Management UI

To inspect channels in the management UI, navigate to the Channels tab and add or remove columns
as needed:

Inspecting Channels and Their State Using CLI Tools

rabbitmqctl list_connections and rabbitmqctl list_channels are the
primary commands for inspecting per-connection channel count and channel details such as the number of
consumers, unacknowledged messages, prefetch and so on.

rabbitmqctl list_connections name channels -q
# => name    channels
# => 127.0.0.1:52956 -> 127.0.0.1:5672    10
# => 127.0.0.1:52964 -> 127.0.0.1:5672    33

The rightmost column contains channel count on the connection.

Table headers can be suppressed:

rabbitmqctl list_connections name channels -q --no-table-headers
# => 127.0.0.1:52956 -> 127.0.0.1:5672    10
# => 127.0.0.1:52964 -> 127.0.0.1:5672    33

To inspect individual channels, use rabbitmqctl list_channels:

rabbitmqctl list_channels -q
# => pid user    consumer_count  messages_unacknowledged
# => <rabbit@mercurio.3.815.0> guest   0   0
# => <rabbit@mercurio.3.820.0> guest   0   0
# => <rabbit@mercurio.3.824.0> guest   0   0
# => <rabbit@mercurio.3.828.0> guest   0   0
# => <rabbit@mercurio.3.832.0> guest   0   0
# => <rabbit@mercurio.3.839.0> guest   0   0
# => <rabbit@mercurio.3.840.0> guest   0   0

Table headers can be suppressed:

rabbitmqctl list_channels -q --no-table-headers
# => <rabbit@mercurio.3.815.0> guest   0   0
# => <rabbit@mercurio.3.820.0> guest   0   0
# => <rabbit@mercurio.3.824.0> guest   0   0
# => <rabbit@mercurio.3.828.0> guest   0   0
# => <rabbit@mercurio.3.832.0> guest   0   0
# => <rabbit@mercurio.3.839.0> guest   0   0
# => <rabbit@mercurio.3.840.0> guest   0   0

It is possible to display a different set of columns:

rabbitmqctl list_channels -q --no-table-headers vhost connection number  prefetch_count messages_unconfirmed
# => /   <rabbit@mercurio.3.799.0> 1   0   0
# => /   <rabbit@mercurio.3.802.0> 1   0   0
# => /   <rabbit@mercurio.3.799.0> 2   0   0
# => /   <rabbit@mercurio.3.799.0> 3   0   0
# => /   <rabbit@mercurio.3.802.0> 2   0   0
# => /   <rabbit@mercurio.3.802.0> 3   0   0
# => /   <rabbit@mercurio.3.799.0> 4   0   0
# => /   <rabbit@mercurio.3.802.0> 4   0   0
# => /   <rabbit@mercurio.3.799.0> 5   0   0
# => /   <rabbit@mercurio.3.799.0> 6   0   0

rabbitmqctl list_channels -s vhost connection number confirm
# => /   <rabbit@mercurio.3.799.0> 1   false
# => /   <rabbit@mercurio.3.802.0> 1   false
# => /   <rabbit@mercurio.3.799.0> 2   false
# => /   <rabbit@mercurio.3.799.0> 3   false
# => /   <rabbit@mercurio.3.802.0> 2   false
# => /   <rabbit@mercurio.3.802.0> 3   false
# => /   <rabbit@mercurio.3.799.0> 4   false
# => /   <rabbit@mercurio.3.802.0> 4   false
# => /   <rabbit@mercurio.3.799.0> 5   false

Publisher Flow Control

Channels that publish messages can outpace other parts of the system, most likely busy queues and queues
that perform replication. When that happens, flow control is applied to
publishing channels and, in turn, connections. Channels and connections that only consume messages
are not affected.

With slower consumers that use automatic acknowledgement mode
it is very likely that connections and channels will experience flow control when writing to
the TCP socket.

Monitoring systems can collect metrics on the number of connections in flow state.
Applications that experience flow control regularly may consider to use separate connections
to publish and consume to avoid flow control effects on non-publishing operations (e.g. queue management).

RabbitMQ cluster has been seen to exhibit some blocking behaviour, and becoming unreachable to external operations after being subjected to certain conditions. The logs reveal channel errors as below, which are caused by certain exchanges not being found.

=ERROR REPORT==== 14-Dec-2015::16:44:40 ===
Channel error on connection <0.5462.0> (192.168.245.3:53044 -> 192.168.245.5:5672, vhost: '/', user: 'rmq_nova_user'), channel 1:
operation queue.bind caused a channel exception not_found: "no exchange 'reply_5f360ace6a4d4a9e83c9d276877e4d4b' in vhost '/'"

This also leads to some channel_termination_timeout exceptions as follows;

=CRASH REPORT==== 14-Dec-2015::16:49:05 ===
  crasher:
    initial call: rabbit_reader:init/4
    pid: <0.7418.0>
    registered_name: []
    exception exit: channel_termination_timeout
      in function  rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 766)
      in call from rabbit_reader:send_error_on_channel0_and_close/4 (src/rabbit_reader.erl, line 1503)
      in call from rabbit_reader:terminate/2 (src/rabbit_reader.erl, line 615)
      in call from rabbit_reader:handle_other/2 (src/rabbit_reader.erl, line 540)
      in call from rabbit_reader:mainloop/4 (src/rabbit_reader.erl, line 502)
      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 427)
      in call from rabbit_reader:start_connection/4 (src/rabbit_reader.erl, line 385)
    ancestors: [<0.7416.0>,<0.2520.0>,<0.2519.0>,<0.2518.0>,rabbit_sup,
                  <0.634.0>]
    messages: [{'EXIT',#Port<0.22137>,normal}]
    links: []
    dictionary: [{{channel,1},
                   {<0.7445.0>,{method,rabbit_framing_amqp_0_9_1}}},
                  {process_name,
                      {rabbit_reader,
                          <<"192.168.245.8:48658 -> 192.168.245.3:5672">>}},
                  {{ch_pid,<0.7445.0>},{1,#Ref<0.0.0.40007>}}]
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 7946
  neighbours:

The following procedures can be carried out to reproduce this problem, assuming a Linux environment, Debian preferrably, RabbitMQ 3.4.3 or above, and preferrably Erlang 17.3;

On a 3 node cluster (NODE_A, NODE_B, NODE_C) with queue(s) decalred with HA policy set to {"ha-mode": "all", "ha-sync-mode":"automatic"} and cluster_partition_handling configured to pause_minority, apply traffic to the cluster whilst carrying out the following sequence of events, on NODE_A for example;

   rabbitmqctl list_channels
   rabbitmqctl stop_app
   rabbitmqctl reset
   rabbitmqctl join_cluster <rabbit@NODE_B>
   rabbitmqctl start_app
   rabbitmqctl list_queues -q synchronised_slave_pids
   sleep 2
   rabbitmq-server stop
   sleep 5
   service rabbitmq-server start
   iptables -A INPUT -s <NODE_B_IP> -p TCP -j DROP
   iptables -A INPUT -s <NODE_C_IP> -p TCP -j DROP

These procedures are repeated on NODE_B and NODE_C, with NODE_B joining NODE_C, and NODE_C joining NODE_A, respectively.

При тестировании очереди TTL в RABBITMQ сегодня возникает ошибка, сообщение об ошибке выглядит следующим образом:

Exception in thread “main” java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:124)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:120)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:142)
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:958)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.queueDeclare(AutorecoveringChannel.java:333)
at com.fu.rabbitemq.ttl.Producer.main(Producer.java:27)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED — inequivalent arg ‘x-message-ttl’ for queue ‘queue_ttl_1’ in vhost ‘/’: received the value ‘10000’ of type ‘signedint’ but current is none, class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:443)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:263)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:136)
… 3 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED — inequivalent arg ‘x-message-ttl’ for queue ‘queue_ttl_1’ in vhost ‘/’: received the value ‘10000’ of type ‘signedint’ but current is none, class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:515)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:340)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:162)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:109)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:676)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:603)
at java.lang.Thread.run(Thread.java:748)

решение

После расследования оказалось, что на сервере уже есть очередь queue_ttl_1 на сервере, и эта очередь не является очередью TTL, поэтому его снова найдут.

Удалите очередь и повторно запустите код. Бегите правильно следующим образом:

I’m trying to start rabbitmq, yet it seems to crash upon start and producing errors:

handshake_error,opening,amqp_error,internal_error and refused for user:

rabbitmq    | 2019-08-14 15:10:16.053 [info] <0.244.0> 
rabbitmq    |  Starting RabbitMQ 3.7.16 on Erlang 22.0.7
rabbitmq    |  Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbitmq    |  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbitmq    | 
rabbitmq    |   ##  ##
rabbitmq    |   ##  ##      RabbitMQ 3.7.16. Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbitmq    |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbitmq    |   ######  ##
rabbitmq    |   ##########  Logs: <stdout>
rabbitmq    | 
rabbitmq    |               Starting broker...
rabbitmq    | 2019-08-14 15:10:16.054 [info] <0.244.0> 
rabbitmq    |  node           : rabbit@rabbitmq
rabbitmq    |  home dir       : /var/lib/rabbitmq
rabbitmq    |  config file(s) : /etc/rabbitmq/rabbitmq.conf
rabbitmq    |  cookie hash    : AIlteC+QMYJQCC1CZZToPg==
rabbitmq    |  log(s)         : <stdout>
rabbitmq    |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq
rabbitmq    | 2019-08-14 15:10:16.065 [info] <0.244.0> Running boot step pre_boot defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.065 [info] <0.244.0> Running boot step rabbit_core_metrics defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.066 [info] <0.244.0> Running boot step rabbit_alarm defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.105 [info] <0.252.0> Memory high watermark set to 12065 MiB (12651329945 bytes) of 30163 MiB (31628324864 bytes) total
rabbitmq    | 2019-08-14 15:10:16.194 [info] <0.262.0> Enabling free disk space monitoring
rabbitmq    | 2019-08-14 15:10:16.194 [info] <0.262.0> Disk free limit set to 50MB
rabbitmq    | 2019-08-14 15:10:16.198 [info] <0.244.0> Running boot step code_server_cache defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.198 [info] <0.244.0> Running boot step file_handle_cache defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.199 [info] <0.267.0> Limiting to approx 1048476 file handles (943626 sockets)
rabbitmq    | 2019-08-14 15:10:16.199 [info] <0.268.0> FHC read buffering:  OFF
rabbitmq    | 2019-08-14 15:10:16.199 [info] <0.268.0> FHC write buffering: ON
rabbitmq    | 2019-08-14 15:10:16.201 [info] <0.244.0> Running boot step worker_pool defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.201 [info] <0.245.0> Will use 8 processes for default worker pool
rabbitmq    | 2019-08-14 15:10:16.201 [info] <0.245.0> Starting worker pool 'worker_pool' with 8 processes in it
rabbitmq    | 2019-08-14 15:10:16.202 [info] <0.244.0> Running boot step database defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.203 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step database_sync defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step codec_correctness_check defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.274 [info] <0.244.0> Running boot step external_infrastructure defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_registry defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_queue_location_random defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_event defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.275 [info] <0.244.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_priority_queue defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Priority queues enabled, real BQ is rabbit_variable_queue
rabbitmq    | 2019-08-14 15:10:16.276 [info] <0.244.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step kernel_ready defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_sysmon_minder defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.277 [info] <0.244.0> Running boot step rabbit_epmd_monitor defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.281 [info] <0.244.0> Running boot step guid_generator defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.286 [info] <0.244.0> Running boot step rabbit_node_monitor defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.286 [info] <0.324.0> Starting rabbit_node_monitor
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step delegate_sup defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step rabbit_memory_monitor defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step core_initialized defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.287 [info] <0.244.0> Running boot step upgrade_queues defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_connection_tracking defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_exchange_parameters defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.307 [info] <0.244.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_policies defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_policy defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_queue_location_validator defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.308 [info] <0.244.0> Running boot step rabbit_vhost_limit defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.309 [info] <0.244.0> Running boot step recovery defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.346 [info] <0.578.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
rabbitmq    | 2019-08-14 15:10:16.348 [error] <0.579.0> CRASH REPORT Process <0.579.0> with 0 neighbours crashed with reason: no match of right hand value {error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}} in rabbit_recovery_terms:open_table/1 line 197
rabbitmq    | 2019-08-14 15:10:16.349 [error] <0.578.0> Unable to recover vhost <<"/">> data. Reason {badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}
rabbitmq    |  Stacktrace [{rabbit_recovery_terms,start,1,[{file,"src/rabbit_recovery_terms.erl"},{line,53}]},{rabbit_queue_index,start,2,[{file,"src/rabbit_queue_index.erl"},{line,502}]},{rabbit_variable_queue,start,2,[{file,"src/rabbit_variable_queue.erl"},{line,483}]},{rabbit_priority_queue,start,2,[{file,"src/rabbit_priority_queue.erl"},{line,92}]},{rabbit_amqqueue,recover,1,[{file,"src/rabbit_amqqueue.erl"},{line,238}]},{rabbit_vhost,recover,1,[{file,"src/rabbit_vhost.erl"},{line,72}]},{rabbit_vhost_process,init,1,[{file,"src/rabbit_vhost_process.erl"},{line,56}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,554}]}]
rabbitmq    | 2019-08-14 15:10:16.350 [error] <0.578.0> CRASH REPORT Process <0.578.0> with 0 neighbours exited with reason: no match of right hand value {error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,...}]}]},...}} in gen_server2:init_it/6 line 589
rabbitmq    | 2019-08-14 15:10:16.350 [error] <0.576.0> Supervisor {<0.576.0>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_process started with rabbit_vhost_process:start_link(<<"/">>) at undefined exit with reason {badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib..."},...]}]},...}}} in context start_error
rabbitmq    | 2019-08-14 15:10:16.350 [warning] <0.244.0> Unable to initialize vhost data store for vhost '/'. The vhost will be stopped for this node.  Reason: {shutdown,{failed_to_start_child,rabbit_vhost_process,{badmatch,{error,{{{badmatch,{error,{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}}}
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step empty_db_check defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step rabbit_looking_glass defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step background_gc defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Running boot step connection_tracking defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.351 [info] <0.244.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@rabbitmq
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@rabbitmq
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step routing_ready defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step pre_flight defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step notify_cluster defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.352 [info] <0.244.0> Running boot step networking defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.354 [warning] <0.589.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
rabbitmq    | 2019-08-14 15:10:16.355 [info] <0.603.0> started TCP listener on [::]:5672
rabbitmq    | 2019-08-14 15:10:16.355 [info] <0.244.0> Running boot step direct_client defined by app rabbit
rabbitmq    | 2019-08-14 15:10:16.449 [info] <0.8.0> Server startup complete; 0 plugins started.
rabbitmq    |  completed with 0 plugins.
rabbitmq    | 2019-08-14 15:10:18.598 [info] <0.607.0> accepting AMQP connection <0.607.0> (10.0.0.69:44814 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.599 [info] <0.610.0> accepting AMQP connection <0.610.0> (10.0.0.69:44812 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.600 [info] <0.614.0> accepting AMQP connection <0.614.0> (10.0.0.69:44820 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.600 [info] <0.617.0> accepting AMQP connection <0.617.0> (10.0.0.69:44818 -> 10.0.0.138:5672)
rabbitmq    | 2019-08-14 15:10:18.656 [error] <0.607.0> Error on AMQP connection <0.607.0> (10.0.0.69:44814 -> 10.0.0.138:5672, vhost: 'none', user: 'guest', state: opening), channel 0:
rabbitmq    |  {handshake_error,opening,
rabbitmq    |                  {amqp_error,internal_error,
rabbitmq    |                              "access to vhost '/' refused for user 'guest': vhost '/' is down",
rabbitmq    |                              'connection.open'}}

Please advise.

related articles:Building Rabbitmq Server High Available Clusters

Specific error message:

2018-05-04 11:21:48.116 ERROR 60848 --- [.168.0.202:8001] o.s.a.r.c.CachingConnectionFactory       : Channel shutdown: connection error
2018-05-04 11:21:48.116 ERROR 60848 --- [.168.0.202:8001] o.s.a.r.c.CachingConnectionFactory       : Channel shutdown: connection error
2018-05-04 11:21:48.125  INFO 60848 --- [nge.consumer1-8] o.s.a.r.l.SimpleMessageListenerContainer : Restarting [email protected]: tags=[{amq.ctag-d_wIlZIGxM3f0fsxkmYQfA=my_test_exchange.consumer1}], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:8001/,1), conn: [email protected] Shared Rabbit Connection: [email protected] [delegate=amqp://[email protected]:8001/, localPort= 56258], acknowledgeMode=AUTO local queue size=0
2018-05-04 11:21:48.126  INFO 60848 --- [nge.consumer1-9] o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: [manager1:8001]
2018-05-04 11:21:48.393  INFO 60848 --- [nge.consumer1-9] o.s.a.r.c.CachingConnectionFactory       : Created new connection: rabbitConnectionFactory#2b8bd14b:12/[email protected] [delegate=amqp://[email protected]:8001/, localPort= 56260]
2018-05-04 11:21:49.059  INFO 60848 --- [nge.consumer1-8] o.s.a.r.l.SimpleMessageListenerContainer : Restarting [email protected]: tags=[{amq.ctag-T1HyrOd5Ykr_VQZDwxRslA=stream_exchange.consumer1}], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:8001/,2), conn: [email protected] Shared Rabbit Connection: [email protected] [delegate=amqp://[email protected]:8001/, localPort= 56260], acknowledgeMode=AUTO local queue size=0

Spring boot Configure Rabbitmq (using a Haproxy load balancing):

spring:
  application:
    name: stream-rabbitmq-producer
  rabbitmq:
    host: manager1
    port: 8001
    username: admin
    password: admin123456

Recently, when using the Rabbitmq cluster (Haproxy load balancing), frequent appearances of error messages, but the message can be normal consumption, if only the single-machine version Rabbitmq is used (not using haproxy), there is no error.

I have been troubled by this problem for a long time, Google found a lot of information, nor did it find a solution, unintentional to find an article:RabbitMQ and HAProxy: a timeout issue

The article says that if you use Haproxy to configure Rabbitmq, you will encounter a client connection timeout problem.

Why do you have this problem? Because Haproxy is configured with client connection timeout parameters (timeout client ms), If the client connection exceeds the configured parameter, then Haproxy will delete this client connection.

Rabbitmq client uses permanent connection to the agent, never timeout, why do you have problems?Because if Rabbitmq is in a non-active state within a period of time, then Haproxy will automatically turn off the connection (a little pit).

How do you solve this problem? We saw Haproxy provided aclitcpkaParameter configuration, it can send from the clientTCP keepalivedata pack.

We use it, but it is found to have the above problem after it is configured.

why?

[…]the exact behaviour of tcp keep-alive is determined by the underlying OS/Kernel configuration[…]

What do you mean? it meansTCP keepaliveThe delivery of the packet depends on the operating system / kernel configuration.

We can use the command to view (in the server where Haproxy is located)tcp_keepalive_timeConfiguration):

[[email protected] ~]# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

tcp_keepalive_timeDefault configuration time 2 hours, indicating sendingTCP keepaliveThe interval between packets is 2 hours, or it is sent every 2 hours.TCP keepalivedata pack.

It’s clear, let’s configure it in Haproxy.clitcpkaParameters, but because the system is sentTCP keepaliveThe interval between packets is too long, far exceeding Haproxytimeout clientTimeout (default seems to be 2 seconds), so the client connection is deleted by Haproxy ruthlessly, and then constantly rebuilt.

Said so much, how should we solve this problem?

Two options:

• Modify systemtcp_keepalive_timeConfigure, interval is below Haproxy configurationtimeout clientTimeout (because it is possible to affect other system services, not recommended).
• Modify Haproxytimeout clientTimeout, configuration is greater than the systemtcp_keepalive_timeInterval (recommended)

Because of the systemtcp_keepalive_timesendTCP keepaliveThe data package interval is 2 hours, so we will be in Haproxytimeout clientTimeout, set to 3 hours:

timeout client  3h
timeout server  3h

A complete example configuration:

[[email protected] ~]# cat /etc/haproxy/haproxy.cfg
global
    log     127.0.0.1  local0 info
global
    log     127.0.0.1  local1 notice
    daemon
global
    maxconn 4096

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    retries 3
    option  abortonclose
    maxconn 4096
    timeout connect  5000ms
    timeout client  3000ms
global
    timeout server  3000ms
    balance roundrobin

listen private_monitoring
    bind    0.0.0.0:8000
    mode    http
    option  httplog
    stats   refresh  5s
    stats   uri  /stats
    stats   realm   Haproxy
    stats   auth  admin:admin

listen rabbitmq_admin
    bind    0.0.0.0:8002
    server  manager1 manager1:15672
    server  manager2 manager2:15672
    server  manager3 manager3:15672

listen rabbitmq_cluster
    bind    0.0.0.0:8001
    mode    tcp
    option  tcplog
    balance roundrobin
    timeout client  3h
    timeout server  3h
    server  manager1  manager1:5672  check  inter  5000  rise  2  fall  3
    server  manager2  manager2:5672  check  inter  5000  rise  2  fall  3
    server  manager3  manager3:5672  check  inter  5000  rise  2  fall  3

Run Haproxy and then the Rabbitmq test successfully:

In a conversation, parties greet each other, exchange verbal banter, and
eventually continue on their way. A similar form of communication occurs
over low-level TCP connections exposing lightweight channels in RabbitMQ.
This article examines how clients, consumers, and brokers pass information in
RabbitMQ.

RabbitMQ was originally developed to support AMQP 0.9.1 which is the «core»
protocol supported by the RabbitMQ broker. Here are the

channels

used to send messages over

TCP connections.

What is a connection?

A connection (TCP) is a link between the client and the broker, that performs
underlying networking tasks including initial authentication, IP resolution,
and networking.

What is a channel?

Connections can multiplex over a single TCP connection, meaning that
an application can open «lightweight connections» on a single connection.
This «lightweight connection» is called a channel. Each connection can
maintain a set of underlying channels.

Many applications needs to
have multiple connections to the broker, and instead of having
many connections an application can reuse the connection, by instead, create
and delete channels. Keeping many TCP connections open at the same time is
not desired, as they consume system resources. The handshake process for a
connection is also quite complex and
requires at least 7 TCP packets or more if TLS is used.

A channel acts as a virtual connection inside a TCP connection.
A channel reuses a connection, forgoing the need to reauthorize and open a
new TCP stream. Channels allow you to use resources more efficiently (more
about this later in this article).

Every AMQP protocol-related operation occurs over a channel.

A connection is created by opening a physical TCP connection to the target
server. The client resolves the hostname to one or more IP addresses before
sending a handshake. The receiving server then authenticates the client.

To send a message or manage queues, a connection is created with the broker
before establishing a channel through a client. The channel packages the
messages and handles protocol operations. Clients send messages through
the channel’s
basic_publish
method. Queue creation and maintenance also occur here, such as AMQP commands
like
queue.create
and
exchange.create
are all sent over AMQP, on a channel.

Closing a connection closes all associated channels.

Publish a message to the RabbitMQ broker

We will look at a simple example from the Python library
Pika.

1. As with all clients, you establish a TCP connection.
2. After that, a logical channel is created for sending data or performing
   other operations (like the creation of a queue). You provide authorization
   information when instantiating a
   BlockingConnection
   since the broker verifies this information on a per-connection basis.
3. A message is routed to the queue, over the channel.
4. The connection is closed (and so the are all channels in the connection).

connection = pika.BlockingConnection(connection_parameters)
channel = connection.channel()
channel.basic_publish(exchange="my_exchange",
  routing_key="my_route",
  body= bytes("test_message")
)
connection.close()

Configuring the number of channels

We recommend to use the operator limit for connections and
channels.
Use
channel_max
to configure the max amount of allowed channels on a
connection. This variable corresponds to
rabbit.channel_max
in the new config format. Exceeding this limit results in a fatal error.
Use
connections_max
to configure the max amount of allowed connections.

A common question we get is how many channels one should have per RabbitMQ
connection, or how many channels is optimal. It’s hard to give an answer to
that since it always depends on the setup. Ideally, you should establish
one connection per process with a dedicated channel given to each new thread.

Setting
channel_max
to 0 means «unlimited». This could be a dangerous move, since applications
sometimes have channel leaks.

Avoiding connection and channel leaks

Two common user mistakes are channel and connection leaks, when a client
opens millions of connections/channels, causing RabbitMQ to crash due to memory issues.
To help catch these issues early, CloudAMQP provides alarms that can be
enabled.

Often, a channel or connection leak is the result of failing to close either
when finished.

Recommendations for connections and channels

Here follow some recommendations of how to use, and not to use connections
and channels.

Use long-lived connection

Each channel consumes a relatively small amount of memory on the client,
compared to a connection. Too many connections can be a heavy burden on the
RabbitMQ server memory usage. Try to keep long-lived connections and instead
open and close channels more frequently, if required.

We recommend that each process only creates one TCP connection and uses
multiple channels in that connection for different threads.

Separate the connections for publishers and consumers

Use at least one connection for publishing and one for consuming for each
app/service/process.

RabbitMQ can apply back pressure on the TCP connection
when the publisher is sending too many messages for the server to handle. If
you consume on the same TCP connection, the server might not receive the
message acknowledgments from the client, thus affecting the consumer
performance. With a lower consume speed, the server will be overwhelmed.

Don’t share channels between threads

Use one channel per thread in your application, and make sure that you don’t
share channels between threads as most clients don’t make channels
thread-safe.

CloudAMQP allows you to scale your instances to meet demand while providing
mechanisms to troubleshoot leaks. If you have any questions, you can reach out
to us at
support@cloudamqp.com