Error unable to perform an operation on node rabbitmq

Error: unable to perform an operation on node ‘rabbit@rabbitmq2’. #7

Hi, I’ve cloned and docker-compose up and got this error logs

rabbitmq2_1 | RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the —erlang-cookie switch instead.
rabbitmq2_1 | Stopping rabbit application on node rabbit@rabbitmq2 .
rabbitmq2_1 | Error: unable to perform an operation on node ‘rabbit@rabbitmq2’. Please see diagnostics information and suggestions below.
rabbitmq2_1 |
rabbitmq2_1 | Most common reasons for this are:
rabbitmq2_1 |
rabbitmq2_1 | * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
rabbitmq2_1 | * CLI tool fails to authenticate with the server (e.g. due to CLI tool’s Erlang cookie not matching that of the server)
rabbitmq2_1 | * Target node is not running
rabbitmq2_1 |
rabbitmq2_1 | In addition to the diagnostics info below:
rabbitmq2_1 |
rabbitmq2_1 | * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq2_1 | * Consult server logs on node rabbit@rabbitmq2
rabbitmq2_1 | * If target node is configured to use long node names, don’t forget to use —longnames with CLI tools
rabbitmq2_1 |
rabbitmq2_1 | DIAGNOSTICS
rabbitmq2_1 | ===========
rabbitmq2_1 |
rabbitmq2_1 | attempted to contact: [rabbit@rabbitmq2]
rabbitmq2_1 |
rabbitmq2_1 | rabbit@rabbitmq2:
rabbitmq2_1 | * connected to epmd (port 4369) on rabbitmq2
rabbitmq2_1 | * epmd reports: node ‘rabbit’ not running at all
rabbitmq2_1 | no other nodes on rabbitmq2
rabbitmq2_1 | * suggestion: start the node
rabbitmq2_1 |
rabbitmq2_1 | Current node details:
rabbitmq2_1 | * node name: ‘rabbitmqcli-45-rabbit@rabbitmq2’
rabbitmq2_1 | * effective user’s home directory: /var/lib/rabbitmq
rabbitmq2_1 | * Erlang cookie hash: gnzLDuqKcGxMNKFokfhOew==
rabbitmq2_1 |
rabbitmq3_1 | RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the —erlang-cookie switch instead.
rabbitmq3_1 | Stopping rabbit application on node rabbit@rabbitmq3 .

and idea why is that?

The text was updated successfully, but these errors were encountered:

Источник

Can’t stop the app while is starting #1853

Thank you for using RabbitMQ and for taking the time to report an
issue.

Does This Belong to GitHub or RabbitMQ Mailing List?

Important: please first read the CONTRIBUTING.md document in the
root of this repository. It will help you determine whether your
feedback should be directed to the RabbitMQ mailing list [1] instead.

Please Help Maintainers and Contributors Help You

In order for the RabbitMQ team to investigate your issue, please provide
as much as possible of the following details:

Two nodes are in rabbitmq cluster

1. start both nodes, running docker-compose up -d :
   Docker compose file:

1. login into queue ( host rabbit-master ) and run:

1. remove both containers to simulate a outage (used docker-compose down )
2. start only queue ( host rabbit-master ) running docker-compose up -d queue .

Rabbit can not start as can not reach queue-2 ( host rabbit-master-2 )
In the logs:

1. login into queue ( host rabbit-master ) and run:

Till here everything is as expected.

1. login into queue ( host rabbit-master ) and run:

At the same time, in the server logs i have

After 10 retries queue ( host rabbit-master ) exits as expected.

The question is:
How to detach the queue ( host rabbit-master ) node and allow to start the cluster with only that node?

The text was updated successfully, but these errors were encountered:

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team).

We get at least a dozen of questions through various venues every single day, often light on details.
At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because GitHub is a tool our team uses heavily nearly every day, the signal/noise ratio of issues is something we care about a lot.

Источник

FQDN problem with rabbitmqctl #613

Bit of a background to the situation:

• two AWS EC2 nodes, each running a docker container with rabbitmq (image is based on https://github.com/docker-library/rabbitmq/blob/master/management/Dockerfile)
• cluster setup (and working)
• firewall free — ports (amqp, erland dist, empd and management) bound to host
• rabbitmq containers had their hostname set to FQDN such as ip-10-200-1-00.eu-west-1.compute.internal that resolve fine from each node

rabbitmq-server started OK, we then join the second node to the first one using FQDN:

This works OK too and cluster is formed. However, when we attempt to use rabbitmqctl within either of these two containers, it wouldn’t work complaining erlang distribution failed. I’ve looked at https://www.rabbitmq.com/ec2.html and the suggestions about the hostname did not resolve our problem.
The response from rabbitmqctl:

Notice that current node details section lists node name in its short form even though in that case hostname was set to FQDN.

Thus, cluster works using full host names but rabbitmqctl fails. It failed to connect to the local and remote node, using FQDN as seen above as well as short hostname, giving the same message each time.

Since the hostnames are managed by AWS and short version easily resolvable, we decided to try using short hostnames for docker container hostname and join_cluster command while RABBITMQ_USE_LONGNAME was set to false . This resulted in cluster as well as rabbitmqctl working OK.

Is this expected behaviour since it’s at odds with the guide? From our case it seems that FQDN do not work with rabbitmqctl , guessing there is a mismatch between current hostname and what rabbitmqctl expects.

The text was updated successfully, but these errors were encountered:

Источник

Error unable to perform an operation on node rabbitmq

When building a rabbitmq cluster, when adding memory nodes, an exception is thrown: Error: unable TO perform an operation ON node ‘[email protected]’. Please see diagnostics information AND suggestions below.

First of all, suppose the current server is A, we execute on A server rabbitmqctl join_cluster [email protected] —ram, which means that we want to add server A to B as a memory node, and then execute this command to report the above error

To sum up, this problem can be solved according to the following ideas:

1. Check whether the .erlang.cookie file in the rabbitmq home directory of the two servers A and B exists and whether the content is the same (use the cat command output). In addition, if possible, change the .erlang.cookie file in the user’s home directory Set to sum The .erlang.cookie file in the rabbitmq home directory Consistent

Note: This is the rabbitmq home directory. To be precise, it is not. Because rabbitmq starts up, you need to set the HOME environment variable. Generally, you add the HOME environment variable in the rabbitmq-env file. If you don’t set the HOME variable, an exception may be thrown erlexec: HOME must be set

This should be to check the directory specified by the HOME variable configured in rabbitmq-env on the two servers A and B. Erlang.cookie exists and the content is consistent, and we generally point this HOME variable to the root directory of rabbitmq

2、 rabbitmqctl join_cluster [email protected] —ram It seems that B in the command can only be host, not IP, so you need to add the corresponding IP—host mapping in /etc/hosts, after adding, see if you can ping through

3. Restart rabbitmq related processes in server A and B:

4. First suspend the application in A, then specify join_cluster, if the execution is successful, then start the application, execute the following command in A

Источник

rabbitmqctl’s erlang cookie is not the same as $HOME/.erlang.cookie #252

TL;DR

• See this doc section on the Erlang cookie
• See this rabbitmq-users thread where this has been answered
• If you don’t mean to set the cookie via RABBITMQ_ERLANG_COOKIE (not the most secure option), make sure that env variable is not set since CLI tools will pick it up. Use the $HOME/.erlang.cookie file with 400 permissions or the —erlang-cookie switch.

Original Question

I’ve set docker container’s environment variable HOME to /var/lib/rabbitmq 。
After rabbitmq started, I could see its cookie hash like this :

however, when I execute command rabbitmqctl -l status ， it gave me following error:

It seems that rabbitmqctl didn’t read erlang cookie from $HOME/.erlang.cookie (i.e. /var/lib/rabbitmq/.erlang.cookie).
when I specify the erlang cookie explicitly and execute rabbitmqctl -l —erlang-cookie $(cat $HOME/.erlang.cookie) status ， it gives me normal correct output.

I suspect this is a severe BUG

The text was updated successfully, but these errors were encountered:

rabbitmq-server version is 3.7.5

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team).

We get at least a dozen of questions through various venues every single day, often light on details.
At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because GitHub is a tool our team uses heavily nearly every day, the signal/noise ratio of issues is something we care about a lot.

Источник

Bit of a background to the situation:

• two AWS EC2 nodes, each running a docker container with rabbitmq (image is based on https://github.com/docker-library/rabbitmq/blob/master/management/Dockerfile)
• cluster setup (and working)
• firewall free — ports (amqp, erland dist, empd and management) bound to host
• rabbitmq containers had their hostname set to FQDN such as ip-10-200-1-00.eu-west-1.compute.internal that resolve fine from each node

rabbitmq-server started OK, we then join the second node to the first one using FQDN:

rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@ip-10-200-2-00.eu-west-1.compute.internal
rabbitmqctl start_app

This works OK too and cluster is formed. However, when we attempt to use rabbitmqctl within either of these two containers, it wouldn’t work complaining erlang distribution failed. I’ve looked at https://www.rabbitmq.com/ec2.html and the suggestions about the hostname did not resolve our problem.
The response from rabbitmqctl:

root@ip-10-200-1-100:/# rabbitmqctl -n rabbit@ip-10-200-1-100.eu-west-1.compute.internal list_vhosts
Error: unable to connect to node 'rabbit@ip-10-200-1-100.eu-west-1.compute.internal': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@ip-10-200-1-100.eu-west-1.compute.internal']

rabbit@ip-10-200-1-100.eu-west-1.compute.internal:
  * connected to epmd (port 4379) on ip-10-200-1-100.eu-west-1.compute.internal
  * epmd reports node 'rabbit' running on port 25682
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?

current node details:
- node name: 'rabbitmq-cli-1045@ip-10-200-1-100'
- home dir: /var/lib/rabbitmq
- cookie hash: ***

Notice that current node details section lists node name in its short form even though in that case hostname was set to FQDN.

Thus, cluster works using full host names but rabbitmqctl fails. It failed to connect to the local and remote node, using FQDN as seen above as well as short hostname, giving the same message each time.

Since the hostnames are managed by AWS and short version easily resolvable, we decided to try using short hostnames for docker container hostname and join_cluster command while RABBITMQ_USE_LONGNAME was set to false. This resulted in cluster as well as rabbitmqctl working OK.

Is this expected behaviour since it’s at odds with the guide? From our case it seems that FQDN do not work with rabbitmqctl, guessing there is a mismatch between current hostname and what rabbitmqctl expects.

Недавно установленный rabbitmq не запустился, проверьте статус, появится следующая подсказка:

[[email protected] ~]# rabbitmqctl status
Status of node [email protected] ...
Error: unable to perform an operation on node '[email protected]'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on http://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node [email protected]

DIAGNOSTICS
===========

attempted to contact: [[email protected]]

[email protected]:
  * connected to epmd (port 4369) on localhost
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on localhost
  * suggestion: start the node

Current node details:
 * node name: '[email protected]'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: 8d5k0JKKV+xMlxN+LtpGjg==

Просмотреть журнал: /var/log/rabbitmq/log/crash.log

2019-01-25 23:19:00 =ERROR REPORT====
Failed to start Ranch listener {acceptor,{0,0,0,0,0,0,0,0},5672} in ranch_tcp:listen([{cacerts,'...'},{key,'...'},{cert,'...'},{port,5672},{ip,{0,0,0,0,0,0,0,0}},inet6,{backlog,128},{nodelay,true},{linger,{true,0}},{exit_on_close,false}]) for reason eaddrinuse (address already in use)
2019-01-25 23:19:00 =CRASH REPORT====
  crasher:
    initial call: supervisor:ranch_acceptors_sup/1
    pid: <0.350.0>
    registered_name: []
    exception exit: {{listen_error,{acceptor,{0,0,0,0,0,0,0,0},5672},eaddrinuse},[{ranch_acceptors_sup,listen_error,5,[{file,"src/ranch_acceptors_sup.erl"},{line,66}]},{ranch_acceptors_sup,init,1,[{file,"src/ranch_acceptors_sup.erl"},{line,44}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [<0.348.0>,<0.347.0>,rabbit_sup,<0.256.0>]
    message_queue_len: 0
    messages: []
    links: [<0.348.0>]
    dictionary: [{logger,error_logger}]
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 27
    reductions: 1618
  neighbours:
2019-01-25 23:19:00 =SUPERVISOR REPORT====
     Supervisor: {<0.348.0>,ranch_listener_sup}
     Context:    start_error
     Reason:     {listen_error,{acceptor,{0,0,0,0,0,0,0,0},5672},eaddrinuse}
     Offender:   [{pid,undefined},{id,ranch_acceptors_sup},{mfargs,{ranch_acceptors_sup,start_link,[{acceptor,{0,0,0,0,0,0,0,0},5672},ranch_tcp]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]

Отслеживаемый порт 5672 занят, справочная статья Baidu:https://yq.aliyun.com/ziliao/490972
Оказалось, что ActiveMQ был включен на моем компьютере, а RabbitMQ был перезапущен после выключения. Успех!

[[email protected] log]# rabbitmq-server start

  ##  ##
  ##  ##      RabbitMQ 3.7.10. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: /var/log/rabbitmq/[email protected]
                    /var/log/rabbitmq/[email protected]_upgrade.log

              Starting broker...
 completed with 3 plugins.

Статус просмотра:

[[email protected] ~]# rabbitmqctl status
Status of node [email protected] ...
[{pid,5828},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.7.10"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.10"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.10"},
      {rabbit,"RabbitMQ","3.7.10"},
      {os_mon,"CPO  CXC 138 46","2.4.7"},
      {amqp_client,"RabbitMQ AMQP Client","3.7.10"},
      {rabbit_common,
          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",
          "3.7.10"},
      {cowboy,"Small, fast, modern HTTP server.","2.6.1"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.7.1"},
      {ssl,"Erlang/OTP SSL application","9.1.2"},
      {public_key,"Public key infrastructure","1.6.4"},
      {asn1,"The Erlang ASN1 compiler version 5.0.8","5.0.8"},
      {inets,"INETS  CXC 138 49","7.0.4"},
      {recon,"Diagnostic tools for production use","2.3.6"},
      {mnesia,"MNESIA  CXC 138 12","4.15.5"},
      {jsx,"a streaming, evented json parsing toolkit","2.9.0"},
      {cowlib,"Support library for manipulating Web protocols.","2.7.0"},
      {crypto,"CRYPTO","4.4"},
      {xmerl,"XML parser","1.3.19"},
      {lager,"Erlang logging framework","3.6.5"},
      {goldrush,"Erlang event stream processor","0.1.9"},
      {compiler,"ERTS  CXC 138 10","7.3.1"},
      {syntax_tools,"Syntax tools","2.1.6"},
      {sasl,"SASL  CXC 138 11","3.3"},
      {stdlib,"ERTS  CXC 138 10","3.7"},
      {kernel,"ERTS  CXC 138 10","6.2"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 21 [erts-10.2.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [hipe]n"},
 {memory,
     [{connection_readers,0},
      {connection_writers,0},
      {connection_channels,0},
      {connection_other,2820},
      {queue_procs,0},
      {queue_slave_procs,0},
      {plugins,835316},
      {other_proc,25064708},
      {metrics,205316},
      {mgmt_db,153240},
      {mnesia,72864},
      {other_ets,2749152},
      {binary,158536},
      {msg_index,28912},
      {code,23353730},
      {atom,1172689},
      {other_system,9888797},
      {allocated_unused,8186432},
      {reserved_unallocated,1372160},
      {strategy,rss},
      {total,[{erlang,63686080},{rss,73244672},{allocated,71872512}]}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{http,15672,"::"}]},
 {vm_memory_calculation_strategy,rss},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,416348569},
 {disk_free_limit,50000000},
 {disk_free,24810864640},
 {file_descriptors,
     [{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},
 {processes,[{limit,1048576},{used,373}]},
 {run_queue,1},
 {uptime,52},
 {kernel,{net_ticktime,60}}]

　　 When building a rabbitmq cluster, when adding memory nodes, an exception is thrown:Error: unable TO perform an operation ON node ‘[email protected]’. Please see diagnostics information AND suggestions below.

　　 First of all, suppose the current server is A, we execute on A serverrabbitmqctl join_cluster [email protected] —ram, which means that we want to add server A to B as a memory node, and then execute this command to report the above error

　　To sum up, this problem can be solved according to the following ideas:

1. Check whether the .erlang.cookie file in the rabbitmq home directory of the two servers A and B exists and whether the content is the same (use the cat command output). In addition, if possible, change the .erlang.cookie file in the user’s home directory Set to sumThe .erlang.cookie file in the rabbitmq home directoryConsistent

　　Note: This is the rabbitmq home directory. To be precise, it is not. Because rabbitmq starts up, you need to set the HOME environment variable. Generally, you add the HOME environment variable in the rabbitmq-env file. If you don’t set the HOME variable, an exception may be thrown erlexec: HOME must be set

　　 This should be to check the directory specified by the HOME variable configured in rabbitmq-env on the two servers A and B. Erlang.cookie exists and the content is consistent, and we generally point this HOME variable to the root directory of rabbitmq

　　2、rabbitmqctl join_cluster [email protected] —ramIt seems that B in the command can only be host, not IP, so you need to add the corresponding IP—host mapping in /etc/hosts, after adding, see if you can ping through

　　3. Restart rabbitmq related processes in server A and B: 　　

　　#Find rabbitmq related processes
　　ps -ef | grep rabbitmq
 　　#Kill, you can use-9 Force kill
　　sudo kill 5760 5761 5917
 　　#Start rabbitmq, add one after the command&Means to start in the background
　　sudo rabbitmq-server &

　　4. First suspend the application in A, then specify join_cluster, if the execution is successful, then start the application, execute the following command in A

    # Stop application
　　sudo rabbitmqctl stop_app
 　　#Add A as a memory node to B,--The ram parameter means that it is a memory node, if this parameter is not taken, it is a disk node
　　sudo rabbitmqctl join_cluster [email protected] --ram
 　　# If the above command is executed successfully, start the application 
　　sudo rabbitmqctl start_app