Rabbitmq-server: FQDN problem with rabbitmqctl

[jurajseffer]

Bit of a background to the situation:

• two AWS EC2 nodes, each running a docker container with rabbitmq (image is based on https://github.com/docker-library/rabbitmq/blob/master/management/Dockerfile)
• cluster setup (and working)
• firewall free - ports (amqp, erland dist, empd and management) bound to host
• rabbitmq containers had their hostname set to FQDN such as ip-10-200-1-00.eu-west-1.compute.internal that resolve fine from each node

rabbitmq-server started OK, we then join the second node to the first one using FQDN:

rabbitmqctl stop_app
rabbitmqctl join_cluster [email protected]
rabbitmqctl start_app

This works OK too and cluster is formed. However, when we attempt to use rabbitmqctl within either of these two containers, it wouldn't work complaining erlang distribution failed. I've looked at https://www.rabbitmq.com/ec2.html and the suggestions about the hostname did not resolve our problem.
The response from rabbitmqctl:

root@ip-10-200-1-100:/# rabbitmqctl -n [email protected] list_vhosts
Error: unable to connect to node '[email protected]': nodedown

DIAGNOSTICS
===========

attempted to contact: ['[email protected]']

[email protected]:
  * connected to epmd (port 4379) on ip-10-200-1-100.eu-west-1.compute.internal
  * epmd reports node 'rabbit' running on port 25682
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?

current node details:
- node name: 'rabbitmq-cli-1045@ip-10-200-1-100'
- home dir: /var/lib/rabbitmq
- cookie hash: ***

Notice that current node details section lists node name in its short form even though in that case hostname was set to FQDN.

Thus, cluster works using full host names but rabbitmqctl fails. It failed to connect to the local and remote node, using FQDN as seen above as well as short hostname, giving the same message each time.

Since the hostnames are managed by AWS and short version easily resolvable, we decided to try using short hostnames for docker container hostname and join_cluster command while RABBITMQ_USE_LONGNAME was set to false. This resulted in cluster as well as rabbitmqctl working OK.

Is this expected behaviour since it's at odds with the guide? From our case it seems that FQDN do not work with rabbitmqctl, guessing there is a mismatch between current hostname and what rabbitmqctl expects.

[michaelklishin]

Please ask questions on rabbitmq-users.

Yes, it is currently expected. Perhaps it'd be a good idea for rabbitmqctl to pay more attention to RABBITMQ_USE_LONGNAME. rabbitmqctl accepts the -n option which allows you to use any node/host name, however.

[jurajseffer]

Thanks Michael. It's not only a question but a report since the guide is at odds with what you say is expected. If you look at my output above you'll see that I used the -n option but it still failed.

[CVTJNII]

I'm hitting the same problem. Please reopen.

=INFO REPORT==== 18-Feb-2016::21:59:16 ===
node           : [email protected]
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config (not found)
cookie hash    : kxwbVsIOZbVw8c9RU5ItXQ==
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/[email protected]

# rabbitmqctl -n [email protected] list_users
Error: unable to connect to node '[email protected]': nodedown

DIAGNOSTICS
===========

attempted to contact: ['[email protected]']

[email protected]:
  * connected to epmd (port 4369) on trrabbitmq01.vagrant.tjnii.local
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?

current node details:
- node name: 'rabbitmq-cli-1562@trrabbitmq01'
- home dir: /var/lib/rabbitmq
- cookie hash: kxwbVsIOZbVw8c9RU5ItXQ==

Note that the cookie hashes match and the nodename passed matches the output from the rabbit log. As @jurajseffer pointed out rabbitmqctl is using the short name. Per http://erlang.org/faq/problems.html "you cannot have a system where some nodes use fully qualified names and others use short names".

[CVTJNII]

Setting RABBITMQ_USE_LONGNAME=true in the environment resolves this for me.

[michaelklishin]

CLI documentation now covers this case and 3.7.0 introduced --longnames.

[geokala]

@michaelklishin Have you considered adding --longnames to the suggestions in the error output, or is that something that a separate issue should be opened for?

[michaelklishin]

@geokala it's already the case as of https://github.com/rabbitmq/rabbitmq-cli/commit/08fbfe0dbf1465349ad6c0d592bba70e9549b58d:

rabbitmqctl status --erlang-cookie abc123
Status of node rabbit@warp10 ...
Error: unable to perform an operation on node 'rabbit@warp10'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@warp10
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: [rabbit@warp10]

rabbit@warp10:
  * connected to epmd (port 4369) on warp10
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
  * TCP connection succeeded but Erlang distribution failed

  * Authentication failed (rejected by the remote node), please check the Erlang cookie


Current node details:
 * node name: 'rabbitmqcli-15856-rabbit@warp10'
 * effective user's home directory: /Users/antares
 * Erlang cookie hash: 6ZoYxCjLONXyYIU2eJIuAw==

[michaelklishin]

…and https://github.com/rabbitmq/rabbitmq-cli/pull/312 was included into 3.7.12.