Rabbitmq: ECONNRESET on first connection

[ath88]

When starting a container from the image rabbitmq:3-management, the first connection I start will result in a ECONNRESET.

Apparently, the container starts accepting connections, but throws them away. Trying to reconnect will work fine, whereafter there is no problem.

From my perspective, its fine not to accept connections from the very start, and allow some start-up time. But once connections are accepted, they should work. :)

There is nothing revealing in the docker log.

[michaelklishin]

What evidence do you have that it's the "container that starts accepting connections but throws them away"?

Have you checked server logs, did a traffic capture with tcpdump and analyzed it or is it a guess?

RabbitMQ nodes take some time to start and protocol listeners are started towards the end, in particular for plugins such as STOMP or MQTT.

[ath88]

It is a guess, yes. I was hoping that someone might recognize the problem. :) If noone recognizes it, Ill gather some data and return.

I merely see my connections failing, when trying to connect, initially.

[michaelklishin]

@ath88 please collect data before opening issues, not after. If I was a maintainer of this repo this issue would be already closed as I don't have the time to take guesses.

Server logs are mandatory to suggest anything here, a traffic capture might be necessary as well depending on what the logs reveal.

[michaelklishin]

Aslo, a way to reproduce would be great because "initially" is not particularly specific. Please help others help you.

[ath88]

Sure thing! :) Closing the issue with a reference to the 'guidelines' would be fine, as I did not manage to find them.

A short example in Node.js:

const amqp = require("amqplib/callback_api");

amqp.connect("amqp://localhost:5672", function(error) {
    if(error) return console.error(error);
    console.info("Connection established");
});

Running the script in a bash while-loop will result in a series of ECONNREFUSED when there is nothing to connect to. I then start a RabbitMQ container with docker run -d --rm -p 5672:5672 rabbitmq. Now I receive a single:

{ Error: read ECONNRESET
    at exports._errnoException (util.js:1018:11)
    at TCP.onread (net.js:572:26) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }

.. then a series of:

Error: Socket closed abruptly during opening handshake
    at Socket.endWhileOpening (/home/ath88/test/node_modules/amqplib/lib/connection.js:258:17)
    at emitNone (events.js:91:20)
    at Socket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

.. until I get a "Connection established" and the script doesn't terminate, as expected.

As previously mentioned, docker logs reveals nothing, as far as I can see: https://pastebin.com/iNbwPD5J

[ath88]

Also, a wireshark-trace from when I start the RabbitMQ-container:

[ath88]

@michaelklishin Was this sufficient information, or are there any other things I could add? Is there any way to get a more verbose server log from the container?

[michaelklishin]

There are no rejected connections in the RabbitMQ log. There are TCP reset (refused connection, connection close) frames sent back when a client tries to connect ot port 5672 and that happens multiple times. Wireshark timestamps are relative so I cannot correlate them to those in the RabbitMQ node log but it lasts for about 2.5 seconds, which is even less than it often takes a node to boot.

So, in all likelihood the node simply isn't fully booted (and thus there are no TCP listeners on port 5672) by the time your app tries to connect. There is nothing this image or RabbitMQ can do about that. Make your apps wait for, say, 5-10 seconds before attempting to connect or simply make them handle connection failures.

[michaelklishin]

@tianon can you think of a Docker-specific reason why TCP connections can be refused for ~ 2.5 seconds but not after that?

[tianon]

It's _possible_ that there was some kind of delay between Docker starting the container and getting iptables rules in place to route traffic, but IIRC those get set up before the container is even launched, so that seems really unlikely. If the traffic is trying to go to localhost, it could be something due to docker-proxy (which is a separate process used for hairpin loopback connections), and it having a startup delay, especially if the box has heavy CPU contention.

I'd recommend not using localhost and instead using the address of the container itself if possible (for example, in your Node container, use --link or a Docker network so that DNS can be used to locate the other container somewhat more dynamically).

[kyrcha]

Like @tianon mentioned using the address of the container (docker inspect <hash> | grep IPAddress) worked like this:

const amqp = require("amqplib/callback_api");

amqp.connect("amqp://172.17.0.2:5672", function(error) {
    if(error) return console.error(error);
    console.info("Connection established");
});`

or mapping 5672 port:

docker run -d --hostname my-rabbit --name some-rabbit-mgt -p 8090:15672 -p 5672:5672 rabbitmq:3-management

worked with using localhost:

const amqp = require("amqplib/callback_api");

amqp.connect("amqp://localhost:5672", function(error) {
    if(error) return console.error(error);
    console.info("Connection established");
});

[tianon]

Closing, since this appears resolved. :+1: