Flower: Connectivity problems: "missed heartbeats" and "Failed to capture events"

[robertour]

I have been chasing an error message for days thinking it was a problem in my tasks (using Celery with RabbitMQ).

I am getting every 3~4 minutes the missed heartbeats from client, timeout: 60s message in the RabbitMQ logs:

=ERROR REPORT==== 24-Jan-2019::14:03:07 ===
closing AMQP connection <0.10540.0> (127.0.0.1:52352 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 24-Jan-2019::14:06:07 ===
closing AMQP connection <0.10524.0> (127.0.0.1:52348 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s

=INFO REPORT==== 24-Jan-2019::14:06:08 ===
accepting AMQP connection <0.15037.0> (127.0.0.1:58596 -> 127.0.0.1:5672)

A few more INFO REPORT here ...

=INFO REPORT==== 24-Jan-2019::14:06:09 ===
connection <0.15103.0> (127.0.0.1:58618 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

=ERROR REPORT==== 24-Jan-2019::14:09:09 ===
closing AMQP connection <0.15076.0> (127.0.0.1:58614 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 24-Jan-2019::14:09:09 ===
closing AMQP connection <0.15103.0> (127.0.0.1:58618 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s

At some point, I just decided to turn off Flower, and the error stopped appear. Turn it on, and appear it again. So I went to the Flower log (never really look at it before), and also found the error Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.. Here is the log:

[E 190124 13:52:01 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 13:52:01 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 13:52:02 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 13:52:02 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 13:52:02 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 13:52:02 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 13:52:03 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 13:52:03 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 13:55:03 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 13:55:03 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 13:55:04 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 13:55:04 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 13:55:04 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 13:55:04 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 13:55:05 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 13:55:05 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 14:00:05 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 14:00:05 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 14:00:06 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 14:00:06 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 14:00:06 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 14:00:06 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 14:00:07 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 14:00:07 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 14:06:07 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 14:06:07 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 14:06:08 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 14:06:08 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[E 190124 14:06:08 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[E 190124 14:06:08 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190124 14:06:09 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//
[I 190124 14:06:09 mixins:224] Connected to amqp://guest:**@127.0.0.1:5672//

It is also weird that it is consistently failing twice, and then succeding twice (could it be a double report?).
Any ideas what could this be?

Here are some important packages in my environment:

amqp==2.3.2
celery==4.2.1
Django==1.11.16
flower==0.9.2

Also the version of my RabbitMQ is 3.6.9.

I would like to fix this or, at least, hide it so I don't get false alarms when I look into the logs.

[peter-slovak]

I'm observing the same behaviour with:

flower==0.9.2
celery==4.1.0
RabbitMQ 3.6.15

I tried fiddling with broker_heartbeat value and specifying multiple addresses in broker_url (RabbitMQ is clustered) but it seems the lower the value of broker_heartbeat, the more often I get the error.

I also tried to set broker_pool_limit to None, which should turn off connection pooling and establish a new connection to the MQ each time. Oddly enough, this brought the error count to a half - now I don't get two errors in a time period like @robertour described, just one. So maybe this solves a part of the problem.

FWIW, both Flower and RabbitMQ are running in Docker Swarm in my setup.

[jonaslimads]

broker_heartbeat = 0 fixed this annoying error for me. broker_pool_limit = None or any other value didn't seem to impact this issue.

[peter-slovak]

Huh, it never occurred to me to turn off the heartbeat completely. Anyway, confirming that the error is gone now, thanks @jonaslimads .

[robertour]

Could you give some details on how/where to set the broker_heartbeat? Is this on the flower, celery or rabbitmq side? The last thing I want is affecting the processes.

[peter-slovak]

Sorry for the late response @robertour . The setting is modified in Flower configuration. Personally, I prefer configuring Flower via celery.config_from_object(obj), so in my case the broker_heartbeat is an attribute of obj. But I believe you can just as easily put the option into the default flowerconfig.py file.

[joaodlf]

I have changed my broker from Redis to Rabbitmq and I am also seeing these errors in my flower log.

Flower itself appears to be working correctly. Logs are polluted with:

[I 190409 15:56:09 mixins:229] Connected to amqp://user:**@host:5672/user
[E 190409 15:59:09 events:123] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.

[nk2017ffm]

Logging at an "error" level while retrying seems excessive? Looking at the code, the try interval is always duplicated and flower will just keep trying forever, maybe there should be a configurable timeout when this should be logged as an error?