Flower: Flower connect to RabbitMQ but fail on inspect methods

[lendoly]

Hello,

I have one machine with RabbitMQ and Celery, I'm trying to put Flower on it to monitorize the tasks, the problem is when I try to start the flower with the command:

celery flower -A myproj

it connect correctly to Rabbit:

[I 150519 14:09:54 command:114] Visit me at http://localhost:5555
[I 150519 14:09:54 command:116] Broker: amqp://user:@localhost:5672//
[I 150519 14:09:54 command:119] Registered tasks:
...
[I 150519 14:09:54 mixins:225] Connected to amqp://user:@localhost:5672//

the porblem is the inspect method failed:

[W 150519 14:09:56 control:43] 'stats' inspect method failed
[W 150519 14:09:56 control:43] 'active_queues' inspect method failed
[W 150519 14:09:56 control:43] 'registered' inspect method failed
[W 150519 14:09:56 control:43] 'scheduled' inspect method failed
[W 150519 14:09:56 control:43] 'active' inspect method failed
[W 150519 14:09:56 control:43] 'reserved' inspect method failed
[W 150519 14:09:56 control:43] 'revoked' inspect method failed
[W 150519 14:09:56 control:43] 'conf' inspect method failed

So the management web is empty.

[aliasmrchips]

Same issue... but with redis as the broker.

flower==0.8.2

[mher]

What is the output of celery -A myproj inspect ping command?

[aliasmrchips]

Error: No nodes replied within time constraint.

[lendoly]

same error as @aliasmrchips

[mher]

Error: No nodes replied within time constraint. means no worker replies to queries. It can happen if the workers are busy. You can try to increase the timeout (--inspect_timeout option in flower and celery inspect -t with celery).

[aliasmrchips]

Sorry, no worker was running before... If I start a worker, I get:

-> celery@hostname: OK
pong

So, the problem seems to be starting flower before the worker(s). If I start a worker and then start flower, it works as expected. If I shut down both celery worker(s) and flower and then start flower _before_ the celery worker, I get:

Unknown worker 'celery@hostname'

I believe this is the case whether or not it is a previously seen worker or a new one. In either case, I cannot inspect the workers if they are started after flower is running. Is this correct?

[mher]

There is a Refresh button on the dashboard page. It refreshes workers by resending inspect command. If you launch workers after flower just refresh workers.

[aliasmrchips]

Ah... Thanks! This fixed the "issue" for me. That is, it was not really an issue at all. Anyway, for anyone else who may experience this: The Refresh button is a choice under the drop down menu that has Shut Down selected by default. In order to Refresh, you have to select a worker (or all of them) first.

[lendoly]

Ok with the --inspect_timeout now is working correctly. Thanks

[nguaman]

I have the same issue, but i cant fixit.
anyone can paste the full celery command related of how this could work.

this is the result of my inspect ping
celery -A tasks inspect ping
Error: No nodes replied within time constraint.

[darkpixel]

I appear to be running into the same/similar issue. There is no 'refresh' on the dashboard tab, but it does show my worker, and that it is online.

Clicking on the worker gives Unknown worker 'celery@c98ff5d2e475'.

I'm using RabbitMQ for my broker and Redis for my results backend.

Jobs run perfectly, I just can't use flower to look at a worker.

09:49:09 [aaron@praxis:~/code/uitintranet] [uitintranet] develop ± pip freeze | grep -i flower
flower==0.8.2
09:50:45 [aaron@praxis:~/code/uitintranet] [uitintranet] develop ± DATABASE_URL=postgres://uitintranet:[email protected]/uitintranet PRODUCTION=True BROKER_URL=amqp://intranet:[email protected]:5672/intranet REDIS_URL=redis://tetrad.utilizeit.com:6379/0 python manage.py celery inspect ping
-> celery@c98ff5d2e475: OK
        pong
09:51:04 [aaron@praxis:~/code/uitintranet] [uitintranet] develop ± 

[jackyfkc]

@darkpixel The refresh button is on the dashboard page, workers' table. You will see a triangle button there.

[darkpixel]

It wasn't there. No worries. I ran into an old multithreading bug between celery and python and switched to python-rq last weekend. The dashboard isn't as nice. ;)

[asmodehn]

I had this issue as well.

When starting flower, many inspect methods failed.
As a result /api/workers returns empty.

I manually refreshed on the dashboard, then /api/workers returned the workers.

=> In a changing environment (workers going up and down ), it would be important to have flower periodically refreshing by itself, in order to get the updated workers data...
Is this currently doable ? Is this feature in a roadmap already ?

[tbindi]

There should be a way to gracefully say that it was not able to see any workers when the Flower started to avoid method failed messages that don't send out the right message for any developer.

[delijati]

I got the same error with python3.5 ... I had to remove the --autoreload

[riyadparvez]

+1, same issue.

[Xepe]

Hello,

I was working with celery 3.1.24, I reviewed my packages and I found that I was using kombu 3.0.36, I downgraded it to previous version 3.0.35 and with that I solved my issue.

Hope this help.

[fatpo]

i gei it...

cmd > celery -A task worker
another cmd > celery flower

it works....

[imomaliev]

https://stackoverflow.com/a/54570067/3627387 reposting my answer here

Problem here that, flower starts before celery is ready

This could be easily checked with celery inspect ping

Here is example from my project start_flower.sh

#!/bin/sh

until timeout -t 10 celery -A project inspect ping; do
    >&2 echo "Celery workers not available"
done

echo 'Starting flower'
celery -A project flower