Flower: Flower fails on inspect methods for both RabbitMQ & Redis backends

[lukik]

Hi,

I have an issue in using Flower to monitor celery. Almost identical to issue #395 but when all options there failed I tried multiple backends just to see if one of them was having an issue. So I've tried both Redis 2.8.4 and RabbitMQ 3.5.4. All of them fail with the errors as seen below

I have tried setting the --inspect_timeout option to 10000 -which looked like the chosen resolution there- but the error still comes up.

I started by using the latest celery-3.1.18 and flower-0.8.3 and when that failed I reverted to using celery-3.1.17 and flower-0.7.3 and those immediate previous versions -which I have working elsewhere- failed

Mind you celery_beat seems to be working as usual so looks like a flower thing

RabbitMQ Backend Error
[I 150825 08:21:58 mixins:225] Connected to amqp://guest:**@127.0.0.1:5672//
[W 150825 08:22:38 control:44] 'stats' inspect method failed
[W 150825 08:22:38 control:44] 'active_queues' inspect method failed
[W 150825 08:22:38 control:44] 'registered' inspect method failed
[W 150825 08:22:38 control:44] 'scheduled' inspect method failed
[W 150825 08:22:38 control:44] 'active' inspect method failed
[W 150825 08:22:38 control:44] 'reserved' inspect method failed
[W 150825 08:22:38 control:44] 'revoked' inspect method failed
[W 150825 08:22:38 control:44] 'conf' inspect method failed
[I 150825 08:24:43 web:1908] 200 GET /dashboard?updateinterval=30000 (10.0.2.2) 15.41ms
[I 150825 08:25:00 web:1908] 304 GET /broker (10.0.2.2) 23.93ms
[I 150825 08:25:04 web:1908] 304 GET /monitor (10.0.2.2) 13.40ms
[I 150825 08:25:04 web:1908] 200 GET /monitor/succeeded-tasks?lastquery=1440480337 (10.0.2.2) 1.56ms
[I 150825 08:25:04 web:1908] 200 GET /monitor/completion-time?lastquery=1440480337 (10.0.2.2) 1.04ms
[I 150825 08:25:04 web:1908] 200 GET /monitor/failed-tasks?lastquery=1440480337 (10.0.2.2) 0.99ms
[I 150825 08:25:04 web:1908] 304 GET /monitor/broker (10.0.2.2) 12.90ms
[I 150825 08:25:07 web:1908] 200 GET /tasks?limit=100 (10.0.2.2) 18.86ms
[I 150825 08:25:08 web:1908] 304 GET /dashboard (10.0.2.2) 14.87ms

Redis Backend Error
[I 150825 08:49:30 mixins:225] Connected to redis://localhost:6379/0
[W 150825 08:50:14 control:44] 'stats' inspect method failed
[W 150825 08:50:14 control:44] 'active_queues' inspect method failed
[W 150825 08:50:14 control:44] 'registered' inspect method failed
[W 150825 08:50:14 control:44] 'scheduled' inspect method failed
[W 150825 08:50:14 control:44] 'active' inspect method failed
[W 150825 08:50:14 control:44] 'reserved' inspect method failed
[W 150825 08:50:14 control:44] 'revoked' inspect method failed
[W 150825 08:50:14 control:44] 'conf' inspect method failed

flowerconfig
BROKER_URL = 'amqp://guest:guest@localhost:5672//'
broker_api = 'http://guest:guest@localhost:15672/api/'
port = 5555

Supervisor
[program:flower]
command = /opt/apps/venv/my_app/bin/celery flower --app=celery_config.celeryapp --conf=flowerconfig.py --inspect_timeout=10000
directory = /vagrant/apps/my_app/celery_config
user = www-data
autostart = true
autorestart = true
startsecs = 10
redirect_stderr = true
stderr_logfile = /var/log/celery/flower.err.log
stdout_logfile = /var/log/celery/flower.out.log

Even running directly from console i.e. flower -A celery_config.celeryapp --broker_api=http://guest:guest@localhost:15672/api also fails with the above errors

Images for both versions of flower used in this test
3.1.18

3.1.17

RabbitMQ and Redis were both installed using default settings.

[mher]

What is the output of celery inspect ping?

[lukik]

The output is: Error: No nodes replied within time constraint.

[mher]

This is not related to flower. Your workers don't reply to celery inspect commands.

[lukik]

Okay.. Thanks for the assistance. If I may ask, is that a settings thing in celery's configuration or somewhere else? I've checked on tasks and even in the monitoring and I can't seem to find a setting related to that. Thanks.

[mher]

It either connected with the load on workers or some setup/connectivity problem

[lukik]

You were right the issue was with my queues. A fabric file for deployment had missed some permissions and so they weren't running apart. For some reason the Scheduled ones were running which I mistakenly assumed that everything was right.

When that worked I ran into an error of 'Unable to get stats for 'celery@XXXX' worker but I resolved it using your guide in #438 to refresh the dashboard...

Thanks.