Flower: Flower not finding queue or workers when monitoring apps not on the same instance

[rschwiebert]

I am deploying Flower with Chef on a sentinel node.

I've been trying to start the flower service with path/to/bin/celery --broker=redis://brokerurl/0 --port=5555 flower

I can navigate to the Flower page just fine, but no tasks appear. When I go to the broker tab, no queues are listed.

What could be wrong? What else is needed?

[mher]

Have you checked the flower log?

[rschwiebert]

I had a look at the related supervisor logs in /var/log/supervisor/ but did not see anything amiss. Unless there is an obvious error message, I have no idea what I'm looking for. @mher Can you be more specific about what I should look for? Are there more logs I should be looking at not in /supervisor?

When I restart a particular Flower, this is what is generated in the stderr log:

[I 150728 12:42:04 command:106] Visit me at http://localhost:5555
[I 150728 12:42:04 command:107] Broker: brokername
[I 150728 12:42:04 mixins:225] Connected to brokername

[rschwiebert]

I found out that it wasn't because I am using --broker and not -A, so I modified the original issue title. I have tried to set up flowers with -A now, and I still can't the resulting flower to see queues and workers.

I built up three celery app objects and used them with three celery flower -A appname invocations to set up flowers on three different ports. These are supposed to report back on their respective redis instances.

The Flower webpages work perfectly, they just don't have any content. The flowers living natively with each application work just fine. Keep in mind my goal is to use a sentinel instance with Flower on it to monitor several other instances running their own disparate celery queues.

[jackyfkc]

Have you start some workers with the same broker?

[rschwiebert]

The working node monitors show the queue, workers, and tasks. @jackyfkc

[jackyfkc]

The broker queues displayed on UI are determined by the workers (queue_names = ControlHandler.get_active_queue_names()). So the root reason here is no workers (events) could be captured by flower.

@rschwiebert, to make sure that flower has indeed got events from workers, you could add a log logger.info('---> event: %s, worker: %s', event_type, worker_name) before line 42.

Also, you could start flower with --logging=debug to get a verbose log.

Hope it helps!

[rschwiebert]

@jackyfkc Thanks, the clue about the workers is very valuable.

Executing with --logging=debug is unfortunately hard given the deployment. Anyhow, I imagine the result of any logging experiment would be that the flower isn't receiving events from workers. We're focusing on finding a reason that the workers aren't talking to this instance. Any input you might have would be welcome.

One thing I've been trying to determine is what the flower really needs to run. Beyond the broker, does it need anything else?

[akoumjian]

Is this still an issue? We also experienced this about a year ago and I'm curious if it's still happening.

[mannysz]

Still experiencing this. Does someone solved the problem? In my case it shows all queues, except by the default queue.

[BuckinghamIO]

We really need a fix for this, also here is another ticket with relevent information: https://github.com/mher/flower/issues/253

[mannysz]

Just realized that it doesn't show queues with no scheduled tasks. Scheduling a single task for each queue worked for me. We definitely need a fix for that.

[schmod]

I also seem to be experiencing this on my end if I run flower as a standalone process (ie. separate from my queue workers, and without -A)

[ghost]

I'm having the same problem. Even though celery inspect active is showing all the workers as active. One of the worker is always showing offline.

[arianpasquali]

Same issue here. I can see workers and everything else, but nothing on tasks tab view.
Everything is working fine though

[trianglesis]

Happens to me right now.
Working as designed few months before this moment.
Now it shows only "Loading..." but workers are up and running and can be shown on my website, but not at Dashboard of Flower

[trianglesis]

But it show workers for a moment! And then they're disappear from webpage!

[trianglesis]

[eadmaster]

similar issue here, all the workers disappeared from the dashboard when i enabled a file logger in the setttings.py of my django app (but they are still working), may be this is somehow related to logging?

[paranoidi]

Same issue, installed on two supposedly identical environments. In one I can see workers and on other nothing. VERY difficulty to debug, even with debug logs nothing helpful is present.

[schmod]

I'll save you time. The code to do this using a Redis backend simply is not implemented. It doesn't exist. There's nothing to debug, and fixing it was decidedly non-trivial when I last looked into it...

[pizzapanther]

All my tasks queues were not showing up because 2 of my workers were running on the same machine with the same name. When I used the -n option everything started working correctly.

[michaeljonaitis]

I know that in this case the broker is Redis. But, I ran into a similar issue using a RabbitMQ cluster as the broker. In this case you specify two brokers, the "master" broker MUST be the fist on listed when you run the flower command. If you have the slave first and your worker server configs have the brokers also reversed the workers will properly process jobs but you will be unable to see them in flower.

flower --address=0.0.0.0 --port=5555 --broker='amqp://BROKERMASTER;amqp://BROKERSLAVE'

[giacomolicari]

Hi,
I'm experiencing the same issue when running flower docker container with Redis, while my celery tasks run correctly on a separate docker container.

I can't see the list of available tasks on Flower. However, once a task is executed it appears on the dashboard.

Below the log taken from Flower execution. As you can see only default or system tasks are detected:

[I 191115 12:52:47 command:147] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']

Is there a way or workaround to avoid running Flower on the same "context" as the python celery project and to get _registered tasks_ detected?

Thank you !

[mher]

Try to configure flower via celery application object with -A option. You can debug the setup with celery inspect -A command.

[Pastooh]

Hi,
I'm experiencing the same issue when running flower docker container with Redis, while my celery tasks run correctly on a separate docker container.

I can't see the list of available tasks on Flower. However, once a task is executed it appears on the dashboard.

Below the log taken from Flower execution. As you can see only default or system tasks are detected:

[I 191115 12:52:47 command:147] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']

Is there a way or workaround to avoid running Flower on the same "context" as the python celery project and to get _registered tasks_ detected?

Thank you !

Hi.
You cat get a JSON responce at :5555/api/workers and parse registered attr
Don`t forget to use --basic_auth )