Flower: Timeout when trying to view worker tab

[cyrenity]

I get a "Gateway Timeout (Error 504)" whenever i try to open a worker in flower. Have not found any error in debug log.

When i try to access flower API /api/workers it comes up with a blank json response. Here is my platform version

1.0.0-dev 4.1.0 (latentcall) flower -> flower:1.0.0-dev tornado:5.0 babel:2.5.3 software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.12 billiard:3.5.0.3 py-amqp:2.2.2 platform -> system:Linux arch:64bit, ELF imp:CPython loader -> celery.loaders.app.AppLoader settings -> transport:amqp results:disabled

Here is how i am starting celery worker and flower daemon

/opt/project/app/bin/celery -A tasks worker -B -E -l info --concurrency=5 -Q celery

/opt/project/app/bin/flower -A tasks --natural-time --url_prefix=flower --address=127.0.0.1 --port=5555 --inspect --enable-events

I tried installing on a fresh ubuntu server with same results :(

any help would be really appreciated.

[cyrenity]

installing torando version 4.5.2 fixed the issue

pip install torando==4.5.2

[ravitdx]

Hi,
Even changing the tornado to 4.5.2 doesn't solve my problem. I cannot open flower dashboard itself.
flower==0.9.2 tornado==4.5.2 babel==2.5.3 and celery==4.1.1

I don't use any Virtual env. My command in supervisor is

[program:celery-flower]
command=flower -A lbb --natural-time --url_prefix=flower --port=5555 --inspect --enable-events

We use uwsgi. Our application is based on django

server_tokens off;

upstream application {
server lbb-app:8000;
}

upstream application2 {
server lbb-app:5555;
}
server {
listen 80 default_server;

charset utf-8;

client_max_body_size 4G;
server_tokens off;

add_header X-Content-Type-Options nosniff;

add_header X-XSS-Protection "1; mode=block";

add_header Cache-Control "no-store";

add_header Pragma "no-cache";

add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";

location /work {
include uwsgi_params;
uwsgi_pass application;
}

location / {
include uwsgi_params;
uwsgi_pass application;
}
location /flower/ {
include uwsgi_params;
uwsgi_pass application2;
}
}

if I use proxy_pass instead I get 502. We deploy this on AWS.

when I type our URL it shows up normally. If I try to go to <website_url>/flower/ it shows 504 error. Where am I doing wrong. Is --broker_api compulsory for flower dashboard to appear.

logs from celery-flower-stderr:

[I 180605 08:25:06 command:106] Visit me at http://localhost:5555
[I 180605 08:25:06 command:107] Broker: amqp://guest:**@ec2-34-240-112-43.eu-west-1.compute.amazonaws.com:5672//
[W 180605 08:25:06 state:74] Broker info is not available if --broker_api option is not configured. Also make sure RabbitMQ Management Plugin is enabled (rabbitmq-plugins enable rabbitmq_management)
[I 180605 08:25:06 mixins:224] Connected to amqp://guest:**@ec2-34-240-112-43.eu-west-1.compute.amazonaws.com:5672//

please help me out.

[ravitdx]

Now I added thebroker-api=http://guest:**@ec2-xx-2xx-1xx-xx.eu-west-1.compute.amazonaws.com:15672/api/

And rabbitmq management command

now I am getting this.

[I 180606 03:56:15 command:139] Visit me at http://localhost:5555
[I 180606 03:56:15 command:144] Broker: amqp://guest:**@ec2-xx-2xx-1xx-xx.eu-west-1.compute.amazonaws.com:5672//
[I 180606 03:56:15 command:147] Registered tasks: 
    ['XXXXXXXXX',
     'XXXXXXXXXXXXXXXXXXXX',
     'TaskExporXXXXXXXXXX',
     'TaskEXXXXXXXXXXXXX',
     'celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap',
     'celery_haystack.tasks.CeleryHaystackSignalHandler',
     'celery_haystack.tasks.CeleryHaystackUpdateIndex',
     'lxxxxxxxxxxxxxxxxxxxxxx',
     'lxxxxxxxxxxxxxx',
     'lxxxxxxxxxxxxxxxxxxxx',
     'lXXXXXXXXXXXXX']
[I 180606 03:56:15 mixins:224] Connected to amqp://guest:**@ec2-xx-2xx-1xx-xx.eu-west-1.compute.amazonaws.com:5672//
[W 180606 03:56:17 control:44] 'stats' inspect method failed
[W 180606 03:56:17 control:44] 'active_queues' inspect method failed
[W 180606 03:56:17 control:44] 'registered' inspect method failed
[W 180606 03:56:17 control:44] 'scheduled' inspect method failed
[W 180606 03:56:17 control:44] 'active' inspect method failed
[W 180606 03:56:17 control:44] 'reserved' inspect method failed
[W 180606 03:56:17 control:44] 'revoked' inspect method failed
[W 180606 03:56:17 control:44] 'conf' inspect method failed

But I still cannot open the dash board. it still show 504 error.

[axiaoxin]

+1
tornado 5.0.2
celery 4.2.0
flower 0.9.2

[burgosa]

+1
celery==4.1.0
amqp==2.2.2
flower==0.9.2

[ZhenningLang]

I have no ideas what causes this, but I solved this problem by modification of api/control.py/ControlHandler.update_workers(from async to sync)

The origin file:

class ControlHandler(BaseHandler):
    # ...
    @classmethod
    @gen.coroutine
    def update_workers(cls, app, workername=None):
        logger.debug("Updating %s worker's cache...", workername or 'all')

        futures = []
        destination = [workername] if workername else None
        timeout = app.options.inspect_timeout / 1000.0
        inspect = app.capp.control.inspect(
            timeout=timeout, destination=destination)
        for method in cls.INSPECT_METHODS:
            futures.append(app.delay(getattr(inspect, method)))

        results = yield futures  # <--- stucked

        for i, result in enumerate(results):
            if result is None:
                logger.warning("'%s' inspect method failed",
                               cls.INSPECT_METHODS[i])
                continue
            for worker, response in result.items():
                if response is not None:
                    info = cls.worker_cache[worker]
                    info[cls.INSPECT_METHODS[i]] = response
                    info['timestamp'] = time.time()
    # ...

My version:

class ControlHandler(BaseHandler):
    # ...
    @classmethod
    @gen.coroutine
    def update_workers(cls, app, workername=None):
        logger.debug("Updating %s worker's cache...", workername or 'all')

        results = []
        destination = [workername] if workername else None
        timeout = app.options.inspect_timeout / 1000.0
        inspect = app.capp.control.inspect(
            timeout=timeout, destination=destination)
        for method in cls.INSPECT_METHODS:
            results.append(getattr(inspect, method)())

        for i, result in enumerate(results):
            if result is None:
                logger.warning("'%s' inspect method failed",
                               cls.INSPECT_METHODS[i])
                continue
            for worker, response in result.items():
                if response is not None:
                    info = cls.worker_cache[worker]
                    info[cls.INSPECT_METHODS[i]] = response
                    info['timestamp'] = time.time()
    # ...

[mckdev]

We are also having this issue. Flower is working well in general, it sees the queues, workers, processed tasks, graphs are working etc... Only when clicking the worker it timeouts with 504.

celery = 4.2.0
pyramid-celery = 3.0.0
redis = 3.4.1
flower = 0.9.4
kombu = 4.6.7

The only solution that works is what @ZhenningLang posted above. After implementing his change the worker page opens and all the information seems to be there.

Downgrading Tornado to 4.5.3 fixes this, but then there are other issues, because latest Flower requires Tornado>=5.0.0, so this is not an option.

Any chance of getting this fixed?

[mckdev]

I have created a patch with solution shared by @ZhenningLang and made it switchable with an option async_inspect, which can be set to False to do the worker inspect synchronously as a workaround fix.

It's not pretty, but it solves our problem for the time being. Feel free to use it:
https://github.com/celeraone/flower/commit/5ac2ba84b7611d9fb1307f02f90b8153f258e88e

[mher]

The latest version added a timeout for the update requests. You can also increase the timeout with --inspect-timeout option