(EDIT) Thanks a ton for flower, by the way, really appreciate all the hard work!
I got this page when attempting to view a worker at this URL:
http://example.com:5555/worker/celery@raynor
Traceback:
Unknown Celery version
Traceback (most recent call last):
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/web.py", line 1415, in _execute
result = yield result
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/gen.py", line 870, in run
value = future.result()
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/gen.py", line 879, in run
yielded = self.gen.send(value)
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/flower/views/workers.py", line 34, in get
self.render("worker.html", worker=dict(worker, name=name))
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/flower/views/__init__.py", line 21, in render
super(BaseHandler, self).render(*args, **kwargs)
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/web.py", line 704, in render
html = self.render_string(template_name, **kwargs)
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/web.py", line 811, in render_string
return t.generate(**namespace)
File "/home/azureuser/codalab-competitions/venv/local/lib/python2.7/site-packages/tornado/template.py", line 278, in generate
return execute()
File "worker_html.generated.py", line 264, in _tt_execute
_tt_tmp = task['name'] # worker.html:202 (via base.html:55)
TypeError: string indices must be integers
ckcollab
All 8 comments
Same for me
Python 3.5.2
flower==0.9.1
belibak
on 13 Feb 2017
same.
Python 2.7.6
flower==0.9.1
1905
on 22 Feb 2017
I'm not able to reproduce this issue with the versions you mentioned. Can you run celery inspect active when you see the error?
mher
on 19 Mar 2017
I have a slightly more convoluted configuration, namely:
Celery 3.1.24 running w/ python 3.5.2 (several queues)
Flower 0.9.1 running on _another host_ w/ python 2.7.12
both use RabbitMQ broker which runs on its own host, rabbitmq version 3.5.7-1ubuntu0.16.04.1 (packaged for Ubuntu, naturally)
exactly the same error and traceback
celery inspect active returns:
-> [email protected]: OK
- empty -
-> [email protected]: OK
- empty -
-> [email protected]: OK
- empty -
There are active jobs, as "Processed" counters increment, but they get completed rather quickly, so all three queues are empty most of the time
mbravorus
on 11 Jun 2017
To add to the above; upon further consideration, host running flower had celery 4.0.2, while the monitored host had celery pinned to 3.1.24 by requirements file.
After I manually downgraded celery to 3.1.24 on the flower host, the problem went away.
Perhaps that will help you pin down the problem.
mbravorus
on 11 Jun 2017
Got same issue
A simple debug code
--- control.py.orig 2017-09-01 17:34:01.000000000 +0300
+++ control.py 2017-09-01 17:44:55.751735808 +0300
@@ -37,6 +37,9 @@
futures.append(app.delay(getattr(inspect, method)))
results = yield futures
+ import sys
+ for r in results:
+ sys.stderr.write("r == {}\n".format(r))
for i, result in enumerate(results):
if result is None:
gives me the following:
r == {u'celery@debian': {u'clock': u'659404', u'pid': 30470, u'broker': {u'transport_options': {}, u'login_method': u'AMQPLAIN', u'hostname': u'10.10.10.10', u'userid': u'user', u'insist': False, u'connect_timeout': 4, u'ssl': False, u'virtual_host': u'vhost', u'heartbeat': None, u'uri_prefix': None, u'port': 5672, u'transport': u'amqp', u'alternates': []}, u'prefetch_count': 4, u'total': {u'task1': 175, u'task2': 10444, u'celery.backend_cleanup': 8, u'task3': 175, u'task4': 8}, u'rusage': {u'majflt': 0, u'nsignals': 0, u'minflt': 26176, u'maxrss': 77912, u'inblock': 0, u'nswap': 0, u'idrss': 0, u'msgrcv': 0, u'ixrss': 0, u'isrss': 0, u'nvcsw': 782711, u'stime': 48.636, u'oublock': 0, u'msgsnd': 0, u'nivcsw': 101270, u'utime': 453.752}, u'pool': {u'processes': [30475], u'max-concurrency': 1, u'timeouts': [0, 0], u'writes': {u'raw': u'10810', u'all': u'100.00%', u'total': 10810, u'avg': u'100.00%', u'inqueues': {u'active': 0, u'total': 1}}, u'put-guarded-by-semaphore': False, u'max-tasks-per-child': u'N/A'}}}
r == {u'celery@debian': [{u'exclusive': False, u'name': u'celery', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}]}
r == {u'celery@debian': {u'error': u"KeyError(u'registered',)"}}
r == {u'celery@debian': {u'error': u"KeyError(u'scheduled',)"}}
r == {u'celery@debian': {u'error': u"KeyError(u'active',)"}}
r == {u'celery@debian': {u'error': u"KeyError(u'reserved',)"}}
r == {u'celery@debian': {u'error': u"KeyError(u'revoked',)"}}
r == {u'celery@debian': {u'error': u"KeyError(u'conf',)"}}
Flower and celery versions:
% ./flower --version
1.0.0-dev
4.1.0 (latentcall)
flower -> flower:1.0.0-dev tornado:4.5.2 babel:2.5.0
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.9
billiard:3.5.0.3 py-amqp:2.2.1
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
celery 3.1.19
django-celery 3.1.17
serge-name
on 1 Sep 2017
Similar error here, steps to reproduce:
1) Flower is deployed using official Docker image, so it runs following versions of Python packages:
Package Version
---------- -------
amqp 2.3.2
Babel 2.6.0
billiard 3.5.0.3
celery 4.2.0
flower 0.9.2
kombu 4.2.1
pip 10.0.1
pytz 2018.5
redis 2.10.6
setuptools 39.2.0
tornado 5.0.2
vine 1.1.4
wheel 0.31.1
2) Celery for my worker (only one currently) is separate image/container, and version is 3.1.18.
3) RabbitMQ is used as a broker (with multiple queues) as third image, version is 3.7.6.
All three images are running in the same Docker compose.
Everything working fine, except when I try to see worker (as OP).
My stack trace is:
flower | [E 180719 16:32:16 web:1621] Uncaught exception GET /worker/celery@abb7a4b776c3 (172.18.0.1)
flower | HTTPServerRequest(protocol='http', host='localhost:5555', method='GET', uri='/worker/celery@abb7a4b776c3', version='HTTP/1.1', remote_ip='172.18.0.1')
flower | Traceback (most recent call last):
flower | File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1543, in _execute
flower | result = yield result
flower | File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1099, in run
flower | value = future.result()
flower | File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1113, in run
flower | yielded = self.gen.send(value)
flower | File "/usr/local/lib/python3.7/site-packages/flower/views/workers.py", line 34, in get
flower | self.render("worker.html", worker=dict(worker, name=name))
flower | File "/usr/local/lib/python3.7/site-packages/flower/views/__init__.py", line 21, in render
flower | super(BaseHandler, self).render(args, kwargs)
flower | File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 750, in render
flower | html = self.render_string(template_name, *kwargs)
flower | File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 891, in render_string
flower | return t.generate(namespace)
flower | File "/usr/local/lib/python3.7/site-packages/tornado/template.py", line 346, in generate
flower | return execute()
flower | File "worker_html.generated.py", line 264, in _tt_execute
flower | _tt_tmp = task['name'] # worker.html:212 (via base.html:55)
flower | TypeError: string indices must be integers
Stack trace from my worker:
allworkers_1 | [2018-07-19 09:32:16,144: DEBUG/MainProcess] pidbox received method stats() [reply_to:{u'routing_key': u'5addeb45-ac2b-3f7a-94a5-e9b100acf315', u'exchange': u'reply.celery.pidbox'} ticket:1d767243-f56b-49e3-92eb-7a793e545535]
allworkers_1 | [2018-07-19 09:32:16,148: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'b1b8571e-d504-330a-9f3e-d0ab5c3d1688', u'exchange': u'reply.celery.pidbox'} ticket:f284ca08-b45b-4758-9dcd-fc5936dbcb94]
allworkers_1 | [2018-07-19 09:32:16,159: DEBUG/MainProcess] pidbox received method registered(taskinfoitems=[]) [reply_to:{u'routing_key': u'094d9745-9e98-38ce-a4eb-ed68d7df92de', u'exchange': u'reply.celery.pidbox'} ticket:6e344e81-971d-44d1-8caf-df0780446831]
allworkers_1 | [2018-07-19 09:32:16,161: ERROR/MainProcess] pidbox command error: KeyError(u'registered',)
allworkers_1 | Traceback (most recent call last):
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 105, in dispatch
allworkers_1 | reply = handle(method, kwdict(arguments))
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 123, in handle_call
allworkers_1 | return self.handle(method, arguments)
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 120, in handle
allworkers_1 | return self.handlersmethod
allworkers_1 | KeyError: u'registered'
allworkers_1 | [2018-07-19 09:32:16,172: DEBUG/MainProcess] pidbox received method scheduled() [reply_to:{u'routing_key': u'7f8e5009-9f37-3c25-83c8-357c455faf75', u'exchange': u'reply.celery.pidbox'} ticket:0ba4a929-a1c9-4b79-9e63-fb010a8a1066]
allworkers_1 | [2018-07-19 09:32:16,172: ERROR/MainProcess] pidbox command error: KeyError(u'scheduled',)
allworkers_1 | Traceback (most recent call last):
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 105, in dispatch
allworkers_1 | reply = handle(method, kwdict(arguments))
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 123, in handle_call
allworkers_1 | return self.handle(method, arguments)
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 120, in handle
allworkers_1 | return self.handlersmethod
allworkers_1 | KeyError: u'scheduled'
allworkers_1 | [2018-07-19 09:32:16,206: DEBUG/MainProcess] pidbox received method reserved() [reply_to:{u'routing_key': u'b1b8571e-d504-330a-9f3e-d0ab5c3d1688', u'exchange': u'reply.celery.pidbox'} ticket:0b3f7a60-281f-4dc4-8784-7c5288633977]
allworkers_1 | [2018-07-19 09:32:16,207: ERROR/MainProcess] pidbox command error: KeyError(u'reserved',)
allworkers_1 | Traceback (most recent call last):
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 105, in dispatch
allworkers_1 | reply = handle(method, kwdict(arguments))
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 123, in handle_call
allworkers_1 | return self.handle(method, arguments)
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 120, in handle
allworkers_1 | return self.handlersmethod
allworkers_1 | KeyError: u'reserved'
allworkers_1 | [2018-07-19 09:32:16,211: DEBUG/MainProcess] pidbox received method revoked() [reply_to:{u'routing_key': u'094d9745-9e98-38ce-a4eb-ed68d7df92de', u'exchange': u'reply.celery.pidbox'} ticket:29bc2c4a-010f-4772-9243-8cbbc40bba50]
allworkers_1 | [2018-07-19 09:32:16,211: ERROR/MainProcess] pidbox command error: KeyError(u'revoked',)
allworkers_1 | Traceback (most recent call last):
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 105, in dispatch
allworkers_1 | reply = handle(method, kwdict(arguments))
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 123, in handle_call
allworkers_1 | return self.handle(method, arguments)
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 120, in handle
allworkers_1 | return self.handlersmethod
allworkers_1 | KeyError: u'revoked'
allworkers_1 | [2018-07-19 09:32:16,215: DEBUG/MainProcess] pidbox received method active() [reply_to:{u'routing_key': u'5addeb45-ac2b-3f7a-94a5-e9b100acf315', u'exchange': u'reply.celery.pidbox'} ticket:f7d1c263-c66d-4d77-b08a-935423d17316]
allworkers_1 | [2018-07-19 09:32:16,221: ERROR/MainProcess] pidbox command error: KeyError(u'active',)
allworkers_1 | Traceback (most recent call last):
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 105, in dispatch
allworkers_1 | reply = handle(method, kwdict(arguments))
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 123, in handle_call
allworkers_1 | return self.handle(method, arguments)
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 120, in handle
allworkers_1 | return self.handlersmethod
allworkers_1 | KeyError: u'active'
allworkers_1 | [2018-07-19 09:32:16,257: DEBUG/MainProcess] pidbox received method conf(with_defaults=False) [reply_to:{u'routing_key': u'7f8e5009-9f37-3c25-83c8-357c455faf75', u'exchange': u'reply.celery.pidbox'} ticket:51c1f6a9-e856-4c86-a04b-3c1276dd14f0]
allworkers_1 | [2018-07-19 09:32:16,258: ERROR/MainProcess] pidbox command error: KeyError(u'conf',)
allworkers_1 | Traceback (most recent call last):
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 105, in dispatch
allworkers_1 | reply = handle(method, kwdict(arguments))
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 123, in handle_call
allworkers_1 | return self.handle(method, arguments)
allworkers_1 | File "/usr/local/lib/python2.7/dist-packages/kombu/pidbox.py", line 120, in handle
allworkers_1 | return self.handlersmethod
allworkers_1 | KeyError: u'conf'
allworkers_1 | [2018-07-19 09:32:16,470: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
stojan-jovic
on 19 Jul 2018
woo thanks for fixing this @mher !
ckcollab
on 2 Jul 2020
Related issues
Xepe
路
4Comments
Darkheir
路
3Comments
arnpow
路
6Comments
Joe-Heffer-Shef
路
4Comments
ghost
路
3Comments
Most helpful comment
To add to the above; upon further consideration, host running flower had celery 4.0.2, while the monitored host had celery pinned to 3.1.24 by requirements file.
After I manually downgraded celery to 3.1.24 on the flower host, the problem went away.
Perhaps that will help you pin down the problem.