celery: 3.1.23 (Cipater)
rabbitmq-server:3.2.4-1
i found some task send to rabbitmq and then celery worker get duplicate task .
function:
def do_backup(name):
....
for example:
server: do_backup.delay("/tmp/__36")
worker02:
[2016-06-23 20:38:27,182: INFO/MainProcess] Received task: tasks.do_backup[40159186-1b88-4d98-aabf-839bb4b7a852]
worker12:
[2016-06-23 20:36:21,647: INFO/MainProcess] Received task: tasks.do_backup[d265ffba-fa21-48b6-baad-2c0185a3168f]
........
i make sure the server only send a "/tmp/__36"
They don't seem to be duplicates as they have different task ids?
thanks for your reply, It is a fact that the same data (/tmp/__36) send to two works . I think the one task don't send ack in time, and so the rabbitmq send the this task to another one?
I have the same issue, I can see on the beat log that the task is only sent once but received twice by the worker a few seconds apart. When using CELERY_ENABLE_UTC=True
, the worker receive two tasks, one in UTC time and the other in local time.
I'm using Celery 3.1.23 (Cipater) with Redis 3.2.3.
Same Issue, but taskid is the same @ask @DurandA. Celery 3.1.23, Rabbit 3.6.5
I've also added some debug to verify that is ack -> https://github.com/cesterlizi/celery/blob/3daf891f7db10f3f4c6cd89da3c39cb637400b1f/celery/worker/job.py#L532-L534
late ack is enabled
[2016-10-07 12:08:25,875: INFO/MainProcess] Received task: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:25,876: DEBUG/MainProcess] TaskPool: Apply
[2016-10-07 12:08:25,876: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:25,876: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] pid:26289
......
[2016-10-07 12:08:26,587: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] return '0'
[2016-10-07 12:08:26,587: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:26,587: INFO/MainProcess] Task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] succeeded in 0.710850441828s: 0
[2016-10-07 12:08:27,504: INFO/MainProcess] Received task: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:32,776: DEBUG/MainProcess] TaskPool: Apply
[2016-10-07 12:08:40,273: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:40,273: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] pid:26286
....
[2016-10-07 12:08:40,286: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] return '1'
[2016-10-07 12:08:40,286: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:40,286: INFO/MainProcess] Task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] succeeded in 0.0134955914691s: 1
@ask I'am also attaching a log where yo can see that the task is duplicated and executed in parallel
[2016-10-07 12:13:25,966: INFO/MainProcess] Received task: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:25,966: DEBUG/MainProcess] TaskPool: Apply
[2016-10-07 12:13:25,967: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:25,967: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] pid:26287
[2016-10-07 12:13:27,464: INFO/MainProcess] Received task: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:27,464: DEBUG/MainProcess] TaskPool: Apply
[2016-10-07 12:13:27,465: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:27,465: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] pid:26291
[2016-10-07 12:13:28,723: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] return '0'
[2016-10-07 12:13:28,723: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:28,724: INFO/MainProcess] Task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] succeeded in 2.75686794892s: 0
[2016-10-07 12:13:28,912: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] return
Hi
I found that duplication is generated when I publish a task. I this case I'am sending the task to rabbit using JAVA rabbitmq API with message confirmation. For some reason (yet not understood) confirmation fails and I am sending the task again and two task message are enqueued to rabbitmq so two tasks are consumed and executed by celery worker
@ask dismiss my assertion in this issue
Carlos
Hi all
We've found same issue in several our projects when we used Redis as broker but seems it's not broker related problem. It's really looks like (we've checked the logs) any delayed task (ETA used, for example with countdown
argument) several workers pulls same task (with same Task ID). And I found a lot of posts in Internet that describes that situation but in different words (post, issue). And there is repository specifically for that case. May be that lock should be implemented at Celery level for any broker?
We using this our snippet to prevent such task behavior:
import datetime
from celery import Task
from celery.utils.log import get_task_logger
from django.conf import settings
from django.core.cache import get_cache
logger = get_task_logger(__name__)
# noinspection PyAbstractClass
class TaskWithLock(Task):
"""
Base task with lock to prevent multiple execution of tasks with ETA.
It's happens with multiple workers for tasks with any delay (countdown, ETA).
You may override cache backend by setting `CELERY_TASK_LOCK_CACHE` in your Django settings file
"""
abstract = True
lock_expiration = 60 * 60 * 24 # 1 day
cache = get_cache(getattr(settings, 'CELERY_TASK_LOCK_CACHE', 'default'))
@property
def lock_key(self):
"""Unique string for task as lock key"""
return '%s_%s' % (self.__class__.__name__, self.request.id)
def acquire_lock(self):
"""Set lock"""
result = self.cache.add(self.lock_key, True, self.lock_expiration)
logger.debug('Lock for %s at %s %s', self.request.id, datetime.datetime.now(), 'succeed' if result else 'failed')
return result
def release_lock(self):
"""Release lock"""
result = self.cache.delete(self.lock_key)
logger.debug('Lock release for %s at %s %s', self.request.id, datetime.datetime.now(), 'succeed' if result else 'failed')
return result
def retry(self, *args, **kwargs):
"""We need to release our lock to let the first process take current task in execution on retry"""
logger.debug('Retry requested %s, %s', args, kwargs)
self.release_lock()
return super(TaskWithLock, self).retry(*args, **kwargs)
def __call__(self, *args, **kwargs):
"""Checking for lock existence"""
if self.acquire_lock():
logger.debug('Task %s execution with lock started', self.request.id)
return super(TaskWithLock, self).__call__(*args, **kwargs)
logger.debug('Task %s skipped due lock detection', self.request.id)
Feel free to test and point to any errors in this snippet.
@Skyross I've been investigating a similar issue this morning as we are using Redis as our broker too. How long are your ETAs set to? We have multiple tasks with ETAs set anywhere in between 1-3 days. I suspect our issue (maybe yours as well) could possibly be this:
http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#redis-visibility-timeout
@andynguyen11, I am not familiar with Redis but duplicate tasks happen on other brokers for various reasons (see https://github.com/celery/celery/issues/2976) and this solution would help.
@Skyross - Thanks for this. I see your solution uses django cache, which works for me but I am thinking if we can find a more generic solution. How about checking the list of existing scheduled tasks, which one can retrieve by using celery.control.inspect().scheduled()
.
Skip if the task is already scheduled e.g.
scheduled = celery.control.inspect().scheduled()
if scheduled and self.request.id in [item['request']['id'] for sublist in celery.control.inspect().scheduled().values() for item in sublist]:
Also, are you using this for workflows or single tasks ?
Hi all again,
The previous version of my code contains at least one error: all tasks with retry
will be skipped too. I'm testing new version of that class. As you may see now we don't release lock but just waiting for its expiration.
Also I created public repository for that sandbox purpose (maybe later I'll move it to GitHub later if needed)
@andynguyen11 Yes, you were right! I came to the same reason. Anyway, sometimes we need to create delay further in the future than the visibility_timeout
value.
@topalhan Feel free to provide your version and compare performance :+1:
UPD: moved repo to https://gitlab.com/the-past/sandbox/Django-1.8.13-Celery-3.1.23
what's it's situation after 4.1/master?
Hi! I also encountered duplication problem.
We have ETA tasks for 1-4 hours ahead. In the attached log You can see, that task with id 06eaa034-0e0c-49e8-88ec-08279c871cb1
applied twice and executed twice (client device received two same push-notifications).
Already tried huge VISIBILITY_TIMEOUT.
2018-01-15T14:17:01.102524+00:00 app[worker.1]: [2018-01-15 14:17:01,102: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f05a9003598> (args:('notifications.event_update_tasks.send_event_update_notification', '06eaa034-0e0c-49e8-88ec-08279c871cb1', {'lang': 'py', 'task': 'notifications.event_update_tasks.send_event_update_notification', 'id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'eta': '2018-01-15T14:17:00.010508+00:00', 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'parent_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'argsrepr': '(15446, 163)', 'kwargsrepr': '{}', 'origin': 'gen11@e8457c44-4b6c-4d27-b317-9c6b7cc069d0', 'redelivered': True, 'reply_to': '6d7d0c3d-b5d2-3141-850f-6bf23ac825af', 'correlation_id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[15446, 163], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
2018-01-15T14:17:01.102886+00:00 app[worker.1]: [2018-01-15 14:17:01,102: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f05a9003598> (args:('notifications.event_update_tasks.send_event_update_notification', '06eaa034-0e0c-49e8-88ec-08279c871cb1', {'lang': 'py', 'task': 'notifications.event_update_tasks.send_event_update_notification', 'id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'eta': '2018-01-15T14:17:00.010508+00:00', 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'parent_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'argsrepr': '(15446, 163)', 'kwargsrepr': '{}', 'origin': 'gen11@e8457c44-4b6c-4d27-b317-9c6b7cc069d0', 'reply_to': '6d7d0c3d-b5d2-3141-850f-6bf23ac825af', 'correlation_id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[15446, 163], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
2018-01-15T14:17:01.103941+00:00 app[worker.1]: [2018-01-15 14:17:01,103: DEBUG/ForkPoolWorker-1] send_event_update_notification
2018-01-15T14:17:01.104265+00:00 app[worker.1]: [2018-01-15 14:17:01,104: DEBUG/ForkPoolWorker-3] send_event_update_notification
2018-01-15T14:17:01.104976+00:00 app[worker.1]: [2018-01-15 14:17:01,104: DEBUG/MainProcess] Task accepted: notifications.event_update_tasks.send_event_update_notification[255ce589-f8f7-4b7e-9d4f-aba88aa91038] pid:10
2018-01-15T14:17:01.105307+00:00 app[worker.1]: [2018-01-15 14:17:01,105: DEBUG/MainProcess] Task accepted: notifications.event_update_tasks.send_event_update_notification[06eaa034-0e0c-49e8-88ec-08279c871cb1] pid:12
2018-01-15T14:17:01.105661+00:00 app[worker.1]: [2018-01-15 14:17:01,105: DEBUG/MainProcess] Task accepted: notifications.event_update_tasks.send_event_update_notification[06eaa034-0e0c-49e8-88ec-08279c871cb1] pid:13
2018-01-15T14:17:01.106536+00:00 app[worker.1]: [2018-01-15 14:17:01,104: DEBUG/ForkPoolWorker-4] send_event_update_notification
having the same problem here, duplicate task_id sent to multiple workers within 10 secs. Is there any existing solution?
Moreover, in my situation the task has no countdown.
I'm having the same issue duplicate task with duplicate task on redis ...
i solve it with a decorator (but it's more a hack than something correct) based on https://stackoverflow.com/questions/26831103/avoiding-duplicate-tasks-in-celery-broker
from redis import Redis
from functools import wraps
def run_only_one_instance(lock_name):
def real_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
print("run only once " + lock_name + str(args) + str(kwargs))
try:
sentinel = Redis(host="redis").incr(lock_name + str(args) + str(kwargs))
if sentinel == 1:
# I am the legitimate running task
print("running func")
return func(*args, **kwargs)
else:
# Do you want to do something else on task duplicate?
print("run is already done escaping")
Redis(host="redis").decr(lock_name + str(args) + str(kwargs))
except Exception as e:
Redis(host="redis").decr(lock_name + str(args) + str(kwargs))
# potentially log error with Sentry?
# decrement the counter to insure tasks can run
# or: raise e
return wrapper
return real_decorator
hope it will help some people
in case someone need help to use it it's how i use it:
@app.task
@run_only_one_instance("name_"+"get_all_issues")
def get_all_issues():
EDIT i've forget the return on the wrapper part now it's corrected
can this be added in docs? or any patch in upstream?
If you are speaking about my code
Of course it can be added to docs or patch.
But wait till tomorow i will make a better version whithout hardcoded stuff and i may have found a way to remove the lock name variable and replace it with some data from function
Will do it asap :)
so here is a version that is not hardcoded (but it still work only with redis):
lock_name based on module name + function name + args +kwargs (module name is in the case you got two function with the same name in different module who need to run with the same redis)
redis server address is the only parameter and default to 'redis'
i've also added some exception logging with error from the standard logging lib
and finaly i've removed the print
from redis import Redis
from functools import wraps
from logging import error
def run_only_one_instance(redis_addr: str = "redis"):
def real_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
try:
sentinel = Redis(host=redis_addr).incr(func.__module__+func.__name__ + str(args) + str(kwargs))
if sentinel == 1:
return func(*args, **kwargs)
else:
pass
Redis(host=redis_addr).decr(func.__module__+func.__name__ + str(args) + str(kwargs))
except Exception as e:
Redis(host=redis_addr).decr(func.__module__+func.__name__ + str(args) + str(kwargs))
error(e)
return wrapper
return real_decorator
i've forgot to say it but actualy it work only on python3 (if you want python 2 support remove the ':str' on the definition of run_only_one_instance)
would you mind sending a PR on master?
done but first time I do a PR.
I hope I have done it correctly.
the file is inside celery/utils/
@tagashy why would lock based on args,kwargs and not based on task_id?
also, str(args) / str(kwargs) isn't very useful since if it has objects pickled it would fail to compare properly, I would suggest using hash(args)
@danield137
@tagashy why would lock based on args,kwargs and not based on task_id?
How do you access task_id? as my snippet was a wrapper i was thinking of using all the data available to make something like an uuid (not a real one of course) this way if you call foo(bar,toto=toto) and foo(bar,toto) they will not interfere with each other but if you know how to get the task_id it could be great too :)
also, str(args) / str(kwargs) isn't very useful since if it has objects pickled it would fail to compare properly, I would suggest using hash(args)
you are right hash would be better than str
i had discover hash func just recently so i had not think about it when writing this snippet
New here, just adding a note that we are seeing the same thing. Duplicate work. Running with Celery and Redis. We have high number of duplicate tasks being executed. Celery 4.1.0 and Redis 2.10.6
We have the same issue as @Runamok81 mentioned. Celery 4.1 with Redis.
here is my case, if anyone might need
My celery version:
3.1.25
My situation:
same task id sent to worker twice.
My solution:
clear redis data
I had just renamed the queue,
I had followed the code to kombu.Exchange.DirectExchange.deliver
, and I found that celery put a message to the old queue name (not exist in my code anymore) & new queue. Also the worker somehow received both (wired since there is a --queues parameter provided).
I guess Redis was keeping outdated queue config, so I just clear them all, since I won't change the queue name frequently
Hi, I'm experiencing same problem today - i have many tasks with task_id being duplicated about 40 times, i guess it's a consequence of task being retried many times.
celery 4.1.0, redis
here's the content of queue sync
by one task id:
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:12:25.727044+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "1227b338-5afd-4023-ac86-ea4f673560e4"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:17:11.319382+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "127128f4-e4b7-4b60-a6e7-f8b5eba0b282"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:20:49.547095+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "a66d51ab-fa26-4861-b74a-b54344a3ce8b"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:25:36.980849+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "623fa9db-53fb-4f02-8a71-e049a18777ae"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:32:56.648995+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "7e0da523-2ccb-4d55-b12a-218d11a7881f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:50:30.738049+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "efd0511a-7545-47e2-9748-ac2e57aa1c02"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:54:43.477435+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "2c023d9f-0455-453e-a040-c6c1a12dca0a"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:55:20.960713+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "68190480-dbad-4857-ba6b-cb508b9bb0a0"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:57:55.310463+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "0ee01ec6-945d-43eb-a21c-e23a64ada8a9"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:00:03.463454+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "e9bc1755-b086-4420-9934-9e8ed767f9a2"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:01:01.604583+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "7f2c116c-70da-45f7-92fb-1133ad6baf2f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:04:31.824229+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "b989b9b0-cdc4-4263-ad4f-7896c4e8ec31"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:07:32.695926+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "54072f13-e20c-4470-9b77-7482b87f8299"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:09:45.829355+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "04c3b6e7-656a-44a5-a377-dd1d08cc4fff"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:17:10.650890+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "df1dd1fa-7e37-42cf-9ce9-2ced878f153e"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:21:16.596645+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "94c82c7c-e0ac-4eaf-b295-ae8cf015625f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:27:23.403676+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "d12b6235-ef1f-4b5e-914c-01a4fc4510a8"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:27:33.804446+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "15f1a689-134e-4b38-873e-427ea5f0a7e9"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:29:47.491450+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "9c0725aa-6309-4b4b-8089-d42645f1b1dd"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:30:43.054268+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "38930a4e-3c3f-4413-ac73-df5a690c64a3"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:30:51.653718+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "ffb0c08d-698e-47d4-996f-c765dae15b6c"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:31:12.896793+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "ffba1969-c220-47fe-a80f-adf2bf8c9447"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:32:54.138908+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "b160b0ff-1184-4d3a-98c8-3cfa24e99f1c"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:33:00.545273+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "e4bb3b01-8a5b-4e0d-b4db-f5f54bc689c7"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:46:50.739562+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "3877e915-e9ec-4a02-a6a5-157c2c630de7"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:48:19.864291+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "c7037ac1-c182-4c86-8a90-2f53f0e5a177"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:55:55.877660+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "2eaef446-1a38-492f-a701-6861a8738d08"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:03:22.490942+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "867e611e-5fcf-401c-a2f5-88c88bec5ad9"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:37:34.111795+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "582067d4-ca18-4cc0-a14d-c110b82a9ccf"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:50:23.583620+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "3002dfd5-a1f3-47ca-bf09-8518783b61df"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:54:30.586274+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "f3727ffd-67c3-43ae-ac98-707ffdf935e8"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:55:17.658527+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "bf5463fb-97b0-409b-828f-2d371d91fec5"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:08:37.190744+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "4a4522ba-3c39-43a4-9e6a-8940b0e2db6f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:08:48.661990+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "93ac7edb-0e92-4762-91b7-3e3320085370"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:22:16.080770+00:00", "expires": null, "group": null, "retries": 24, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "13c3f8fd-8866-4291-bb54-2665e4fd3c4f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:25:35.367521+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "c64d5b7c-4fad-48ed-be0a-cb4e3e490d23"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:28:03.246552+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "4ea22f97-ab6b-44ed-b84e-a201257fe996"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:30:00.382080+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "da580041-9844-40af-b5de-4d87148a59e4"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:32:17.004536+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "f43f7bac-76d2-40b9-bed1-c50ed01337c2"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:35:24.480076+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "[email protected]"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "[email protected]", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "dd4396c0-6d60-47f3-ae54-e4fe72f8f8ec"}}
updated to celery 4.3.0rc2, the problem still exists, tasks being duplicated overnight multiple times so 20 task can grow up to 12000. Can you please help me with this issue? which version should I downgrade to avoid this bug?
It seems like this is a very old bug. I don't know which version of Celery does not contain it.
In any case, since this happens on Celery 3 and we don't support it anymore I suggest the following:
One of you who can reproduce this should provide a test case or attempt to fix it and send a PR.
Providing a DP here. My setting is similar to many others, celery 4.2.1 with redis and the duplicated task is a delayed task with ETA. Multiple workers received the task at the same time.
any testcase?
Same issue with redis broker, delay task with ETA future than visibility_timeout will create duplicate task
celery==4.3.0
billiard==3.6.0.0
kombu==4.6.3
pytz==2019.1
redis==3.2.1
vine==1.3.0
switch to rabbitmq solve the problem.
Any update if there is going to be a fix for AWS-SQS soon, as we have a similar problem there too!
may I know its update with celery==4.4.0
I still observer this issue with celery 4.4.0. I will check if I can reproduce this issue with a test case or some simpler reproduce steps.
Everything is fine with
-celery = {extras = ["redis"],version = "==4.2.1"}
-kombu = '==4.2.2.post1'
We notice this issue after doing the upgrade:
+celery = {extras = ["redis"],version = "==4.3.0"}
+kombu = '==4.4.0'
and the issue persists with
celery = {extras = ["redis"],version = "==4.4.0"}
"kombu": {
"hashes": [
"sha256:2a9e7adff14d046c9996752b2c48b6d9185d0b992106d5160e1a179907a5d4ac",
"sha256:67b32ccb6fea030f8799f8fd50dd08e03a4b99464ebc4952d71d8747b1a52ad1"
],
"version": "==4.6.7"
},
Update on 3/3:
The issue seems gone after downgrade to 'celery[redis]==4.2.1' 'kombu==4.2.2.post1'.
Update on 3/6:
Found a lot of data in Redis' unacked
key. Here are some of them.
'12fb3ab3-d784-42eb-b8ce-f2a4dedf86db': '[{"body": "gAIoSmkS5ABN+wdVMEZOM3kxQmdHT1phd3d4YVlxcF92U0YwNVJ5YWdmSDFHM0tXZkVZMDVEV18xS0NBcHEBfXECKFUHYWNjb3VudEppEuQAVQxzdWJzY3JpcHRpb25Kj+sDAHV0cQN9fXEEKFgFAAAAY2hvcmRxBU5YCQAAAGNhbGxiYWNrc3EGTlgIAAAAZXJyYmFja3NxB05YBQAAAGNoYWlucQhOdYdxCS4=", "headers": {"origin": "gen23299@test-celery-worker-1", "lang": "py", "task": "foo.celery.tasks.credentials.requests.notify_webhook", "group": null, "root_id": "0eb82b08-5318-40bb-92b6-a19095a79ab2", "expires": null, "retries": 11, "timelimit": [null, null], "argsrepr": "(14946921, 2043, \'FN3y1BgGOZawwxaYqp_vSF05RyagfH1G3KWfEY05DW_1KCAp\', {\'account\': 14946921, \'subscription\': 256911})", "eta": "2020-03-06T09:51:14.024135+00:00", "parent_id": "a875de24-b48f-459e-bae4-173e3c85b387", "id": "bae2803a-ec5c-4ca7-8ed2-2ce2b755265e", "redelivered": true, "shadow": null, "kwargsrepr": "{}"}, "content-type": "application/x-python-serialize", "properties": {"origin": "gen23298@test-celery-worker-1", "body_encoding": "base64", "delivery_info": {"routing_key": "celery-bg", "exchange": ""}, "delivery_mode": 1, "priority": 0, "correlation_id": "bae2803a-ec5c-4ca7-8ed2-2ce2b755265e", "reply_to": "cd5a7d96-b46d-3446-ae18-a3010a2e838a", "delivery_tag": "12fb3ab3-d784-42eb-b8ce-f2a4dedf86db"}, "content-encoding": "binary"}, "", "celery-bg"]',
'07d5c057-40e8-4aa9-9a96-23f14a11307d': '[{"body": "gAIoSmkS5ABN+wdVMEZOM3kxQmdHT1phd3d4YVlxcF92U0YwNVJ5YWdmSDFHM0tXZkVZMDVEV18xS0NBcHEBfXECKFUHYWNjb3VudEppEuQAVQxzdWJzY3JpcHRpb25Kj+sDAHV0cQN9fXEEKFgFAAAAY2hvcmRxBU5YCQAAAGNhbGxiYWNrc3EGTlgIAAAAZXJyYmFja3NxB05YBQAAAGNoYWlucQhOdYdxCS4=", "headers": {"origin": "gen23299@test-celery-worker-1", "lang": "py", "task": "foo.celery.tasks.credentials.requests.notify_webhook", "group": null, "root_id": "0eb82b08-5318-40bb-92b6-a19095a79ab2", "expires": null, "retries": 11, "timelimit": [null, null], "argsrepr": "(14946921, 2043, \'FN3y1BgGOZawwxaYqp_vSF05RyagfH1G3KWfEY05DW_1KCAp\', {\'account\': 14946921, \'subscription\': 256911})", "eta": "2020-03-06T09:50:17.118106+00:00", "parent_id": "a875de24-b48f-459e-bae4-173e3c85b387", "id": "bae2803a-ec5c-4ca7-8ed2-2ce2b755265e", "redelivered": true, "shadow": null, "kwargsrepr": "{}"}, "content-type": "application/x-python-serialize", "properties": {"origin": "gen23298@test-celery-worker-1", "body_encoding": "base64", "delivery_info": {"routing_key": "celery-bg", "exchange": ""}, "delivery_mode": 1, "priority": 0, "correlation_id": "bae2803a-ec5c-4ca7-8ed2-2ce2b755265e", "reply_to": "cd5a7d96-b46d-3446-ae18-a3010a2e838a", "delivery_tag": "07d5c057-40e8-4aa9-9a96-23f14a11307d"}, "content-encoding": "binary"}, "", "celery-bg"]',
'5c840cb9-6e20-4958-8f58-1a3502c3f8cb': '[{"body": "gAIoVSAxNTFlYWNkNDM5NDE0YjZjYjg3MGQyMGI3NzJjYTYyOHEBTfsHVQRmYWtlcQJ9cQNVA2Zvb1UaMjAyMC0wMy0wNlQwOTozMjozNi4xNTQ5OTlzdHEEfX1xBShYBQAAAGNob3JkcQZOWAkAAABjYWxsYmFja3NxB05YCAAAAGVycmJhY2tzcQhOWAUAAABjaGFpbnEJTnWHcQou", "headers": {"origin": "gen23299@test-celery-worker-1", "lang": "py", "task": "foo.celery.tasks.credentials.requests.notify_webhook", "group": null, "root_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "expires": null, "retries": 11, "timelimit": [null, null], "argsrepr": "(\'151eacd439414b6cb870d20b772ca628\', 2043, \'fake\', {\'foo\': \'2020-03-06T09:32:36.154999\'})", "eta": "2020-03-06T09:47:22.954949+00:00", "parent_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "redelivered": true, "shadow": null, "kwargsrepr": "{}"}, "content-type": "application/x-python-serialize", "properties": {"origin": "gen23299@test-celery-worker-1", "body_encoding": "base64", "delivery_info": {"routing_key": "celery-bg", "exchange": ""}, "delivery_mode": 1, "priority": 0, "correlation_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "reply_to": "4f0b394f-dc4e-37bc-9259-b0a1f3400d81", "delivery_tag": "5c840cb9-6e20-4958-8f58-1a3502c3f8cb"}, "content-encoding": "binary"}, "", "celery-bg"]',
'9e9eb118-4799-4224-8640-607c0d4189fc': '[{"body": "gAIoVSAxNTFlYWNkNDM5NDE0YjZjYjg3MGQyMGI3NzJjYTYyOHEBTfsHVQRmYWtlcQJ9cQNVA2Zvb1UaMjAyMC0wMy0wNlQwOTozMjozNi4xNTQ5OTlzdHEEfX1xBShYBQAAAGNob3JkcQZOWAkAAABjYWxsYmFja3NxB05YCAAAAGVycmJhY2tzcQhOWAUAAABjaGFpbnEJTnWHcQou", "headers": {"origin": "gen23299@test-celery-worker-1", "lang": "py", "task": "foo.celery.tasks.credentials.requests.notify_webhook", "group": null, "root_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "expires": null, "retries": 12, "timelimit": [null, null], "argsrepr": "(\'151eacd439414b6cb870d20b772ca628\', 2043, \'fake\', {\'foo\': \'2020-03-06T09:32:36.154999\'})", "eta": "2020-03-06T09:56:23.530909+00:00", "parent_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "shadow": null, "id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "kwargsrepr": "{}"}, "content-type": "application/x-python-serialize", "properties": {"origin": "gen23298@test-celery-worker-1", "body_encoding": "base64", "delivery_info": {"routing_key": "celery-bg", "exchange": ""}, "delivery_mode": 1, "priority": 0, "correlation_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "reply_to": "4f0b394f-dc4e-37bc-9259-b0a1f3400d81", "delivery_tag": "9e9eb118-4799-4224-8640-607c0d4189fc"}, "content-encoding": "binary"}, "", "celery-bg"]',
'44a7f92c-5ad7-47e3-8af0-4067c3cbc446': '[{"body": "gAIoVSAxNTFlYWNkNDM5NDE0YjZjYjg3MGQyMGI3NzJjYTYyOHEBTfsHVQRmYWtlcQJ9cQNVA2Zvb1UaMjAyMC0wMy0wNlQwOTozMjozNi4xNTQ5OTlzdHEEfX1xBShYBQAAAGNob3JkcQZOWAkAAABjYWxsYmFja3NxB05YCAAAAGVycmJhY2tzcQhOWAUAAABjaGFpbnEJTnWHcQou", "headers": {"origin": "gen23298@test-celery-worker-1", "lang": "py", "task": "foo.celery.tasks.credentials.requests.notify_webhook", "group": null, "root_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "expires": null, "retries": 12, "timelimit": [null, null], "argsrepr": "(\'151eacd439414b6cb870d20b772ca628\', 2043, \'fake\', {\'foo\': \'2020-03-06T09:32:36.154999\'})", "eta": "2020-03-06T09:53:20.107464+00:00", "parent_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "shadow": null, "redelivered": true, "kwargsrepr": "{}"}, "content-type": "application/x-python-serialize", "properties": {"origin": "gen23299@test-celery-worker-1", "body_encoding": "base64", "delivery_info": {"routing_key": "celery-bg", "exchange": ""}, "delivery_mode": 1, "priority": 0, "correlation_id": "4cd3108b-f75a-442f-9e3a-477576f65af0", "reply_to": "4f0b394f-dc4e-37bc-9259-b0a1f3400d81", "delivery_tag": "44a7f92c-5ad7-47e3-8af0-4067c3cbc446"}, "content-encoding": "binary"}, "", "celery-bg"]',
It seems like the issue happens after upgrade kombu from 4.2.2.post1 to 4.3.0.
Used git bisect
locate the first bad commit is a307726faa8b64a71e04932b762498f51c1d7208
. This may be relavant because we use Redis + celery gevent thread and the changes seems also relavant.
git bisect start
# bad: [b4be5cf2e7e5ca538246e254bc6ebac5d6c5f477] Bump version: 4.2.2 → 4.3.0
git bisect bad b4be5cf2e7e5ca538246e254bc6ebac5d6c5f477
# good: [06ee7855814f66280b1b3277b0ac7ff4d8f3e76d] Mention 4.2.2-post1 in changelog.
git bisect good 06ee7855814f66280b1b3277b0ac7ff4d8f3e76d
# good: [c627c6d9d1f29ae0bcb03931d5f24001e60f72a9] Bump version: 4.2.0 → 4.2.1
git bisect good c627c6d9d1f29ae0bcb03931d5f24001e60f72a9
# bad: [c48d201ee2afe80cc6a30d9e58beb982d20fc220] Allow setting boto3.sqs.create_queue Attributes via transport_options (#957)
git bisect bad c48d201ee2afe80cc6a30d9e58beb982d20fc220
# bad: [62087a67cfbae4e87718fbb917f5140aa66e8287] Fix a deprecation warning about logger.warn() (#924)
git bisect bad 62087a67cfbae4e87718fbb917f5140aa66e8287
# good: [87d6423289caa776fc10a8d7cdb3d4b741460cfa] fix/Broadcast-fail-by-give-queue-kwargs (#906)
git bisect good 87d6423289caa776fc10a8d7cdb3d4b741460cfa
# bad: [200a60a228bd836542e9e9a2759af1d915139d1c] Add support for 'rediss' scheme for secure redis connections. (#910)
git bisect bad 200a60a228bd836542e9e9a2759af1d915139d1c
# bad: [9e64dafa617d4403edfa07e7ead015ce081046c0] Allow SimpleQueue to pass 'queue_arguments' to Queue object. (#900)
git bisect bad 9e64dafa617d4403edfa07e7ead015ce081046c0
# bad: [a307726faa8b64a71e04932b762498f51c1d7208] on worker restart - restore visible regardless to time (#905)
git bisect bad a307726faa8b64a71e04932b762498f51c1d7208
# first bad commit: [a307726faa8b64a71e04932b762498f51c1d7208] on worker restart - restore visible regardless to time (#905)
Alright. My case already be mentioned in https://github.com/celery/kombu/issues/1098.
@auvipy @thedrow
I encountered the bug and managed to reproduce it (Celery 4.4.5 + SQS) with a fairly simple test case:
Task code:
from celery import Celery
app = Celery('proj', broker='sqs://')
@app.task()
def run_once():
print('Hi')
return 1
The calling code:
In [2]: tasks.run_once.apply_async(countdown=1830)
Out[2]: <AsyncResult: 71153f8e-1fae-4cee-9923-4753ec0f2d45>
Celery log:
[2020-06-08 14:21:38,328: INFO/MainProcess] Received task: proj.tasks.run_once[71153f8e-1fae-4cee-9923-4753ec0f2d45] ETA:[2020-06-08 11:52:06.057666+00:00]
[2020-06-08 14:51:40,114: INFO/MainProcess] Received task: proj.tasks.run_once[71153f8e-1fae-4cee-9923-4753ec0f2d45] ETA:[2020-06-08 11:52:06.057666+00:00]
[2020-06-08 14:52:06,967: WARNING/ForkPoolWorker-2] Hi
[2020-06-08 14:52:06,968: WARNING/ForkPoolWorker-3] Hi
[2020-06-08 14:52:06,972: INFO/ForkPoolWorker-2] Task proj.tasks.run_once[71153f8e-1fae-4cee-9923-4753ec0f2d45] succeeded in 0.006056700000044657s: 1
[2020-06-08 14:52:06,972: INFO/ForkPoolWorker-3] Task proj.tasks.run_once[71153f8e-1fae-4cee-9923-4753ec0f2d45] succeeded in 0.005694999999832362s: 1
You need to run the celery worker before calling the task and you must leave it running until the task runs.
@noamkush Interesting to see the timestamps, looks like the task is accepted first and then maybe re-scheduled after 30 minutes? And then executed twice?
fixes are welcome
So I figured out this is not an actual bug but SQS behavior. 30 minutes is celery's default visibility timeout and the message was simply redelivered to celery. Which brings me to the following:
So I figured out this is not an actual bug but SQS behavior. 30 minutes is celery's default visibility timeout and the message was simply redelivered to celery. Which brings me to the following:
1. Documentation is wrong, it states the default visibility timeout is 30 seconds, which is in fact AWS's default but not Celery's (I'll maybe open a PR later). 2. Maybe this is preventable? Shouldn't SQS broker periodically extend visibility timeout?
both can be improved :) looking forward to your PR
can you check this https://github.com/celery/kombu/issues/1098#issuecomment-773083078
2. Maybe this is preventable? Shouldn't SQS broker periodically extend visibility timeout?
This is actually a good idea. Can you please open a new feature request.
We're going to measure the duration of tasks anyway so we can actually calculate some heuristic to help us extend the visibility timeout.
@noamkush Are you able to issue a PR for us regarding the documentation?
Why is the task being redelivered? If the task is acked after the first time, the task should go away from SQS, right?
Visibility timeout should not matter in this case, shouldn't it?
The task is received from sqs but isn't acked since there's a countdown. So celery waits till countdown is over but doesn't ack and meanwhile the visibility timeout expires and the task is redelivered (this is actually noted in caveats). What I suggested is that the celery process that is waiting for countdown would extend the visibility timeout so it won't be redelivered.
Most helpful comment
Hi all
We've found same issue in several our projects when we used Redis as broker but seems it's not broker related problem. It's really looks like (we've checked the logs) any delayed task (ETA used, for example with
countdown
argument) several workers pulls same task (with same Task ID). And I found a lot of posts in Internet that describes that situation but in different words (post, issue). And there is repository specifically for that case. May be that lock should be implemented at Celery level for any broker?We using this our snippet to prevent such task behavior:
Feel free to test and point to any errors in this snippet.