Django-celery-beat: Interval tasks from db causing tasks to be called continuously

[remcinerney]

Hi have setup django-celery-beat as described here:
http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#using-custom-scheduler-classes

However if I create an interval task in the DB, the scheduler is firing off that tasks continuously (it should be every second). I've attached a screen shot of the beat worker output (showing the constantly streaming task), and the settings for the interval task within admin.

I'm running celery with one worker and one beat:
worker: celery -A proj worker -l info
beat: celery -A proj beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler

I'm running:
Django==1.11.5
celery==4.1.0
redis==2.10.6
django-celery-beat==1.1.0
django-celery-results==1.0.1

UPDATE:
There seems to be a problem with the timings of the scheduler. If I set the interval to (e.g., 2 minutes), then the task is called every minute. If I set the interval to a minute or less, then the scheduler goes nuts. I think there must be a bug, whereby 60 seconds is being taken off the scheduled interval time...?

[chkdmin]

+1

[amitripshtos]

+1

[slenart]

+1

[miakramer]

I had a similar issue, except for debug purposes I had an interval of two minutes. The scheduler sent nothing until several hours after starting, at which point it had the behaviour above.

[astrikov-d]

+1

[ShaheedHaque]

See my comments in #96 and #128. I followed up to see if I could reproduce this issue (or #109). So, I set the interval down to 5s, and since each task took between 30 and 60 seconds (a random delay), the Rabbitmq queue built up gradually. However, once I set the enabled flag to false, the queue drained and the system gradually quiesced itself.

I then set the interval to zero, and as expected, the tasks flowed rapidly:

[2018-02-15 09:32:39,914: INFO/MainProcess] Scheduler: Sending due task Porsche:w1 (xxx.yyy.zzz)
[2018-02-15 09:32:39,918: INFO/MainProcess] Scheduler: Sending due task Porsche:w1 (xxx.yyy.zzz)
[2018-02-15 09:32:39,923: INFO/MainProcess] Scheduler: Sending due task Porsche:w1 (xxx.yyy.zzz)
[2018-02-15 09:32:39,927: INFO/MainProcess] Scheduler: Sending due task Porsche:w1 (xxx.yyy.zz)

and the queue built up:

Every 5.0s: sudo rabbitmqctl list_queues name messages messages_ready messages_unacknowledged                                         baba: Thu Feb 15 09:37:10 2018

Listing queues
celeryev.26f75434-7b2d-4cb1-9f48-b4d7df05db4e   0       0       0
celeryev.2e2b1d10-e8c9-4892-b672-20f3c1cde813   0       0       0
[email protected]       0       0       0
celeryev.b8f1d47d-ae1f-4dc6-bd32-e151e869a08e   0       0       0
celery  6952    6936    16

Again, once I set the enabled flag to false, the queue drained and the system gradually started to quiesce itself. So, with commit 2005bbe, I can reproduce the high task rate only by setting the interval to zero. Then, since my tasks take much longer to run than zero to run, the queue also builds up (i.e. making it look like #128 is an issue).

Finally, using sudo rabbitmqctl purge_queue celery facilitated recovery of the system by draining the queue. Thus, I can partially reproduce the symptoms reported here, but only by effectively setting up an unstable setup. With luck, this means that as of at least 2005bbe, I cannot usefully reproduce this.

[rafagonc]

+1

[xyzza]

+1 same problem

OS X 10.11.6
Python 3.6
celery 4.1.0
django-celery-beat 1.1.1
Django 1.11.6
Redis 3.0.2

I'v implemented a tutorial example from https://pypi.python.org/pypi/django_celery_beat

When I'm runing
celery -A web_app worker -l info -B --scheduler django_celery_beat.schedulers:DatabaseScheduler

scheduler starts sending tasks at every millisecond

both pattern for interval and crontab tasks (I tried intervals less than a 59 seconds)

[xyzza]

I'v got the error! In my case problem was in mismatching django Timezone and celery Timezone options:

I had this in settings.py:

...
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
...
...
CELERY_TIMEZONE = 'Europe/Moscow'

So I discarded option "CELERY_TIMEZONE = 'Europe/Moscow'" and replaced value for default django timezone setting TIME_ZONE for appropriate timezone value, and it works!
Celery beat stops to spam and now it works as it should to be.

After all my settings.py looks like:

...
TIME_ZONE = 'Europe/Moscow'
USE_I18N = True
USE_L10N = True
USE_TZ = True

# CELERY_TIMEZONE = 'Europe/Moscow'
...

I hope my case will help someone else

[srolel]

I'm seeing this as well, and don't have CELERY_TIMEZONE set.

[tillkolter]

+1

[af2tr-snippets]

+1. See my setup:
Python==2.7.14

amqp==2.2.2
APScheduler==3.5.1
Babel==2.2.0
backports-abc==0.5
backports.ssl-match-hostname==3.5.0.1
billiard==3.5.0.3
celery==4.1.0
certifi==2018.1.18
click==6.6
cycler==0.10.0
deprecation==2.0
dill==0.2.6
Django==1.11.11
django-celery-beat==1.1.0
django-celery-monitor==1.1.2
django-templates-macros==0.2
django-widget-tweaks==1.4.1
ephem==3.7.6.0
flower==0.9.2
funcsigs==1.0.2
functools32==3.2.3.post2
future==0.16.0
futures==3.2.0
Jinja2==2.10
kombu==4.1.0
lazy==1.3
lz4==1.0.0
MarkupSafe==1.0
matplotlib==2.0.2
multipledispatch==0.4.9
numpy==1.14.0
packaging==16.8
pandas==0.22.0
psycopg2-binary==2.7.4
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2018.3
PyYAML==3.12
redis==2.10.6
scipy==1.0.0
seaborn==0.7.1
singledispatch==3.4.0.3
six==1.11.0
subprocess32==3.2.7
tornado==4.5.3
tzlocal==1.5.1
ujson==1.35
vine==1.1.4
wrapt==1.10.11
xarray==0.10.0

[allardlamberink]

+1

[rkdgusrnrlrl]

sorry i did not realize that my issue linked here.
so sorry about that. :pray::pray:
i request how i can fix this to github and if that will be fix i will remove this comment too. sorry

[doocan]

I have the same problem when I specify the timezone and set USE_TZ = False.

It will work well when USE_TZ = True.

[craigds]

Stumbled across this, same issue when USE_TZ = False. When USE_TZ = True it works correctly. That's a difficult thing to just change on a large codebase though. Any chance of a fix?

[haroldrandom]

Any updates?

[twz915]

I found that celery.schedules.crontab(nowfun=now) with USE_TZ = False and ENABLE_UTC = False trigger this issue too.

[yaomengsi]

I found the hacking settings like this:

DJANGO_CELERY_BEAT_TZ_AWARE = False
app.now = timezone.now

example:

settings.py

# IMPORTANT: fix django-celery-beat with `USE_TZ = False`, interval tasks be called continuously
# time zone native
DJANGO_CELERY_BEAT_TZ_AWARE = False

celery.py

from __future__ import absolute_import, unicode_literals

import os

from celery import Celery
from django.utils import timezone

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'PROJECT_NAME.settings')

app = Celery('PROJECT_NAME')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# IMPORTANT: fix django-celery-beat with `USE_TZ = False`, interval tasks be called continuously
app.now = timezone.now

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

I hope this will help someone else.

[amitripshtos]

Sorry for getting out of context of the issue itself, but I created a small gist that works great in production for me (The scheduling was mission critical component and I could not risk it anymore using celery-beat)

It's using APScheduler instead celery-beat to schedule my tasks.

https://gist.github.com/amitripshtos/d4840406f7d6374ef8abcc2dca937583

Simply replace "get_tasks", "get_celery_application" and "update_periodic_tasks_from_database" methods with your tasks (can be from database) and your celery application object.

You can adjust timezone:
scheduler = BackgroundScheduler(timezone='America/New_York')

[MirrorInFolwer]

我发现了这样的黑客设置：

DJANGO_CELERY_BEAT_TZ_AWARE =假
app.now = timezone.now

例：

settings.py

＃重要提示：修复Django的芹菜拍用`USE_TZ = FALSE`，间隔任务被连续调用
＃时区本地
DJANGO_CELERY_BEAT_TZ_AWARE  =  假

celery.py

从 __future__  进口 absolute_import，unicode_literals 进口 OS 从芹菜进口芹菜
 从 django.utils 导入时区＃设置默认的Django设置模块的“芹菜”计划。
os.environ.setdefault（' DJANGO_SETTINGS_MODULE '，' PROJECT_NAME.settings '）
app = Celery（' PROJECT_NAME '）＃在这里使用字符串表示工作人员不必序列化＃









子进程的配置对象。
＃ -  namespace ='CELERY'表示所有与芹菜相关的配置键
＃    应该有一个`CELERY_`前缀。
app.config_from_object（' django.conf：设置'，命名空间= '芹菜'）＃重要提示：修复Django的芹菜拍用`USE_TZ = FALSE`，间隔任务被称为连续 
app.now = timezone.now ＃装载任务模块从所有注册的Django应用程序配置。
app.autodiscover_tasks（）

我希望这会帮助别人。

谢谢 think you，这个确实帮助到了我。

[auvipy]

anyone with a fix as or?

[auvipy]

I am closing in favor of https://github.com/celery/django-celery-beat/commit/2005bbe4cacf734be0d2f5345dc3aa6285a483ce