Django-celery-beat: Django Celery Beat admin updating Cron Schedule Periodic task not taking effect?

[kendazheng]

like this
http://stackoverflow.com/questions/40579804/django-celery-beat-admin-updating-cron-schedule-periodic-task-not-taking-effect

[wen96]

Same issue here. Whenever I update or create a PeriodickTask I have to reload beat to get this changes updated on worker. There a process (I guess pulling process) every 3 minutes on beat saying Writing entries... but nothing happens if you don't restart beat. Are we missing something?

Using:
celery 4.0.0
Django 1.9.9
django-celery-beat 1.0.1
Ubuntu 14.04
Redis 2.8.4

Configuration:

BROKER_URL = 'redis://localhost:6379/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERYBEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

[kendazheng]

I have test other celery version and djcelery, it work well. So, it's a bug ?
I have read the source code already. The DatabaseScheduler property method 'schedule' have received the changes, but the method 'reserve' can Not get it. Because I do not know about the project，so I do not know why.

[mheppner]

Not sure if this helps, but with Celery 4, your setting name should be prefixed. BROKER_URL should be CELERY_BROKER_URL and CELERYBEAT_SCHEDULER should be CELERY_BEAT_SCHEDULER. I don't think you need to specify JSON for any of the serializers because it should be default now.

[wen96]

@mheppner yes I know, config was just a copy paste which contains lines which comes from older Celery versions. The worker and beat process shows config on start which is OK and it reads and writes on Redis and on Django database correctly. DB changes simply don't come to reserve function as @kendazheng says. Anyway, thank you.

[mheppner]

@wen96 that's what I figured, but I thought I'd check anyways. I tried using the django-celery-beat scheduler after I made that comment and I also experienced the same issue. A full restart is needed. I'll look at it tonight and see if I can figure out why.

[andrewgleave]

Same here. Any further developments?

[mheppner]

It doesn't look like the scheduler itself is ever checked again and I'm not sure why. I think the beat process itself needs to access DatabaseScheduler.schedule() on each tick for it to even detect if something changed, but it's never accessed after the first time.

[exploreshaifali]

Exact same issue I am facing, and it is working well with celery==3.1.23 and djcelery.

[t-io]

I had the same issue because I forgot to start the worker with __-S django__ like celery -A proj beat -l info -S django
http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

[okfish]

I had the same issue with all options set to worker properly, but had investigated that the schedule has been updated after adding new record and has not been updated after modifying existing record of intervals

[okfish]

However, it was just a message in logs like DatabaseScheduler: Schedule changed. But nothing was really changed in the schedule, nor new task wasn't added neither old task wasn't modified

[roofcat]

I had the same issue, i use command celery -A app beat -l info -S django, but tasks doesn't run.
My celery config file:

"""
from __future__ import absolute_import
import os
from django.conf import settings
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
app = Celery('track')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.update(
CELERY_BROKER_URL='amqp://guest:guest@localhost//',
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
)
"""
my tasks file:
"""
import logging
from app import celery_app

from .models import Note

logger = logging.getLogger(__name__)

@celery_app.task
def create_note():
logger.info('running tasks')
Note.new_note()
logger.info('tasks finished')
"""

The create_note method run if i execute in django shell but celery cannont run staks, why???

[lucadelmonte]

Hi, same issue here, when modifying a task from admin or even via shell nothings happens, when adding or removing celery beat logs "DatabaseScheduler: Schedule changed." and logs the right scheduleed task, but task are note being executed ( if added to db) or keep exececuting ( if removed from db)
$ pip freeze
amqp==2.1.1
anyjson==0.3.3
Babel==2.3.4
billiard==3.5.0.2
celery==4.0.0
Django==1.10.3
django-celery-beat==1.0.1
django-celery-results==1.0.1
flower==0.9.1
kombu==4.0.0
pkg-resources==0.0.0
pytz==2016.7
tornado==4.2
vine==1.1.3

[lucadelmonte]

@roofcat Are you running the command "celery -A app worker -l info" too, in another shell?
Celery beat is the scheduler, celery worker spawns a woker which can process the task scheduled by beat.
This issue is different from yours, we manage to get the tasks running, our changes to the table periodictask are not picked up by the beat scheduler unless we restart it.

[roofcat]

@Squalluca ohh thanks, IT WORKS!!!! i have not realized

[rjbks]

Any updates on this, I am also experiencing this issue. Changes after beat has started log a :
DatabaseScheduler: Schedule changed.
message but the changes do not actually take effect. Newly scheduled tasks dont run and disabled tasks continue to run.

[roofcat]

I have a question with celerybeat daemon doesn't run

My /etc/default/celeryd file:
`ENV_PYTHON="/home/crojas/.virtualenvs/az-track/bin/python"

CELERY_BIN="/home/crojas/.virtualenvs/az-track/bin/celery"

CELERYD_CHDIR="/home/crojas/az-track"

CELERY_APP_ARG="app"

export DJANGO_SETTINGS_MODULE="app.settings"

CELERYD_NODES=3

CELERYD_OPTS="--concurrency=3"

CELERYD_LOG_LEVEL="INFO"

CELERY_CONFIG_MODULE="app.celery"

CELERYD_LOG_FILE="/var/log/celery/%n.log"

CELERYD_PID_FILE="/var/run/celery/%n.pid"

CELERYD_USER="celery"

CELERYD_GROUP="celery"

CELERY_CREATE_DIRS=1
`

And my /etc/default/celerybeat file:
`CELERY_BIN="/home/crojas/.virtualenvs/az-track/bin/celery"

CELERY_APP_ARG="app"

CELERYBEAT_CONFIG_MODULE="app.celery"

CELERYBEAT_OPTS="-S django"

CELERYBEAT_LOG_LEVEL="INFO"

CELERYBEAT_USER="celery"

CELERYBEAT_GROUP="celery"

CELERY_CREATE_DIRS=1
`
My celery worker daemon it works, but celery beat cannot run, why? thanks all

[mheppner]

@roofcat if you're using celery 4 and django-celery-beat, you need to follow the upgrade guide and change your settings to the new format.

[roofcat]

@mheppner i can run celery beat with manual command successful, but cannot run with daemon :(

the error is:
WARNING/MainProcess] DBAccessError
WARNING/MainProcess] (13, 'Permission denied')

in my celery.py config:
"""
from __future__ import absolute_import
import os
from django.conf import settings
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')

app = Celery('test')

app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

app.conf.update(
broker_url='amqp://guest:guest@localhost//',
beat_scheduler='django_celery_beat.schedulers.DatabaseScheduler',
)

[jottunn]

having the same issue, celery 4.0, django 1.10.2, django_celery_beat 1.0.1

[roofcat]

Is normal celery beat cannot run if celery worker is running as daemon??

my ubuntu journalctl log:

Nov 25 15:06:30 az-track-test sudo[8232]: crojas : TTY=pts/1 ; PWD=/home/crojas/az-track ; USER=root ; COMMAND=/bin/su celery -c /home/crojas/.virtualenvs/az-track/bin/celery beat --app=app.celery -f /var/log/celery/beat.log -l INFO --workdir=/home/crojas/az-track --detach --pidfile=/var/run/celery/beat.pid
Nov 25 15:06:30 az-track-test sudo[8232]: pam_unix(sudo:session): session opened for user root by crojas(uid=0)
Nov 25 15:06:30 az-track-test su[8233]: Successful su for celery by root
Nov 25 15:06:30 az-track-test su[8233]: + /dev/pts/1 root:celery
Nov 25 15:06:30 az-track-test su[8233]: pam_unix(su:session): session opened for user celery by crojas(uid=0)
Nov 25 15:06:30 az-track-test su[8233]: pam_systemd(su:session): Cannot create session: Already running in a session
Nov 25 15:06:30 az-track-test su[8233]: pam_unix(su:session): session closed for user celery
Nov 25 15:06:30 az-track-test sudo[8232]: pam_unix(sudo:session): session closed for user root

thanks!

[frnhr]

Same issue as OP, after save in admin, the beat process outputs DatabaseScheduler: Schedule changed. but keeps scheduling the task as if nothing changed.
No difference if using crontab or interval config.

Let's keep the discussion focused on the OP's issue, otherwise it might seem solved when it's not.

[lucadelmonte]

hey @ask , sorry to bother, can we have an update on this? are we doing something wrong? is this expected or is this a bug?

[vladitot]

Hi there, I have same problem.
My celery beat works correctly, but on updating period of running, celery beat keep previous period, so I need to restart beat to load new settings to beat.
I believe anyone has any ideas

[jegesh]

+1

[roofcat]

+1

[wzywno]

Does anyone has workaround or solution for that?

[AlwxSin]

Does anyone know which versions of celery and celery-beat works properly together?
Or maybe which commits?

[roofcat]

django-celery-beat works with celery 4, django-celery (djcelery) works with 3.x

[mef13]

+1

[jegesh]

I switched my project over to django-rq for now. It took me about three hours to learn the technology and make all the necessary changes. It's really simple to get started with.

[TomaszKolek]

Guys, do you have any workaround?

[mheppner]

This is a bug. Please stop adding "+1", suggesting alternative frameworks, asking for working versions, and seeing if people have a workaround. If you are having issues not related to this bug, try posting your question in the user group first.

[TomaszKolek]

Because I needed to do it ASAP and let's say quality of my solution wasn't important - I found a problem in celery not in django-celery-beat. If it will help:

celery/beat.pyclass Service method tick:

H = self._heap
if H is None:
    H = self._heap = [event_t(_when(e, e.is_due()[1]) or 0, 5, e)
                                 for e in values(self.schedule)]
   heapify(H)

It looks H is None only in first execution of this function and never again. So I added in __init__.py one variable old_schedueles (None by default)
Next I added to above if H is None additional part if H is None or len(self.old_schedules) != len(values(self.schedule)) - it actualy means: go to code under if if H is None or if new schedules appeared. At the end I added inside the if self.old_schedulers = values(self.schedules)

Because I did some more operations for debugging - is possible that I did something else. I will try to reproduce my investigation and let you know.

Hope it will help.

[frnhr]

Probably related: https://github.com/celery/celery/issues/3493 "How to dynamically add or remove tasks to celerybeat?"

[wen96]

@frnhr it is not. This is about a bug on "celerybeat-schedule shelve database". The issue you mention is just a question which I think is already answered and should be close.

[wzywno]

In my dirty workaround I override tick method in class DatabaseScheduler(Scheduler): (schedulers.py file)
And in tick method I commented out

        if H is None:

from

        if H is None:
            H = self._heap = [event_t(_when(e, e.is_due()[1]) or 0, 5, e) for e in values(self.schedule)]
            heapify(H)

so now heap is built in every single run of tick method.

[TomaszKolek]

so it looks my solution is a little bit better ;-)
But its great to hear that its enough.

[wzywno]

I found another way to fix this issue.
When DatabaseScheduler decides that sync is required the _heap in Scheduler is set to None. so _heap is rebuilt on next run of tick() method.

So now DatabaseScheduler looks like that:

    @property
    def schedule(self):
        update = False
        if not self._initial_read:
            debug('DatabaseScheduler: initial read')
            update = True
            self._initial_read = True
        elif self.schedule_changed():
            info('DatabaseScheduler: Schedule changed.')
            update = True

        if update:
            self.sync()
            self._schedule = self.all_as_schedule()
            if logger.isEnabledFor(logging.DEBUG):
                debug('Current schedule:\n%s', '\n'.join(
                    repr(entry) for entry in values(self._schedule)),
                )
            # _heap will be rebuilt on the next run
            super(Scheduler, self).__setattr__('_heap', None)
        return self._schedule

super(Scheduler, self).__setattr__('_heap', None) is added in comparison to original version.

I created PR with fix. It is a bit more elegant than workaround :)

[razerraz]

@wzywno
Both of your fixes doesn't work for me, but thanks for the effort
@TomaszKolek
Your fix work for me, but you've written a bunch of typos. To be clear ::
in celery/beat.py, class Service,
constructor (__init__) -> add self.old_schedulers = None
and method tick becomes

def tick(self, event_t=event_t, min=min,
             heappop=heapq.heappop, heappush=heapq.heappush,
             heapify=heapq.heapify, mktime=time.mktime):
        """Run a tick - one iteration of the scheduler.

        Executes one due task per call.

        Returns:
            float: preferred delay in seconds for next call.
        """
        def _when(entry, next_time_to_run):
            return (mktime(entry.schedule.now().timetuple()) +
                    (adjust(next_time_to_run) or 0))

        adjust = self.adjust
        max_interval = self.max_interval
        H = self._heap
        if H is None or len(self.old_schedulers) != len(values(self.schedule)):
            H = self._heap = [event_t(_when(e, e.is_due()[1]) or 0, 5, e)
                          for e in values(self.schedule)]
            heapify(H)
            self.old_schedulers = values(self.schedule)
        if not H:
            return max_interval

        event = H[0]
        entry = event[2]
        is_due, next_time_to_run = self.is_due(entry)
        if is_due:
            verify = heappop(H)
            if verify is event:
                next_entry = self.reserve(entry)
                self.apply_entry(entry, producer=self.producer)
                heappush(H, event_t(_when(next_entry, next_time_to_run),
                                    event[1], next_entry))
                return 0
            else:
                heappush(H, verify)
                return min(verify[0], max_interval)
        return min(adjust(next_time_to_run) or max_interval, max_interval)

[Coread]

@razerraz The comparison of between old and new schedules is insufficient. If the schedule intervals are changed then the required heapify will still not take place. The below code shows a comparison I used for IntervalSchedules. Crontab schedules would require a more involved check of equality.

def schedules_equal(self, a, b):
    if a.keys() != b.keys():
        return False
    for name, model in a.items():
        b_model = b.get(name)
        if not b_model:
            return False
        if not hasattr(model.schedule, "human_seconds"):
            # Not checking crontabs
            continue
        if model.schedule.human_seconds != b_model.schedule.human_seconds:
            return False
    return True

so instead of the length check

if H is None or not self.schedules_equal(self.old_schedulers,self.schedule):

and then

self.old_schedulers = self.schedule

[razerraz]

seems to make sence, thanks

[frnhr]

This problem is still present on 4.0.2 and on master, tested just now.

[bgroff]

I can also confirm that I am still seeing this issue with 4.0.2.

[codefo]

Me too, both on master and 4.0.2

[oortega]

I have the same issue, fix temporaly with crontab

crontab -e

60 * * * * /usr/bin/supervisorctl restart { instance name Beat}

[dmitryTsatsarin]

+1. Celery 4.0.2. Still exists (need restart).

[wzywno]

I forked celery and added @Coread changes.
It works with those changes.
in requirements.txt

git+git://github.com/HealthTechDevelopers/celery@master#celery

[dmitryTsatsarin]

@wzywno Could you please make pull request to original branch? Thanks

[hmphu]

Hi @wzywno, I have tried your repo, that work extremely well. Could you please make a pull request to the original branch? Thanks

[oortega]

@wzywno Can you be the hero and do a pull request please? Thanks

[ma1onso]

This bug continue on fork repository for crontab / periodic task, only fix bug for interval. Thanks @wzywno for this fix :dancer:

[skvoter]

waiting for fix for crontab
@wzywno got some ideas for that fix? if you'll fix it i'd be so grateful to you!

[dmitryTsatsarin]

As for me, I have downgraded to celery 3. Why? Too many strange things in celery 4.
For example, will this error in log file or not? :-)
https://gist.github.com/dmitryTsatsarin/c6e83abc4dd66ec65eee4534217fff60

[skvoter]

@dmitryTsatsarin does the OP issue have place in celery 3?

[wzywno]

@skvoter @alons45 PR updated. Now should work also for crontab scheduling.

[evasilchenko]

Do you guys know by any chance when a new release will be made available with this fix?

[AJRenold]

Hi, we ran into what appears to be this issue when toggling the PeriodicTask.enabled field between True and False. I've just opened https://github.com/celery/django-celery-beat/pull/25 which may be an alternative to this pull request - https://github.com/celery/celery/pull/3791

[oiuww09fn]

@wzywno
after set self.old_schedulers to self.schedule, they will be the same object, and always be the same. it will not take effect immediately when there is new task.

[gitaarik]

I updated @wzywno's pull request to make it PEP8 valid and I squashed the commits for a clean history. All the checks pass now:

https://github.com/celery/celery/pull/3890

@ask what do you think?

[brianmay]

25 was proposed as an alternative solution, and it has been merged into django-celery-beat. However I also experienced the same problems with the latest master version of django-celery-beat.

[brianmay]

My initial test of celery/celery#3890 was flawed, it solves the problem.

[brianmay]

celery/celery#3958 is my pull request that fixes some problems with celery/celery#3890.

[subha-py]

Scheduler for cron is not working with django 11 and celery 4. Any help?

[brianmay]

@subha-py Nothing to do with this pull request, please open a new issue.

[TheBlusky]

Had the same issue: new PeriodicTask were not running if I don't restart the beat.

Thanks to @brianmay (https://github.com/celery/celery/pull/3958), I just changed celery==4.0.2 to git+https://github.com/celery/celery.git#egg=celery (because the PR is merged, but not released) in my requirements.txt and now it's working fine :-)

[nux17]

Well, changes in database (args, task) doesn't repopulate either, only changes in periods/interval and enabling disabling will do. Any solution?

[brianmay]

@nux17 With my patch that was merged in celery/celery#3958 - but not yet released - this works fine for me.

[nux17]

Eh, didn't work for me. Making changes in other things than interval or enabling/disabling doesn't refresh beat.

[brianmay]

@nux17 What version of celery are you using?

[MythriNagaraju]

This didn't work for me either. Deleting a periodic task or disabling aperiodic task is still not getting detected dynamically by celery beat. I am using:

Celery v4.0.0
Django v1.10.5
django-celery-beat: 1.0.1

[brianmay]

Celery 4.0.0 does not have the required change. Celery 4.0.2 is currently the latest release, and it doesn't have the required change either. Until there is a new release of celery, you will need to use the version in git.

[nux17]

I'm using the Git version as you mentioned it and it's still not fully working.

On Jun 18, 2017 01:12, Brian May [e-mail] wrote:

Celery 4.0.0 does not have the required change. Celery 4.0.2 is currently the latest release, and it doesn't have the required change either. Until there is a new release of celery, you will need to use the version in git.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/celery/django-celery-beat/issues/7#issuecomment-309245941, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGHSS3GUg9ZOtsfBAKtKnX5oVGy6YPHPks5sFF1xgaJpZM4KyW9y.

[PlugaruT]

What is the status of this?

[mheppner]

The Celery team is bringing someone on full-time, so I'd imagine new releases will be published very soon. Please consider donating to the fellowship program if you are able to.

[evasilchenko]

At least some of the issues seems to have been addressed as of https://github.com/celery/celery/releases/tag/v4.1.0. :+1:

We are able to change tasks in the django admin without having to restart celery for those changes to take effect.

[brianmay]

Last time I tested it the fix solves the problem for me. Unfortunately the person who says otherwise isn't giving a lot of details. It is possible they are complaining about unrelated problems.

[Pauldic]

It is NOT a good idea closing this while the issue still persisted in the new v4.1.0. and even getting worst.

@shared_task(name='core_app.cleaner_scheduler')
def cleaner_scheduler(id=None):
Body....

With:
celery==4.1.0 Django==1.11.2 django-celery-beat==1.0.1

CELERY_BROKER_TRANSPORT = 'redis'
CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Africa/Lagos'
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

However, I have noticed that the Task runs every 14mins irrespective of the interval as can be seen below. Changing the interval or disabling and re-enabling the Task only logs DatabaseScheduler: Schedule changed. with no significant effect

I need to downgrade to the previous with the patch suggested above which has been working for me. I hope I am not alone in this

[2017-10-14 11:42:46,475: INFO/MainProcess] Scheduler: Sending due task Automatic Cleaner (core_app.cleaner_scheduler)
[2017-10-14 11:44:16,631: INFO/MainProcess] Writing entries...
[2017-10-14 11:47:16,965: INFO/MainProcess] Writing entries...
[2017-10-14 11:50:17,287: INFO/MainProcess] Writing entries...
[2017-10-14 11:53:17,554: INFO/MainProcess] Writing entries...
[2017-10-14 11:56:17,854: INFO/MainProcess] Writing entries...
[2017-10-14 11:56:56,477: INFO/MainProcess] Scheduler: Sending due task Automatic Cleaner (core_app.cleaner_scheduler)
[2017-10-14 11:59:21,738: INFO/MainProcess] Writing entries...
[2017-10-14 12:02:22,041: INFO/MainProcess] Writing entries...
[2017-10-14 12:05:22,289: INFO/MainProcess] Writing entries...
[2017-10-14 12:08:22,573: INFO/MainProcess] Writing entries...
[2017-10-14 12:11:06,477: INFO/MainProcess] Scheduler: Sending due task Automatic Cleaner (core_app.cleaner_scheduler)
[2017-10-14 12:11:26,524: INFO/MainProcess] Writing entries...
[2017-10-14 12:14:26,824: INFO/MainProcess] Writing entries...
[2017-10-14 12:17:27,089: INFO/MainProcess] Writing entries...
[2017-10-14 12:20:27,358: INFO/MainProcess] Writing entries...
[2017-10-14 12:23:27,650: INFO/MainProcess] Writing entries...
[2017-10-14 12:25:16,480: INFO/MainProcess] Scheduler: Sending due task Automatic Cleaner (core_app.cleaner_scheduler)
[2017-10-14 12:26:21,067: INFO/MainProcess] Writing entries...

[ydaniels]

problem still persist

[thedrow]

@Pauldic @ydaniels The problem should be fixed in Celery 4.2.

[nvictor]

@thedrow hey, can you point to any commit? i may need to pull from upstream and not wait for a release.

[ydaniels]

Yea you are right @thedrow tested and confirmed 4.2 is gold

[thedrow]

@nvictor https://github.com/celery/celery/pull/3958 Next time take a look at what closed this issue.

[brianmay]

@thedrow Unless I am mistaken, I believe celery/celery#3958 was merged in time for 4.1. Am I mistaken?

[thedrow]

Seems like it but there were other fixes in 4.2 that affect this bug I think. I'll have to check.

[xeor]

Anyone know the ETA on 4.2?

[yasa1987]

Yes when will version 4.2 be released ? celery 4.1 is catastrophic regarding this point...

Our periodic tasks are incontrollable ; they even don't start at the right time but all of them start at 1pm regardless what is scheduled...

[thedrow]

We still have a few more bugs to merge before the release.

[im-n1]

Take your time guys we respect what you do. On the other hand sooner better ;)

[rodrigorodriguescosta]

Same bug here, crontab not working... any news?

[thedrow]

Please follow https://github.com/celery/celery/issues/4387