Sentry: Background workers haven't checked in recently. This can mean an issue with your configuration or a serious backlog in tasks.

[saytik]

It is fresh installation of sentry version 7.7.0. and then i installed 7.8.0-dev.
We see the same erro in both versions:
Background workers haven't checked in recently. This can mean an issue with your configuration or a serious backlog in tasks.

Celery is running and it work with all tasks...
on the link /_health/?full=1 i see :+1:
{"healthy":{"CeleryAppVersionCheck":true,"CeleryAliveCheck":false},"problems":["Background workers haven't checked in recently. This can mean an issue with your configuration or a serious backlog in tasks."]}

what is wrong ? (sorry for my english)

[dcramer]

It means either celerybeat isnt running (-B on workers) or there is a queue backlog

[jslootbeek]

How can I see if there is a backlog? There shouldn't be. I'm getting this error as well, and am running a separate celerybeat as a separate process. (version 7.7.0)

[dcramer]

It depends on the broker:

http://docs.celeryproject.org/en/2.2/userguide/monitoring.html

[saytik]

I was run it with -B and any errors in celery log file.
Then, i have tried to run celery beat as separate process and i get error that proccess can't create log files (by default it tries to create celery beat log files in the dir where you run the process ). So, i fixed dir path and permissions and all is working now

(sorry for my english, may be this will help you)

[jslootbeek]

Thanks. I'm running redis locally.
celerybeat seems to be running fine (no errors anywhere).
celery status also gives an error.
$ sentry celery status
Error: No nodes replied within time constraint.

healtcheck shows this:
{"healthy":{"CeleryAppVersionCheck":true,"CeleryAliveCheck":false},"problems":["Background workers haven't checked in recently. This can mean an issue with your configuration or a serious backlog in tasks."]}

[saytik]

show the log file of celery starting and the log file of celery beat .
Try to add to celery start command
-l DEBUG

[jslootbeek]

CELERY BEAT LOG FILE:

celery beat v3.1.18 (Cipater) is starting.
__ - ... __ - _
Configuration ->
. broker -> redis://localhost:6379/0
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> /tmp/sentry-celerybeat
. logfile -> [stderr]@%DEBUG
. maxinterval -> now (0s)
[2015-09-02 00:03:36,970: DEBUG/MainProcess] Setting default socket timeout to 30
[2015-09-02 00:03:36,971: INFO/MainProcess] beat: Starting...
[2015-09-02 00:03:37,102: DEBUG/MainProcess] Current schedule:





[2015-09-02 00:03:37,103: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2015-09-02 00:03:37,115: DEBUG/MainProcess] beat: Waking up in 5.45 seconds.
[2015-09-02 00:03:42,579: INFO/MainProcess] Scheduler: Sending due task sync-options (sentry.tasks.options.sync_options)
[2015-09-02 00:03:42,587: DEBUG/MainProcess] beat: Synchronizing schedule...
[2015-09-02 00:03:42,593: DEBUG/MainProcess] sentry.tasks.options.sync_options sent. id->6928c01e-ebd0-4e52-a30f-7001b21f6faa
[2015-09-02 00:03:42,594: INFO/MainProcess] Scheduler: Sending due task flush-buffers (sentry.tasks.process_buffer.process_pending)
[2015-09-02 00:03:42,599: DEBUG/MainProcess] sentry.tasks.process_buffer.process_pending sent. id->426dd725-2986-470b-9176-152fb13d63a8
[2015-09-02 00:03:42,600: DEBUG/MainProcess] beat: Waking up in 9.99 seconds.
[2015-09-02 00:03:52,598: INFO/MainProcess] Scheduler: Sending due task sync-options (sentry.tasks.options.sync_options)
[2015-09-02 00:03:52,603: DEBUG/MainProcess] sentry.tasks.options.sync_options sent. id->4f966feb-9e0e-463b-838d-d6e80b194306
[2015-09-02 00:03:52,603: INFO/MainProcess] Scheduler: Sending due task flush-buffers (sentry.tasks.process_buffer.process_pending)
[2015-09-02 00:03:52,608: DEBUG/MainProcess] sentry.tasks.process_buffer.process_pending sent. id->e5885473-90a1-4fc9-8f27-c3d90531e459
[2015-09-02 00:03:52,609: DEBUG/MainProcess] beat: Waking up in 9.99 seconds.
[2015-09-02 00:04:02,609: INFO/MainProcess] Scheduler: Sending due task sync-options (sentry.tasks.options.sync_options)
[2015-09-02 00:04:02,614: DEBUG/MainProcess] sentry.tasks.options.sync_options sent. id->958b80d1-fb4b-4809-8d6f-d418d55fe910
[2015-09-02 00:04:02,615: INFO/MainProcess] Scheduler: Sending due task flush-buffers (sentry.tasks.process_buffer.process_pending)
[2015-09-02 00:04:02,619: DEBUG/MainProcess] sentry.tasks.process_buffer.process_pending sent. id->0f5b93f1-55f1-4fba-b674-ed66c8b2e77f
[2015-09-02 00:04:02,620: DEBUG/MainProcess] beat: Waking up in 3.48 seconds.

CELERYD LOG FILE:

[2015-09-02 00:02:35,410: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2015-09-02 00:02:35,417: DEBUG/MainProcess] | Worker: Building graph...
[2015-09-02 00:02:35,418: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Queues (intra), Pool, Autoscaler, Autoreloader, StateDB, Consumer}
[2015-09-02 00:02:35,431: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2015-09-02 00:02:35,431: DEBUG/MainProcess] | Consumer: Building graph...
[2015-09-02 00:02:35,446: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Agent, Heart, Tasks, Control, event loop}
[2015-09-02 00:02:35,454: DEBUG/MainProcess] | Worker: Starting Pool

startup scripts:

[program:sentry-worker]
directory=/home/jule/.virtualenvs/sentry/
command=/home/jule/.virtualenvs/sentry/bin/sentry --config=/home/jule/.sentry/sentry.conf.py celery worker -l DEBUG
autostart=true
autorestart=true
redirect_stderr=true
killasgroup=true
user=sentry

[program:sentry-beat]
directory=/home/jule/.virtualenvs/sentry/
command=/home/jule/.virtualenvs/sentry/bin/sentry --config=/home/jule/.sentry/sentry.conf.py celerybeat -l DEBUG
autostart=true
autorestart=true
redirect_stderr=true
killasgroup=true
user=sentry

.sentry.conf.py: http://dpaste.com/0WSGRQ4

[saytik]

try to remove file /tmp/sentry-celerybeat , restart celery beat and check

i have default configurations working
SENTRY_REDIS_OPTIONS = {
'hosts': {
0: {
'host': 'localhost',
'port': 6379,
}
}
}
SENTRY_CACHE = 'sentry.cache.redis.RedisCache'
CELERY_ALWAYS_EAGER = False
BROKER_URL = 'redis://localhost:6379'
SENTRY_RATELIMITER = 'sentry.ratelimits.redis.RedisRateLimiter'
SENTRY_BUFFER = 'sentry.buffer.redis.RedisBuffer'

any another variables like CELERY_*

[saytik]

have you solved your problem ?

[jslootbeek]

No. I disabled celery
On Fri, Sep 4, 2015 at 03:53 saytik [email protected] wrote:

have you solved your problem ?

—
Reply to this email directly or view it on GitHub
https://github.com/getsentry/sentry/issues/1810#issuecomment-137671919.

[dcramer]

Disabling Celery literally breaks Sentry. Do not do this if you want things to work at all.

[jslootbeek]

The worker is running, I don't get the error any longer, but I still get
this:

sentry celery status
Error: No nodes replied within time constraint.

not sure what specific change fixed this. i'll see if i can find out.

messages are coming through.

On Fri, Sep 4, 2015 at 1:14 PM, David Cramer [email protected]
wrote:

Disabling Celery literally breaks Sentry. Do not do this if you want
things to work at all.

—
Reply to this email directly or view it on GitHub
https://github.com/getsentry/sentry/issues/1810#issuecomment-137796057.

_Jule Slootbeek_ | @jslootbeek http://www.twitter.com/jslootbeek | 617
901 6958

[sam95]

You can solve the above issue by running two commands given on the sentry page

SENTRY_CONF=/etc/sentry sentry celery worker
SENTRY_CONF=/etc/sentry sentry celery beat

In two different terminals, alternatively check if celery is running in the background

SENTRY_CONF=/etc/sentry sentry celery status

[sabrehagen]

I am experiencing this issue also. I'm trying to run your given command in a sentry-onpremise docker container but getting the error that celery is not a command. Am I executing this in the wrong place?

/usr/src/sentry # SENTRY_CONF=/etc/sentry sentry celery status                                                                                                                                                                                                                                                                                           

Usage: sentry [OPTIONS] COMMAND [ARGS]...
Error: No such command "celery".

[mattrobenolt]

Any docs that reference sentry celery are wrong and outdated. It should be sentry run worker and sentry run cron.

[mattrobenolt]

I just said, use sentry run worker and sentry run cron. Once they're running, the message should go away within a few minutes.

[dcramer]

Just to stop this email chain, you need to refer to our docs. Choosing to run sentry yourself means you're stuck dealing with running it. We try to help when we can, but GitHub isn't a support channel. I highly recommend just choosing our SaaS service if you can as Sentry is fairly complex infrastructure (and only getting more so as we expand features).

[yangcvo]

Do not know where the problem, who has solved?

[yangcvo]

Background workers haven't checked in recently. It seems that you have a backlog of 89 tasks. Either your workers aren't running or you need more capacity.

[saytik]

I have virtualenv installed in /home/sentry/ and configs in /home/sentry/.sentry
I have supervisor installed on my server with the following confguration for sentry celery
file /etc/supervisor/conf.d/sentry.conf:

[program:sentry-worker]
directory=/home/sentry/
environment=SENTRY_CONF="/home/sentry/.sentry"
command=/home/sentry/bin/sentry run worker -c 4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=syslog
stderr_logfile=syslog
user=sentry
group=sentry

[program:sentry-cron]
directory=/home/sentry/
environment=SENTRY_CONF="/home/sentry/.sentry"
command=/home/sentry/bin/sentry run cron
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=syslog
stderr_logfile=syslog
user=sentry
group=sentry

[yangcvo]

My supervisord.conf configuration
After the completion of execution or the problem: Background workers haven't checked in recently. It seems that you have a backlog of Either your workers aren't 127 tasks. running or you need more capacity.

;[include]
;files = relative/directory/*.ini
[program:sentry-web]
directory=/home/sentry/
environment=SENTRY_CONF="/etc/sentry"
command=/home/sentry/bin/sentry start

celerybeat -l DEBUG

autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=syslog
stderr_logfile=syslog

[program:sentry-worker]
directory=/home/sentry/
environment=SENTRY_CONF="/etc/sentry"
command=/home/sentry/bin/sentry run worker -c 4

celerybeat -l DEBUG

autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=syslog
stderr_logfile=syslog

[program:sentry-cron]
directory=/home/sentry/
environment=SENTRY_CONF="/etc/sentry"
command=/home/sentry/bin/sentry run cron

celerybeat -l DEBUG

autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=syslog
stderr_logfile=syslog

[yangcvo]

(sentry) [root@search_01 ~]# vim /home/sentry/etc/supervisord.conf
(sentry) [root@search_01 ~]# 1443:M 09 Mar 18:52:39.009 * 10 changes in 300 seconds. Saving...
1443:M 09 Mar 18:52:39.010 * Background saving started by pid 16845
16845:C 09 Mar 18:52:39.040 * DB saved on disk
16845:C 09 Mar 18:52:39.040 * RDB: 8 MB of memory used by copy-on-write
1443:M 09 Mar 18:52:39.111 * Background saving terminated with success

[yangcvo]

tailf -1000 /tmp/supervisord.log
2017-03-09 18:53:47,250 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:53:48,252 INFO spawned: 'sentry-worker' with pid 17057
2017-03-09 18:53:49,254 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-03-09 18:53:50,316 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:53:51,319 INFO spawned: 'sentry-worker' with pid 17065
2017-03-09 18:53:52,321 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-03-09 18:53:53,432 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:53:54,434 INFO spawned: 'sentry-worker' with pid 17073
2017-03-09 18:53:55,436 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-03-09 18:53:56,531 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:53:57,533 INFO spawned: 'sentry-worker' with pid 17081
2017-03-09 18:53:58,535 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-03-09 18:53:59,599 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:54:00,602 INFO spawned: 'sentry-worker' with pid 17093
2017-03-09 18:54:01,604 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-03-09 18:54:02,691 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:54:03,693 INFO spawned: 'sentry-worker' with pid 17113
2017-03-09 18:54:05,373 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-03-09 18:54:05,791 INFO exited: sentry-worker (exit status 1; not expected)
2017-03-09 18:54:06,793 INFO spawned: 'sentry-worker' with pid 17123
2017-03-09 18:54:07,795 INFO success: sentry-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Thank you very much，
But look at your configuration with my side or a little problem: sentry-worker PID process has been switching.

[dcramer]

Our issue tracker is not a support forum and I'm locking this thread to stop further noise to maintainers.