Django-celery-beat: Lost connection to MySQL breaks Celery

Created on 4 Dec 2017 · 4Comments · Source: celery/django-celery-beat

Here is stacktrace, which we got at one of our servers:

[2017-12-03 00:00:20,621: ERROR/ForkPoolWorker-6] Task polemarch.main.tasks.tasks.ScheduledTask[c13325d1-4191-4d70-bc27-a8a312e241a7] raised unexpected: OperationalError(2013, 'Lost connection to MySQL server during query')
Traceback (most recent call last):
  File "//opt/polemarch/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "//opt/polemarch/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "//opt/polemarch/lib/python2.7/site-packages/polemarch/main/utils.py", line 583, in wrapper
    return task_cls(*args, **kwargs).start()
  File "//opt/polemarch/lib/python2.7/site-packages/polemarch/main/utils.py", line 607, in start
    return self.run()
  File "polemarch/main/tasks/tasks.py", line 50, in polemarch.main.tasks.tasks.ScheduledTask.run
  File "polemarch/main/models/tasks.py", line 137, in polemarch.main.models.tasks.PeriodicTask.execute
  File "polemarch/main/models/projects.py", line 140, in polemarch.main.models.projects.Project.execute
  File "polemarch/main/models/projects.py", line 122, in polemarch.main.models.projects.Project._send_hook
  File "polemarch/main/models/projects.py", line 57, in polemarch.main.models.projects.Project.get_hook_data
  File "polemarch/main/models/projects.py", line 72, in polemarch.main.models.projects.Project.type
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
    num = len(clone)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
    self._fetch_all()
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
    raise original_exception
OperationalError: (2013, 'Lost connection to MySQL server during query')
Traceback (most recent call last):
  File "//opt/polemarch/lib/python2.7/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "//opt/polemarch/lib/python2.7/site-packages/celery/beat.py", line 634, in run
    self.service.start(embedded_process=True)
  File "//opt/polemarch/lib/python2.7/site-packages/celery/beat.py", line 568, in start
    self.sync()
  File "//opt/polemarch/lib/python2.7/site-packages/celery/beat.py", line 571, in sync
    self.scheduler.close()
  File "//opt/polemarch/lib/python2.7/site-packages/polemarch/main/celery_beat_scheduler.py", line 24, in close
    self).close()
  File "//opt/polemarch/lib/python2.7/site-packages/celery/beat.py", line 352, in close
    self.sync()
  File "//opt/polemarch/lib/python2.7/site-packages/django_celery_beat/schedulers.py", line 233, in sync
    with transaction.atomic():
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/transaction.py", line 184, in __enter__
    connection.set_autocommit(False, force_begin_transaction_with_broken_autocommit=True)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/backends/base/base.py", line 411, in set_autocommit
    self._set_autocommit(autocommit)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 307, in _set_autocommit
    self.connection.autocommit(autocommit)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "//opt/polemarch/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 307, in _set_autocommit
    self.connection.autocommit(autocommit)
  File "//opt/polemarch/lib/python2.7/site-packages/pymysql/connections.py", line 760, in autocommit
    self._send_autocommit_mode()
  File "//opt/polemarch/lib/python2.7/site-packages/pymysql/connections.py", line 777, in _send_autocommit_mode
    self.escape(self.autocommit_mode))
  File "//opt/polemarch/lib/python2.7/site-packages/pymysql/connections.py", line 1071, in _execute_command
    raise err.InterfaceError("(0, '')")
InterfaceError: (0, '')

First exception - it is reaction of our worker at MySQL connection problems. Second is our Beat stacktrace. As you can see pymysql raises InterfaceError at begin of transaction. But in in sync() (django_celery_beat/schedulers.py, line 233) we have such code:

        except DatabaseError as exc:
            # retry later
            self._dirty |= _tried
            logger.exception('Database error while sync: %r', exc)

Retry only at DatabaseError! Other exceptions types raises futher and ruin Beat. It can't execute tasks until manual restart. According https://docs.djangoproject.com/en/1.11/ref/exceptions/#database-exceptions InterfaceError is also one of known exceptions during DB work. And, unfortunately for our server, it is not child of DatabaseError in contrast to other exceptions. So I suggest to add it at except clause too.

Source