Docker-airflow: Running airflow using CeleryExecutor expects MysqlDB

[tripti-singh]

While using CeleryExecutor to run workflows, some tasks complete while others stay in Queued mode.
Logs seem to point out that scheduler is looking for Mysql connection:
[2018-03-21 22:23:36,641] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-21 22:23:36,640] {{celery_executor.py:101}} ERROR - Error syncing the celery executor, ignoring it:
[2018-03-21 22:23:36,641] {{base_task_runner.py:98}} INFO - Subtask: [2018-03-21 22:23:36,640] {{celery_executor.py:102}} ERROR - No module named 'MySQLdb'
[2018-03-21 22:23:36,641] {{base_task_runner.py:98}} INFO - Subtask: Traceback (most recent call last):
[2018-03-21 22:23:36,641] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 83, in sync
[2018-03-21 22:23:36,641] {{base_task_runner.py:98}} INFO - Subtask: state = async.state
[2018-03-21 22:23:36,641] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 431, in state
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: return self._get_task_meta()['status']
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 370, in _get_task_meta
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: return self._maybe_set_cache(self.backend.get_task_meta(self.id))
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 352, in get_task_meta
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: meta = self._get_task_meta_for(task_id)
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
[2018-03-21 22:23:36,642] {{base_task_runner.py:98}} INFO - Subtask: return fun(args, kwargs)
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 122, in _get_task_meta_for
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: session = self.ResultSession()
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: *self.engine_options)
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 58, in session_factory
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: engine, session = self.create_session(dburi, *kwargs)
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 44, in create_session
[2018-03-21 22:23:36,643] {{base_task_runner.py:98}} INFO - Subtask: engine = self.get_engine(dburi, *kwargs)
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/celery/backends/database/session.py", line 41, in get_engine
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: return create_engine(dburi, poolclass=NullPool)
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 424, in create_engine
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: return strategy.create(args, *kwargs)
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 81, in create
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: dbapi = dialect_cls.dbapi(dbapi_args)
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 102, in dbapi
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: return __import__('MySQLdb')
[2018-03-21 22:23:36,644] {{base_task_runner.py:98}} INFO - Subtask: ModuleNotFoundError: No module named 'MySQLdb'

Is this a known issue or is there some missing config?
Any pointers would be great.

[smaley07]

What do you have in your Airflow.cfg ?

You should have something like :
celery_result_backend = db+postgresql://airflow:airflow@postgres/airflow
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflow

[RyanStack]

I am experiencing this issue as well. @tripti-singh I think we both may have had a previous airflow.cfg file with result_backend. I adjusted this config to celery_result_backend and the issue resolved.

And this was resolved in https://github.com/puckel/docker-airflow/commit/45fc751c48662f714e471dccba3f093422e4acdc

[tripti-singh]

Thank you @smaley07 @RyanStack . I guess i didn't have the above fix. It is working as expected now after I replaced result_backend with celery_result_backend

[y2k-shubham]

For late-comers, the result_backend property in airflow.cfg was moved to entrypoint.sh script here

[praneel8094]

Thanks @y2k-shubham

[MJSheth]

I tried to make use of celery_result_backend instead of result_backend in airflow.cfg but it is still not using configured DB

Same DB config works for Local and Sequential Executor with result_backend in airflow.config