Docker-airflow: Airflow trigger_dag uses Sequential Executor instead of CeleryExecutor

[PaulIsProgramming]

I want to start a Dag run when a message arrives at my system via a message bus. I'm using the 'airflow trigger_dag my_dag' command as the trigger mechanism.

To test this, I log into the running webserver and enter the 'airflow trigger_dag my_dag' command. It runs but it uses the SequentialExecutor instead of the configured CeleryExecutor. If I start a Dag via the webserver UI, the CeleryExecutor is used and works as expected.

Am I wrong in assuming that the trigger_dag command uses the same config as the webserver's ; i.e. it uses the configured CeleryExecutor?

What is weird is, that if I start a new worker form the same place (logged into webserver) to see whether that worker uses Celery or the Sequential Executor I get mixed messages (see log below):

airflow@50c03f34eb7a:~$ airflow worker
[2019-02-25 14:37:42,020] {{__init__.py:51}} INFO - Using executor SequentialExecutor

-------------- celery@50c03f34eb7a v4.1.1 (latentcall)

---

--- * * * -- Linux-4.9.125-linuxkit-x86_64-with-debian-9.8 2019-02-25 14:37:42

---

• ** ---------- [config]
• ** ---------- .> app: airflow.executors.celery_executor:0x7f5609caa240
• * ---------- .> transport: amqp://guest:*@rabbit:5672//
• ** ---------- .> results: amqp://
• * --- * --- .> concurrency: 16 (prefork)
  -- *** ---- .> task events: OFF (enable -E to monitor tasks in this worker)

---

-------------- [queues]
.> default exchange=default(direct) key=default

[tasks]
. airflow.executors.celery_executor.execute_command

[2019-02-25 14:37:43,037: INFO/MainProcess] Connected to amqp://guest:**@rabbit:5672//
[2019-02-25 14:37:43,048: INFO/MainProcess] mingle: searching for neighbors
[2019-02-25 14:37:43,162] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2019-02-25 14:37:43,380] {{cli_action_loggers.py:69}} ERROR - Failed on pre-execution callback using
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: log

It clearly says in the 2nd line that the SequentialExecutor is used. It also tries to use the local sqlite DB (which I have not initialized, hence the error). However, the worker name (celery@50c03f34eb7a v4.1.1 (latentcall)) indicates, that it is a celery worker and so do these 2 lines:

• ** ---------- .> app: airflow.executors.celery_executor:0x7f5609caa240
  . airflow.executors.celery_executor.execute_command

SO, which one is it? Is the worker a celery worker or a sequential worker. And following this, does 'airflow trigger_dag' uses the Celery executor or the sequential executor.

Thanks!

[Bendabir]

Hello,
I've been facing the same issue for 1 hour now. After investigation, I just noticed that a big part of the Airflow configuration is done within the entrypoint.sh script. According to the Airflow documentation, the preferred order for the config options are :

1. environment variable
2. configuration in airflow.cfg
3. command in airflow.cfg
4. Airflow’s built in defaults

Therefore, when you spawn directly inside your Airflow container, you're missing a part of the Airflow config. Solution : Execute your command from the entrypoint.sh script (I'm currently testing it).

EDIT : It works. If you want to spawn a correctly configured bash inside your container, run something like docker exec -it <container_name> /entrypoint.sh bash !

Hope it helps.

Ben

[PaulIsProgramming]

That helps a lot! It works now and the log output makes sense again.

Thanks so much.