Docker-airflow: Tasks not starting for Dag triggered through airflow UI (CeleryExecutor)

[britishbadger]

Hi,
I'm using the docker-compose-CeleryExecutor.yml with docker 17.05.0-ce on Linux Mint. When I un-pause any of the example dag's and trigger them via the UI they get marked as running but none of the tasks run. I can trigger the tasks manually in the graph view.

I've found https://issues.apache.org/jira/browse/AIRFLOW-43 and made sure my time zone is UTC but it still isn't working.

Any ideas, does this work for others?

[britishbadger]

This seems to only happen if a dag volume isn't shared amongst all the containers.

[darrenhaken]

@britishbadger Can you give more detail on what you mean? I am facing the same issue

[jkuruzovich]

I also had the same issue with both Celery and Local executor.

[gabrielelanaro]

@britishbadger I'm also having the same issue.

Do you mean that the same dag folder should be shared amongst different containers?

[britishbadger]

Hi,
Sorry for the late reply. This was some time ago but from memory this seemed to be solved by ensuring the dag was available to all containers. In our case we used glusterfs to create a "share" that was mapped to each airflow containers as a dag volume. We had separate containers for scheduler, web ui, exectuors etc.

You could use NFS/network share or just ensure that on deployment of new dags a copy of the dag is copied to each volume.

I don't think this was the only issue. We had to also set loose-ish permissions for each component to pick it up correctly.

Hope this helps.