Docker-airflow: Schduler not running when using SequentialExecutor.

[sundios]

Hi,

Every time I run airflow using SequentialExecutor, my dag files are empty and there is a message that says: 'The scheduler does not appear to be running.

The DAGs list may not update, and new tasks will not be scheduled.'

Any idea why is this happening?

When I run airflow with LocalExecutor the scheduler works automatically, but there is no way to save files in the dag folder because it lives in usr/local/airflow/dags which I don't know where that is.

Any help to clarify this?

[jtinsky]

+1 on this issue

[realrnjs]

+1 on this issue

[enys]

Hi,

Could you list the command you used to start airflow? Are you using the provided docker-compose files ?

I'm guessing you have two distinct problems here.
Using the LocalExectuor (and will run into the same problem with other modes), you need a bind mount.
If you look at the docker-compose files, you will see :+1:
volumes: - ./dags:/usr/local/airflow/dags
This mounts your local 'dags' folder, from your current workdir, to /usr/local/airflow/dags, in the running container.

Any luck this works for you ?

[pedromachados]

Is this solved by this PR?

https://github.com/puckel/docker-airflow/pull/359

[sundios]

Hi,

It solves my issue of finding my dags folder. But Scheduler doesnt run with SequentialExecutor

[zhaobinnian]

i have the same problem,
"The scheduler does not appear to be running. Last heartbeat was received 54 minutes ago.The DAGs list may not update, and new tasks will not be scheduled."
are you solved it?

[datarods-svc]

Although PR #359 looks to have solved the issue, the fix is not a part of a tagged release yet. The fix doesn't exist in the most recent tagged release which is version 1.10.4.

What I did in the interim is created a Dockerfile as below:

# Upstream: https://github.com/puckel/docker-airflow/blob/1.10.4/Dockerfile
FROM puckel/docker-airflow:1.10.4

# For SequentialExecutor, apply relevant subset of https://github.com/puckel/docker-airflow/pull/359/files for airflow scheduler to run:
USER root
RUN sed -i 's/LocalExecutor/SequentialExecutor/' /entrypoint.sh
USER airflow

ENV EXECUTOR Sequential
HEALTHCHECK CMD "[ -f /usr/local/airflow/airflow-webserver.pid ]"

I then use the image created using this Dockerfile in my docker-compose.yml. The middle section of the above Dockerfile can presumably be removed after updating to 1.10.5.

[bergalytics]

I had the same message about it not running although the scheduler was actually triggering DAGs.
It could be as simple as the start_date being too recent? The DAG needs a full interval after start date to start running. Some sample DAGs have the following start date:

'start_date': airflow.utils.dates.days_ago(0)

This won't allow tasks to run from the scheduler (for me anyway). Changing the start_date to a static datetime (two days ago if it's a daily schedule) solved it for me:

'start_date': datetime(2019, 12, 20)

[neverbeenthisweeb]

yes, simply you need to make your DAGs run , at least, once.

i.e. if you schedule hourly DAG, you need to set it two hours before
if you schedule weekly DAG, you need to set it two weeks before
so does for daily, you need to set it two days before.