Docker-airflow: Why one container for webserver and scheduler with LocalExecutor

[infused-kim]

First of all, thank you so much for creating this image. It's working great.

I have only recently started using docker and I am still learning the best practice, so please forgive me newbie question.

I am wondering, why did you decide to let the webserver and scheduler share on container when using the LocalExecutor? I though it would be best to always use separate containers for all processes?

Thanks, Kim

[tiagovrtr]

LocalExecutor uses the multiprocessing Python library to create processes run locally, and for this reason, it is a single-host deployment and it is not recommended for production systems.

[infused-kim]

I see, thank you. But doesn't this only mean that the scheduler executes all threads on the same machine / docker container?

My understanding is that the scheduler and webserver are not connected and don't need to be on the same machine / docker container.

So I am wondering why the choice was made to run them in the same container.

Once again, I am just a beginner and I am trying to understand the design of this docker setup to learn how to create my own in the future. I'm not trying to criticize or change anything :)

[tiagovrtr]

It's probably due to this

[grahovam]

I'm using LocalScheduler with a separate container running the webserver and had no issues so far. I havent checked if there's any issues with logs though.

I had to change the entrypoint.sh to be able to run the container without the scheduler.

Edit: There was indeed a problem with the logs, I already fixed it.

The webserver can't find the base-logs when they're not in the container, since they are being written by the scheduler. If you try to read the log of a task instance, you get the following message:

*** Log file does not exist: /usr/local/airflow/logs/dag/task/2019-05-13T02:55:00+00:00/1.log *** Fetching from: http://host:8793/log/dag/task/2019-05-13T02:55:00+00:00/1.log *** Failed to fetch log file from worker. HTTPConnectionPool(host='host', port=8793): Max retries exceeded with url: /log/dag/task/2019-05-13T02:55:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f172c9c1240>: Failed to establish a new connection: [Errno 111] Connection refused',))

You can fix this error by mounting the logs folder into the container.

[sys-temd]

@grahovam Can you post what changes have you made to entrypoint.sh to separate webserver container from scheduler in LocalExecutor? I'm in the same situation and this might be the only solution I've been searching for few hours.

[grahovam]

@sys-temd Sorry, I don't remember :) It's been a year ago and I don't run Airflow on Docker anymore. But as far as I remember, Airflow has published an official Docker image. Probably you should check this out :)

[infused-kim]

Yeah the new official docker image doesn't combine scheduler and webserver. I have documented my process of migrating from puckel's here:
https://github.com/puckel/docker-airflow/issues/536#issuecomment-626333283