Docker-airflow: Failed to fetch logs from worker on CeleryExecutor

[yampelo]

*** Log file does not exist: /usr/local/airflow/logs/$dag_id$/$task_id$/2019-01-18T16:41:33.469813+00:00/4.log
*** Fetching from: http://04e6b0f20a57:8793/log/$dag_id$/$task_id$/2019-01-18T16:41:33.469813+00:00/4.log
*** Failed to fetch log file from worker. 404 Client Error: Not Found for url: http://04e6b0f20a57:8793/log/$dag_id$/$task_id$/2019-01-18T16:41:33.469813+00:00/4.log

I'm getting the above ever since upgrading to the 1.10.1 docker compose.

[zhongjiajie]

I had miss same issue, I still don not understand that, Is anyone know why?

[Kiddinglife]

Try to use shared file system like s3 efs if you running on multi host. in airflow.cfg, you can specify the shared file system.

[yampelo]

I'm running with docker --scale

[yampelo]

I looked into this a bit, it looks like the docker containers aren't able to resolve the other's aliases via DNS. Is there something in the celery compose that's missing? maybe a network setting?

[yampelo]

I found the issue. I had an HTTP Proxy set at the container level, once i unset it, logs are now visible again.

[jiayunpeng]

I'm using non-docker airflow , I had miss same issue.
In my case , the airflow serve_logs is down

nohup airflow serve_logs >> service_logs/airflow/airflow_serve_logs.log 2>&1 &