Charts: [stable/airflow] Errno 13 - Permission denied: '/opt/airflow/logs/scheduler

[axk7812]

Describe the bug
When mounting persistent volume for logs - encountering Permission denied: '/opt/airflow/logs/scheduler due to mismatch in uid:gid permissions in logs directory. Airflow user runs as 50000 and /opt/airflow/logs is airflow:root (uid:gid).

Version of Helm and Kubernetes:
Helm: v3.0.2
Kubernetes: v1.17.0

Which chart:
stable/airflow

What happened:

After installing and configuring Persistent Volume (AWS EFS) storage for logs - airflow is unable to write to /opt/airflow/logs
This is visible during this initial startup script.

*** executing Airflow initdb...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler
    result = factory(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
    os.makedirs(self._get_log_directory())
  File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 374, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
    raise e
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 63, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

What you expected to happen:
Logs should be stored in persistent volume

How to reproduce it (as minimally and precisely as possible):
In this post I share my values.yaml as well as the yaml to create the PV and PVC. Again I'm using EFS storage on AWS.

helm pull stable/airflow --untar=true

Replace the values.yaml with the one in above linked post (after creating PV/PVC from provided yaml)

helm install airflow airflow/ --values airflow/values.yaml --namespace airflow

Anything else we need to know:
I've tried debugging by editing the scheduler-deployment.yaml file and adding some commands in the section where airflow initdb is called. When I create a new directory it works. The uid:gid for this directory is airflow:airflow. When I see who owns /opt/airflow/logs it's airflow:root. Also, when I mount the external volume it gets mounted as 1000:1000. I've tried changing the RUN_AS_USER config but apparently it doesn't work if it's not 50000. So what I don't understand is howcome the startup script running as user 'airflow' is not allowed to create files in a directory with owner airflow? This seems like a bug.

I've tried to set the securityContext for scheduler but this didn't work

  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000

All I got was the error bash: line 2: /home/airflow/airflow_env.sh: Permission denied.

I've also checked the airflow-scheduler deployment and I see that there is a mount for logs:

volumeMounts:
        - mountPath: /opt/airflow/logs
          name: logs-data

And there is a corresponding volume:

volumes:
      - name: logs-data
        persistentVolumeClaim:
          claimName: efs-claim

In the logs section of the helm chart I set enable to true, and set the existing claim as efs-claim. I've also set some configs in the airflow.config section. There are more but these I saw were relevant to maybe the issue.

AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "efs-claim"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"

when I exec into the airflow-scheduler I do not see any log folders created. /opt is empty.

[mykhailoguk]

we are facing the same issue with Permission denied for /opt/airflow/logs/scheduler

[ivanrezic-maistra]

We are facing same problem too

[gj-9lt]

Facing the same issue.

Thing were running perfect when no EFS. Issue occurred when we started using EFS as PV.

[ivanrezic-maistra]

I have solved the issue by calling "chmod -R 777 logs/" from root folder (outside container). Maybe it isn't best solution, but it works.

[gj-9lt]

I have solved the issue by calling "chmod -R 777 logs/" from root folder (outside container). Maybe it isn't best solution, but it works.

how did you manage to do it. I am using helm chart in which I added volume as "/opt/airflow/logs"

[axk7812]

@gj-9lt you can mount a volume to the pod and create a directory at the root directory of the container. Something like /airflow-logs is what I did

[NoyMiranFyber]

How did you manage to solve it? I occur the same error: "PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/dag_processor_manager/dag_processor_manager.log'"

[akuzni2]

I specified the following in the values.yaml file

airflow:
  extraVolumeMounts:
    - name: airflow-efs-volume
      mountPath: /airflow-logs

  extraVolumes:
    - name: airflow-efs-volume
      persistentVolumeClaim:
        claimName: airflow-efs-claim

logs:
  path: /airflow-logs/process-logs

With this you need to have a PVC created airflow-efs-claim or name it whatever you want. Also the log path doesn't have to be /airflow-logs you can name it whatever you want. This mounts a volume at the root of the pod. Now that I'm posting this I'm curious if anyone else who's solved this came up with a different solution?

I don't think this issue should be closed yet, I still think it's a bug because the documentation says that the logs can work if written to /opt/airflow/logs.

Quote from the documentation:

NOTE: it is also possible to persist logs by mounting a PersistentVolume to the log directory (/opt/airflow/logs by default) using airflow.extraVolumes and airflow.extraVolumeMounts.

I feel either the documentation needs to get updated and reflect that writing to this path won't work and to use a different one, or the permissions issue should be solved.

[gj-9lt]

When I tried working with the a normal pod(Which will be created by a root user), everything works fine. But with Airflow, things are not good. I even tried giving full permissions to the directory which I was working with, but no luck.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[akuzni2]

Not stale.

[thesuperzapper]

If this is still an issue, please raise it on the new repo: https://github.com/airflow-helm/charts/tree/main/charts/airflow

The helm/charts repo is being archived soon.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.