Docker-airflow: Error: failed to start container "base": Error response from daemon , Are you trying to mount a directory onto a file (or vice-versa)?

[marclamberti]

Hi guys,
I'm running Airflow with the KubernetesExecutor but I'm stuck with the following error each time I want to trigger a task in a DAG:
Error: failed to start container "base": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/fa95f1f5-133b-11ea-bd9a-0e598fd8e184/volume-subpaths/airflow-config/base/2\\\" to rootfs \\\"/var/lib/docker/overlay2/79f7429ea1e73a1b7360b65de6392b4dc7d112fb655b06ebc208c5edb6207890/merged\\\" at \\\"/var/lib/docker/overlay2/79f7429ea1e73a1b7360b65de6392b4dc7d112fb655b06ebc208c5edb6207890/merged/usr/local/airflow/airflow.cfg\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Here is the full description of the pod:

Name:         paralleldagtask1-ab1056b506b247d18b8d31e8731f8818
Namespace:    airflow-p-ddj8l
Priority:     0
Node:         ip-192-168-100-59.internal/192.168.100.59
Start Time:   Sat, 30 Nov 2019 06:38:11 +0000
Labels:       airflow-worker=e52d1c4d-0763-47dd-82e3-896e28f83cc4
              dag_id=parallel_dag
              execution_date=2019-11-30T06_38_09.973471_plus_00_00
              task_id=task_1
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Failed
IP:           192.168.89.144
IPs:          <none>
Init Containers:
  git-sync-clone:
    Container ID:   docker://ffa6d64d2f341090bedbb4680f3227b3a2464994c2c5f30259c1f4521454c79b
    Image:          gcr.io/google-containers/git-sync-amd64:v2.0.5
    Image ID:       docker-pullable://gcr.io/google-containers/git-sync-amd64@sha256:904833aedf3f14373e73296240ed44d54aecd4c02367b004452dfeca2465e5bf
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 30 Nov 2019 06:38:12 +0000
      Finished:     Sat, 30 Nov 2019 06:38:14 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      GIT_SYNC_REPO:      https://github.com/marclamberti/airflow-dags.git
      GIT_SYNC_BRANCH:    master
      GIT_SYNC_ROOT:      /tmp
      GIT_SYNC_DEST:      dags
      GIT_SYNC_ONE_TIME:  true
    Mounts:
      /usr/local/airflow/airflow.cfg from airflow-config (ro,path="airflow.cfg")
      /usr/local/airflow/dags/ from airflow-dags (rw)
      /usr/local/airflow/logs from airflow-logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dnrvs (ro)
Containers:
  base:
    Container ID:  docker://5dfdc70966ab2168875e8af94b5e9502985b4e338486fd58c8ad5b32e5dcbd6e
    Image:         stibbons31/docker-airflow-dev:2.0dev
    Image ID:      docker-pullable://stibbons31/docker-airflow-dev@sha256:5bc8aae2bd57a53807ccf34422ab0919fad427667931899becafb2f2ca4a1a1c
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -cx
      --
    Args:
      airflow run parallel_dag task_1 2019-11-30T06:38:09.973471+00:00 --local -sd /usr/local/airflow/dags/parallel_dag.py
    State:          Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/fa95f1f5-133b-11ea-bd9a-0e598fd8e184/volume-subpaths/airflow-config/base/2\\\" to rootfs \\\"/var/lib/docker/overlay2/79f7429ea1e73a1b7360b65de6392b4dc7d112fb655b06ebc208c5edb6207890/merged\\\" at \\\"/var/lib/docker/overlay2/79f7429ea1e73a1b7360b65de6392b4dc7d112fb655b06ebc208c5edb6207890/merged/usr/local/airflow/airflow.cfg\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
      Exit Code:    127
      Started:      Sat, 30 Nov 2019 06:38:15 +0000
      Finished:     Sat, 30 Nov 2019 06:38:15 +0000
    Ready:          False
    Restart Count:  0
    Environment:
      AIRFLOW__CORE__DAGS_FOLDER:   /tmp/dags
      AIRFLOW__CORE__EXECUTOR:      LocalExecutor
      AIRFLOW__CORE__AIRFLOW_HOME:  /usr/local/airflow
    Mounts:
      /usr/local/airflow/airflow.cfg from airflow-config (ro,path="airflow.cfg")
      /usr/local/airflow/dags/ from airflow-dags (ro)
      /usr/local/airflow/logs from airflow-logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dnrvs (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  airflow-dags:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  airflow-logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  airflow-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-p-ddj8l-env
    Optional:  false
  default-token-dnrvs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dnrvs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                                   Message
  ----     ------     ----  ----                                                   -------
  Normal   Scheduled  7s    default-scheduler                                      Successfully assigned airflow-p-ddj8l/paralleldagtask1-ab1056b506b247d18b8d31e8731f8818 to ip-192-168-100-59.eu-west-3.compute.internal
  Normal   Pulled     6s    kubelet, ip-192-168-100-59.eu-west-3.compute.internal  Container image "gcr.io/google-containers/git-sync-amd64:v2.0.5" already present on machine
  Normal   Created    6s    kubelet, ip-192-168-100-59.eu-west-3.compute.internal  Created container git-sync-clone
  Normal   Started    6s    kubelet, ip-192-168-100-59.eu-west-3.compute.internal  Started container git-sync-clone
  Normal   Pulled     3s    kubelet, ip-192-168-100-59.eu-west-3.compute.internal  Container image "stibbons31/docker-airflow-dev:2.0dev" already present on machine
  Normal   Created    3s    kubelet, ip-192-168-100-59.eu-west-3.compute.internal  Created container base
  Warning  Failed     3s    kubelet, ip-192-168-100-59.eu-west-3.compute.internal  Error: failed to start container "base": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/fa95f1f5-133b-11ea-bd9a-0e598fd8e184/volume-subpaths/airflow-config/base/2\\\" to rootfs \\\"/var/lib/docker/overlay2/79f7429ea1e73a1b7360b65de6392b4dc7d112fb655b06ebc208c5edb6207890/merged\\\" at \\\"/var/lib/docker/overlay2/79f7429ea1e73a1b7360b65de6392b4dc7d112fb655b06ebc208c5edb6207890/merged/usr/local/airflow/airflow.cfg\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Thanks for your help
Marc

[marclamberti]

Fixed the issues was related to the following line: /usr/local/airflow/airflow.cfg from airflow-config (ro,path="airflow.cfg")

[vikash32]

Fixed the issues was related to the following line: /usr/local/airflow/airflow.cfg from airflow-config (ro,path="airflow.cfg")

@marclamberti how did you fixed it, i am having same issue and banging my head on wall.

[DreadfullyDistinct]

this is caused by the fact that the airflow_configmap parameter is set, but airflow.cfg is not properly configured in Kubernetes configmap. if you have the .cfg already in the image, you can leave the airflow_configmap empty and it wont try mounting it. otherwise configure it in configmap accordingly. here are some examples: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/