Charts: [stable/airflow] volumeMounts[0].name: Not found: "git-clone"`

[Raffiki]

Describe the bug
I'm trying to install Airflow with custom dag repo but I encounter following exception:

helm install airflow6 stable/airflow --version 6.0.0 --set dags.git.gitSync.enabled=true --set dags.git.url=https://github.com/x/airflow-dag --set dags.git.gitSync.refreshTime=60

Error: Deployment.apps "airflow6-web" is invalid: spec.template.spec.containers[0].volumeMounts[0].name: Not found: "git-clone"

Version of Helm and Kubernetes:
Helm: version.BuildInfo{Version:"v3.0.3", GitCommit:"ac925eb7279f4a6955df663a0128044a8a6b7593", GitTreeState:"clean", GoVersion:"go1.13.6"}

K8: kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
`

> helm repo list
NAME    URL
stable  https://kubernetes-charts.storage.googleapis.com/

Which chart:

stable/airflow

[appunni-dishq]

I am facing exactly same issue

[paulobezerr]

It happens with me.
Activate initContainer solves my problem.

[appunni-dishq]

But I don't want initContainer I want sidecar git sync, init container causes another issue with race condition on clone, I am using WriteMany NFS disk.

[appunni-dishq]

Hack solution for now

  extraVolumes:
    - name: git-clone
      configMap:
            name: {{deployment_name}}-git-clone
            defaultMode: 0755
    - name: git-clone-secret
      secret:
            secretName: airflow-secrets
            defaultMode: 0700

Issue is at line 255 https://github.com/helm/charts/blob/5b4091af17df57711f24b2b3a27a4a59fad351e2/stable/airflow/templates/deployments-scheduler.yaml

git clone is under initContainer.isEnabled so if you are setting it false, then that volume is not defined. This solves that problem

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[paulobezerr]

Hi there, we are waiting for approval. :)

[whatnick]

Ran into this bump !!

[appunni-dishq]

I actually would suggest stop using this solution. This ran into many issues as each side car tries to git sync, especially using NFS this can lead to race conditions fast. Instead if you want NFS with shared read, I have setup separate git-sync deployment. Which is the only pod doing gitSync and rest of the machines are reading the files in shared fashion. This was much better setup and had the least issues.

[appunni-dishq]

git-sync.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow-git-sync
  namespace: pipelines
spec:
  replicas: 1
  selector:
    matchLabels:
      name: airflow-git-sync
  template:
    metadata:
      labels:
        name: airflow-git-sync
    spec:
      volumes:
      - name: markdown
        persistentVolumeClaim:
          claimName: airflow-dags
      - name: git-secrets
        secret:
          secretName: airflow-secrets
      containers:
      - name: git-sync
        image: k8s.gcr.io/git-sync:v3.1.3
        volumeMounts:
        - name: markdown
          mountPath: /tmp/git
        - name: git-secrets
          mountPath: /etc/git-secrets
        env:
        - name: GIT_SYNC_REPO
          value: git@host:account/project.git
        - name: GIT_SYNC_DEST
          value: git-sync
        - name: GIT_SYNC_SSH
          value: "true"
        - name: GIT_SSH_KNOWN_HOSTS_FILE
          value: /etc/git-secrets/known_hosts
        - name: GIT_SSH_KEY_FILE
          value: /etc/git-secrets/id_rsa
        - name: GIT_SYNC_WAIT
          value: "1"
        resources:
          limits:
            memory: 20Mi
            cpu: 150m
          requests:
            memory: 15Mi
            cpu: 80m

airflow config in values

airflow:
  extraConfigmapMounts: []

  ## Extra environment variables to mount in the web, scheduler, and worker pods:
  extraEnv:
    - name: AIRFLOW__CORE__FERNET_KEY
      valueFrom:
        secretKeyRef:
          name: airflow-secrets
          key: fernet_key
    - name: AIRFLOW__WEBSERVER__RBAC
      value: "true"
    - name: AIRFLOW__WEBSERVER__AUTHENTICATE
      value: "true"
    - name: AIRFLOW__CORE__DAGS_FOLDER
      value: "/usr/local/airflow/dags/git-sync"
    - name: AIRFLOW__CORE__SQL_ALCHEMY_POOL_SIZE
      value: "16"
    - name: AIRFLOW__CORE__SQL_ALCHEMY_MAX_OVERFLOW
      value: "32"

This is using the shared nfs airflow-dags

[paulobezerr]

Fixed by c22bd37bc0b4fde003a5c5a46bff5935cf9bc0b2.

I think that we could close it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.