Docker-airflow: Running on Kubernetes

[tomazzaman]

This is actually not an issue, just a few gotchas I've encountered when installing this image on Kubernetes that I'd like to share with anyone trying to accomplish the same thing.

The problem with the image is that on every run, it will try to update the airflow.cfg, which I would recommend not to (because it will most likely break). Instead, open the original file and manually edit the details, then create a GlusterFS cluster with an airflow volume (and the cfg file inside), then mount it onto pods under /usr/local/airflow.

Another issue I stumbled upon is that workers and webserver prefer to be on the same logical host, which is not an issue, we can just schedule multiple containers in the same pod.

The final issue is the entrypoint.sh because, as I've written above, it tries to configure our stack _every_ time. To avoid this, we can set our own command in the pod configuration, like this:

# this is inside "containers" key in pod definition
  - command:
    - /bin/bash
    - -ec
    - |
      $(which pip) install --user -r /usr/local/airflow/requirements.txt
      exec airflow worker

So here are the two yaml files I'm using to successfully run our stack, based on this image.

The main deployment that runs three pods, webserver, worker and scheduler:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: airflow
  name: airflow
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: airflow
    spec:
      containers:
      - command:
        - /bin/bash
        - -ec
        - |
          $(which pip) install --user -r /usr/local/airflow/requirements.txt
          exec airflow webserver
        env:
        - name: LOAD_EX
          value: "y"
        - name: AIRFLOW_HOME
          value: /usr/local/airflow
        image: puckel/docker-airflow:1.8.1
        imagePullPolicy: Always
        name: airflow
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - mountPath: /usr/local/airflow
          name: glusterfs
      - command:
        - /bin/bash
        - -ec
        - |
          $(which pip) install --user -r /usr/local/airflow/requirements.txt
          exec airflow scheduler
        env:
        - name: LOAD_EX
          value: "y"
        - name: AIRFLOW_HOME
          value: /usr/local/airflow
        image: puckel/docker-airflow:1.8.1
        imagePullPolicy: Always
        name: scheduler
        volumeMounts:
        - mountPath: /usr/local/airflow
          name: glusterfs
      - command:
        - /bin/bash
        - -ec
        - |
          $(which pip) install --user -r /usr/local/airflow/requirements.txt
          exec airflow worker
        env:
        - name: LOAD_EX
          value: "y"
        - name: AIRFLOW_HOME
          value: /usr/local/airflow
        image: puckel/docker-airflow:1.8.1
        imagePullPolicy: Always
        name: worker
        ports:
        - containerPort: 8793
          protocol: TCP
        volumeMounts:
        - mountPath: /usr/local/airflow
          name: glusterfs      
      restartPolicy: Always
      volumes:
      - glusterfs:
          endpoints: glusterfs
          path: /airflow
        name: glusterfs

And the (optional) flower deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: flower
  name: flower
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flower
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: flower
    spec:
      containers:
      - command:
        - /bin/bash
        - -ec
        - |
          $(which pip) install --user -r /usr/local/airflow/requirements.txt
          exec airflow flower
        env:
        - name: AIRFLOW_HOME
          value: /usr/local/airflow
        - name: FLOWER_PORT
          value: "5555"
        image: puckel/docker-airflow:1.8.1
        imagePullPolicy: Always
        name: flower
        ports:
        - containerPort: 5555
          protocol: TCP
        volumeMounts:
        - mountPath: /usr/local/airflow
          name: glusterfs
      volumes:
      - glusterfs:
          endpoints: glusterfs
          path: /airflow
        name: glusterfs

[skorski]

Thanks for sharing. We are in the process of porting over to kubernetes as well. A few questions for you:

1. Why not use a config map for the airflow.cfg file?
2. Why did you find the working and the webserver needed to be on the same physical host? Was this because of the dns mappings?
3. Did you have any issues with the scheduler restarting and kubernetes entering a crash loop backoff condition?

[tkaymak]

@skorski I had the crashloopbackoff issue and wrote my own shell script that basically triggered the scheduler in a loop but made kubernetes happy. It feels kind of hacky but seems to work

[hporter]

@tomazzaman Thanks - interesting post. We have this running in GKE, though currently only on LocalExecuter as we test things out. Will bear these ideas in mind as we scale up, and feed back anything else we find.

[jkuruzovich]

Would love to see this as a component of this repository project or as a separate one. Been experimenting with a helm chart but it is not working yet.

[rolanddb]

Hi,
There is work in progress at https://github.com/gsemet/charts/tree/airflow to develop a Helm chart.

The pull request in the main Helm chart repository is here: https://github.com/kubernetes/charts/pull/3959

Please give it a try and report any issues.

[javefang]

Just being redirected here from the helm chart https://github.com/helm/charts/tree/master/stable/airflow

Since we are Airflow 1.10.1 now and people are starting to use the KubernetesExecutor. Can we also include airflow[kubernetes] in the Dockerfile by default? That way it will make this image work straight out of the box when using KubernetesExecutor.

I have opened a PR for that
https://github.com/puckel/docker-airflow/pull/285

[javefang]

@jkuruzovich I've found the current stable/airflow:0.9.1 chart not working as well, but I have a PR to fix that. https://github.com/helm/charts/pull/9964

[adyachok]

Just made template file to run Airflow on OpenShift. Tested on OpenShift v3.10 Dedicated.
Template https://github.com/adyachok/incubator-airflow/tree/feature/agyacsok_openshift_integration/openshift
Use slightly modificated puckel docker-airflow image https://cloud.docker.com/u/eandgya/repository/docker/eandgya/os-airflow

[singhania]

I am also running airflow on kubernetes.I am using Azure Kubernetes Services for kubernetes cluster and used the given docker image and pod yamls to launch airflow.And it works fine.Only issue i am facing is no worker pods are getting launched after executor parallelism is reached even though all the worker pods are in completed status.