Charts: [bitnami/airflow] Correct procedure for python package requirements?

[ekhaydarov]

Which chart:
Airflow 4.1.1

Description
It seems like the only way to run python dependencies is to override the image? Is there a suggested method for a project to be able to handle this?

If I want to override the image I cannot install airflow as a subchart because of config issues, ie services do not get configured correctly and then scheduler and server dont register each other. So I am then left with making it a CLI argument on install which is not infrastructure as code.

Am I missing something to be able to make this work? I would just like to have airflow as subchart and it to work with the dags which have requirements and plugins.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.7-gke.2", GitCommit:"96c91b15a936b36701e8704844bc356862440a21", GitTreeState:"clean", BuildDate:"2019-12-13T12:37:28Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

[rafariossaa]

Hi @ekhaydarov ,
Could you give an example of what you want to do ?

[ekhaydarov]

Hi @rafariossaa

if my local Chart.yaml looks like this it would mean airflow would be a subchart.

dependencies:
  - name: airflow
    version: "4.1.1"
    repository: "https://charts.bitnami.com/bitnami"

In order to get the dags into the folder I either make a configmap and shove into it all the dags or i override all the images with my own where i copy all the dags into the correct folders. making values.yaml look like this

  image: &default_image
    registry: "gcr.io"
    repository: "myimage"
    tag: v1
    pullPolicy: Always
  schedulerImage: *default_image
  workerImage: *default_image

I then have to copy the dags and plugins to /opt/bitnami/airflow/[dags,plugins]. However, when running helm install the pods seem to not be able to communicate. the webserver throws the error and the worker never becomes ready.

The scheduler does not appear to be running.

The DAGs list may not update, and new tasks will not be scheduled.

So im curious what the standard approach is to get an airflow instance up and running where there are correct python dependencies installed as well as the scheduler/worker/webserver all being able to communicate properly and have a functional airflow. I can confirm that I am able to helm install bintami/airflow correctly and am able to run example dags. So the issues is in the helm config.

If you would be able to help me run my dags + plugins + extra python packages, one way or another, I would greatly appreciate it

Thanks

Edit: FYI the NOTES.txt information that is necessary to complete the installation is missing. so trying to copy it into the top level NOTES.txt brings about errors like this, which again points to helm config being out of sync and not designed for subchart deployment

coalesce.go:165: warning: skipped value for resources: Not a table.
coalesce.go:199: warning: destination for annotations is a table. Ignoring non-table value <nil>
Error: UPGRADE FAILED: template: local-dags/charts/airflow/templates/_helpers.tpl:206:41: executing "airflow.postgresql.fullname" at <.Values.postgresql.nameOverride>: nil pointer evaluating interface {}.nameOverride

[ekhaydarov]

Ok after further investigation it seems that each part of airflow, ie scheduler, webserver and worker, has a separate image. It seems impractical to rebuild all 3 images every time you change a bit of code. There is no option to mount another volume only a predefined configmap or git repo. Therefore mounting your own pvc is off the table

Again I am confused what is bitnamis approach to develop airflow dags with a quick turnaround time when you have dags + plugins + extra python package dependencies?

[dani8art]

Hi @ekhaydarov

As I understand you want to have the possibility to pass your DAGs, plugins and python dependencies to the chart, right?

• DAGs: currently you can pass them through a git repo using:

airflow:
   cloneDagFilesFromGit:
     enabled: false
     # repository:
     # branch:
     # interval:
     # path:

• Regarding plugins and python dependencies, I understand that they are installed through UI/CLI, in this case, it will be persisted. Currently, we don't support extra PVC / ConfigMap, nevertheless, we are open to study/consider some suggestions. What would it be your use case?

  • Would you need to be able to add DAGs through ConfigMap?
  • Would you need to be able to add Plugins through ConfigMap? Is it possible to do regarding airflow plugins installation process?
  • Would you need to be able to add python dependencies? Where should we add inside a container? are these dependencies installed with plugins?

Regards.

[ekhaydarov]

Hi @dani8art

The main issue of passing the dag through dit repo is that its not very useful for local development, therefore committing to git for each minor changes is a barrier. However, I understand thats definitely not a priority for your team. I am ok with configmap syncing in this case.

Plugins can be added exactly the same way but there is no integration there. This is not difficult either to work around from my side.

Would you need to be able to add DAGs through ConfigMap?

I can already do that in your helm chart.

Would you need to be able to add Plugins through ConfigMap? Is it possible to do regarding airflow plugins installation process?

Any way to install plugins through your helm chart would be great. I am not sure what you mean by your second question. Its as simple as adding the dags through configmap

Would you need to be able to add python dependencies?

Of course, is there a project that has no dependencies? Installing dependencies from UI/CLI after is not very good for CI/CD. IT would be great if anyone can spin up everything just by running helm install, dependencies included. Manual state management is not ideal.

Where should we add inside a container?

This is the main issue. bitnami/airflow has 3 separate images for airflow. scheduler,webserver and worker. I could make it work with overriding the image manually if it was just one image used, but its 3 different images.

Most of the issues are minor. My main concern is the way dependencies are being installed. I have since moved to running my own airflow and using the one image provided by puckel/docker-airflow since image to run all three scheduler,webserver and worker. Much easier to control and manage dependencies etc. I hope this feedback helps your development of this chart and doesnt seem like a criticism.

Thanks

[dani8art]

Hi @ekhaydarov

Thank you so much for your extensive and clear answer and sorry for my late response, I was evaluating and checking your proposals.

There is no doubt that it will be very useful and will help us to improve this chart. It seems that your request makes all the sense, I just wanted to understand better what your necessities were in order to fit them properly. We are going to open an internal task for the team to evaluate including these new features in the chart.

I hope we can work on it ASAP, so we can provide these features to you. Thanks for your understanding and patience.

Regards.

[austinbv]

@ekhaydarov I noticed this issue so it may help you https://github.com/bitnami/charts/issues/2588

[dani8art]

Yes @austinbv thank you for pointing it out, please refer to https://github.com/bitnami/charts/issues/2588#issuecomment-629093931 to solve the issue

[austinbv]

To add - this PR when merged should help you as well https://github.com/bitnami/charts/pull/2595

[ekhaydarov]

nice. great work! look forward to trying these out when they're live

[austinbv]

We have been trying both the bitnami and the stable helm release for airflow and I will say, normally our preference is for bitnami but in this case the non bitnami chart is a lot more feature complete.

Things like git ssh keys, pip installations and tracking the official airflow image all are part of the other chart.

[dani8art]

Hi @austinbv , recently we added some of these features to our chart plus some other extra features that make it more customizable than ever. e.g: extraEnvVars, extraVolumeMounts, extraEnvVarsCM or extraEnvVarsSecret.

Most of the pending one from your list could be cover using these features.

Of course, we have the bandwidth to improve and all the suggestions are very welcome, thank you so much for your feedback!

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.