Argo-cd: Map helm lifecycle hooks to ArgoCD pre/post/sync hooks

Created on 7 Jul 2018 · 24Comments · Source: argoproj/argo-cd

The anchor feature in ksonnet v0.12 is the support of helm registries and chart installations. e.g.:

ks registry add helm-stable https://kubernetes-charts.storage.googleapis.com
ks pkg install helm-stable/redis
ks generate helm-stable-redis hc1

After we upgrade ArgoCD to use ksonnet v0.12, and do a ks show, we will start getting helm hook resources. Since ArgoCD is handling the apply, we will need to build in helm knowledge to skip over hooks during the sync (ksonnet is doing this as well during a ks apply).

Additionally helm supports the following hooks:
https://github.com/kubernetes/helm/blob/master/docs/charts_hooks.md

pre-install: Executes after templates are rendered, but before any resources are created in Kubernetes.
post-install: Executes after all resources are loaded into Kubernetes
pre-delete: Executes on a deletion request before any resources are deleted from Kubernetes.
post-delete: Executes on a deletion request after all of the release's resources have been deleted.
pre-upgrade: Executes on an upgrade request after templates are rendered, but before any resources are loaded into Kubernetes (e.g. before a Kubernetes apply operation).
post-upgrade: Executes on an upgrade after all resources have been upgraded.
pre-rollback: Executes on a rollback request after templates are rendered, but before any resources have been rolled back.
post-rollback: Executes on a rollback request after all resources have been modified.
crd-install: Adds CRD resources before any other checks are run. This is used only on CRD definitions that are used by other manifests in the chart.

As part of an ArgoCD sync, at the minimum, these resources should be ignored. It can be a feature to map the helm hooks as ArgoCD hooks. (e.g. pre-upgrade and post-upgrade could map to PreSync and PostSync).

enhancement

Source

jessesuen

👍10

Most helpful comment

Here's where I stand on this, after studying the use of these hooks in public helm charts.

For a first pass at this, I think the following mapping might solve 70% of peoples use cases:

It's not a 100% clean/accurate mapping, but based on what we observe in public helm charts, it nearly matches the intent of these chart hooks. Even though the mapping is not clean, the key problem it avoids, is introducing application life-cycle state into Argo CD.

jessesuen on 25 Jul 2019

👍5

All 24 comments

Also need to exclude test hooks:
https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md

jessesuen on 11 Jul 2018

It turns out helm hooks are not very widely used, at least not in the public charts. Here are the charts I found using it:

$ grep -r 'helm.sh/hook' . | grep -v test-success | grep -v hook-weight | grep -v hook-delete-policy | grep -v crd-install
./stable/tensorflow-notebook/templates/secrets.yaml:    "helm.sh/hook": pre-install,pre-upgrade
./stable/etcd-operator/templates/etcd-cluster-crd.yaml:    "helm.sh/hook": "post-install"
./stable/etcd-operator/templates/backup-etcd-crd.yaml:    "helm.sh/hook": "post-install"
./stable/etcd-operator/templates/restore-etcd-crd.yaml:    "helm.sh/hook": "post-install"
./stable/prometheus-operator/templates/prometheus-operator/cleanup-crds.yaml:    "helm.sh/hook": pre-delete
./stable/kong/templates/migrations-on-upgrade.yaml:    helm.sh/hook: "pre-upgrade"
./stable/spark-history-server/templates/cleanup-job.yaml:    "helm.sh/hook": pre-delete
./stable/minio/templates/post-install-create-bucket-job.yaml:    "helm.sh/hook": post-install,post-upgrade
./stable/influxdb/templates/post-install-set-auth.yaml:    "helm.sh/hook": post-install
./stable/spinnaker/templates/hooks/install-using-hal.yaml:    "helm.sh/hook": "post-install,post-upgrade"
./stable/spinnaker/templates/hooks/expose-nodeports.yaml:    "helm.sh/hook": "post-install, post-upgrade"
./stable/spinnaker/templates/hooks/cleanup.yaml:    "helm.sh/hook": "pre-delete"
./stable/dex/templates/job-web-certs.yaml:    "helm.sh/hook": post-install
./stable/dex/templates/job-grpc-certs.yaml:    "helm.sh/hook": post-install
./stable/dex/templates/config-openssl.yaml:    "helm.sh/hook": post-install
./stable/sumologic-fluentd/templates/secrets.yaml:    "helm.sh/hook": pre-install,pre-upgrade
./stable/tensorflow-serving/templates/pvc.yaml:    "helm.sh/hook": pre-install
./stable/ark/templates/hook-deploy.yaml:    "helm.sh/hook": post-install
./stable/ark/templates/hook-delete.yaml:    "helm.sh/hook": pre-delete
./stable/mission-control/templates/hooks/create-user.yaml:    "helm.sh/hook": post-install
./stable/sentry/templates/hooks/db-init.job.yaml:    "helm.sh/hook": post-install
./stable/sentry/templates/hooks/user-create.job.yaml:    "helm.sh/hook": post-install
./stable/sumokube/templates/secrets.yaml:    "helm.sh/hook": pre-install,pre-upgrade
./stable/sumokube/templates/config.yaml:    "helm.sh/hook": pre-install,pre-upgrade
./stable/traefik/templates/storeconfig-job.yaml:    "helm.sh/hook": post-install
./incubator/zookeeper/templates/job-chroots.yaml:    "helm.sh/hook": post-install,post-upgrade
./incubator/sparkoperator/templates/webhook-cleanup-job.yaml:    "helm.sh/hook": pre-delete
./incubator/sparkoperator/templates/webhook-init-job.yaml:    "helm.sh/hook": post-install
./incubator/common/README.md:    "helm.sh/hook": "pre-install"
./incubator/common/README.md:"helm.sh/hook": "pre-install,post-install"
./incubator/common/templates/_metadata_annotations.tpl:"helm.sh/hook": {{printf "%s" . | quote}}

Deprioritizing this.

jessesuen on 21 Sep 2018

👎9

The 'crd-install' install hook is getting popular. I guess because this is very convenient way to handle CRDs in helm. We should support this one hook at least.

grep -r 'crd-install' .                                                                                                                                                             (docker-for-desktop/default)

./stable/prometheus-operator/templates/prometheus-operator/crd-alertmanager.yaml:    "helm.sh/hook": crd-install
./stable/prometheus-operator/templates/prometheus-operator/crd-servicemonitor.yaml:    "helm.sh/hook": crd-install
./stable/prometheus-operator/templates/prometheus-operator/crd-prometheus.yaml:    "helm.sh/hook": crd-install
./stable/prometheus-operator/templates/prometheus-operator/crd-prometheusrules.yaml:    "helm.sh/hook": crd-install
./stable/cert-manager/templates/clusterissuer-crd.yaml:    "helm.sh/hook": crd-install
./stable/cert-manager/templates/certificate-crd.yaml:    "helm.sh/hook": crd-install
./stable/cert-manager/templates/issuer-crd.yaml:    "helm.sh/hook": crd-install
./stable/jaeger-operator/templates/crd.yaml:    "helm.sh/hook": crd-install

https://github.com/argoproj/argo-cd/pull/792

alexmt on 16 Nov 2018

What would be the workaround to this?

kuznero on 19 Nov 2018

The 'crd-install' is solved by https://github.com/argoproj/argo-cd/commit/275b9e194d3f091f1400f509c0786f8b443f58e4 . Only workaround right now is to remove 'crd-install' manually.

@kuznero is any other helm hook important for you ?

alexmt on 19 Nov 2018

@alexmt not that I know of for now.

kuznero on 19 Nov 2018

pre-install hooks in combination with hook-weight are very useful, for example in case when we need to create pvc and then do something with persistent volume during installation (but not when we upgrading app). Same could be achieved with init containers and scripting, but it's more complicated, especially when hooks must run in certain order

alexmojo on 22 Jan 2019

This is also important for the influxdb helm chart. It uses a post-install hook to set the admin user for http auth

bstascavage on 21 Mar 2019

The difficulty is that, none of these hooks (pre-upgrade, post-upgrade, pre-install, post-install, pre-delete, post-delete) actually map cleanly to any concepts in Argo CD. Even pre-upgrade does not map cleanly to PreSync, because pre-upgrade is not executed during the initial deploy.

In order to support helm hooks properly, we would likely need to introduce granular Application phases/state. All of these imperative executions of hooks are somewhat antithetical to gitops style of deployments where desired state is stored in git, which makes the implementation of this a bit involved.

jessesuen on 11 Apr 2019

Could we document work-arounds?

alexec on 12 Apr 2019

👍2

I think argocd should at least show an indication there was a hook in the template that wasn't applied If this issue is deprioritizing then maybe we should open a new issue for the indication

OmerKahani on 13 Jun 2019

We could support Helm lifecycles by mapping them to sync-hooks.

If you own the chart you could add the Argo CD hook annotations alongside the Helm ones.

Someone needs to have a good think about this. Good one for community contribution.

alexec on 13 Jun 2019

👍2

We're currently having this issue with KubeDB: https://github.com/kubedb/installer/search?q=helm.sh%2Fhook&unscoped_q=helm.sh%2Fhook ...

Maybe just having some kind of preprocessor to patch the helm template dump with the right annotations before starting the deployment?

dave08 on 13 Jun 2019

👍1

We're about to start using kubedb through argocd as well, this would be awesome - in the meantime I'll likely just fork their chart and add the annotations as @alexec suggested.

I feel like the hooks should be fairly easily parsed and ID'd and if we agree on helm <> argo hook mapping injecting the argo hook could be done during sync based on an option passed into the application manifest? Thoughts?

spencergilbert on 27 Jun 2019

@spencergilbert Would you mind sharing the repo fork in the meantime? I wanted to do the same, but don't have too much time now, so I just deployed it with the script... :face_with_head_bandage:

dave08 on 27 Jun 2019

@dave08 I'll see what I can do.

These look like

"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
...
"helm.sh/hook": pre-delete
"helm.sh/hook-delete-policy": hook-succeeded,hook-failed

post-install/upgrade can probably just map to PostSync
before-hook-creation... not sure about this one
pre-delete... not sure if it's needed if you're using the argo finalizer annotation
hook/succeeded/failed maps to HookSucceeded and HookFailed

Thoughts?

spencergilbert on 27 Jun 2019

Potential mappings:

alexec on 2 Jul 2019

See https://github.com/helm/helm/blob/c3242d1be4ca503fa78b251117d00290b40c3f37/pkg/hooks/hooks.go

alexec on 2 Jul 2019

Here's where I stand on this, after studying the use of these hooks in public helm charts.

For a first pass at this, I think the following mapping might solve 70% of peoples use cases:

jessesuen on 25 Jul 2019

👍5

Why do you have to map then rather than actually implementing them as they were designed? As you have pointed out the mapping isn't clean or accurate and thus isn't really what the ask is.

stormmore on 25 Jul 2019

I've broken this ticket into 5 smaller tickets to make it easier to deliver incremental value. We can close this one when they are 4/5 are complete. #2038 is for some other time.

alexec on 26 Jul 2019

Why do you have to map then rather than actually implementing them as they were designed? As you have pointed out the mapping isn't clean or accurate and thus isn't really what the ask is.

As I mentioned, the only way to support the exact meanings of helm hooks, we would need to introduce lifecycle state into an application. In other words, introduce/maintain a state machine of where the application is in its lifecycle. This state machine could easily become inaccurate (e.g. resources were existing and the argocd app was created to manage those resources after the fact).

We are trying to avoid introducing lifecycle state so the middle ground here is to honor a subset of the common concepts between helm/Argo CD.

jessesuen on 29 Jul 2019

👍3

I ran into this when trying to deploy the sparkoperator via ArgoCD. The installation of the webhook cleanup job installed as part of the sparkoperator got stuck and complains that it cannot be installed because it is unable to reach a service account that is also created by the helm chart.
So if I get the whole discussion here correctly, this is not going to be fixed?

jgoeres on 12 Jun 2020

hi guys, the spark operator exposes a few hooks with "helm.sh/hook": pre-delete annotation. I understand this is not supported by ArgoCD => https://argoproj.github.io/argo-cd/user-guide/helm/#helm-hooks, but the one of those hooks runs as a first resource to be created, essentially failing relying on an already

How does argo treats those annotation, does it change it to a default 'install' now ?