Helmfile: [doc] Recommended way of handling non conventional install such as cert-manager

[victornoel]

HI,

When reading the process to install the cert-manager chart (https://hub.helm.sh/charts/jetstack/cert-manager), you can see two steps before installing the chart:

• installing some CRD prior to the chart
• adding a label on the namespace

How is this handled with helmfile? Is there any way to automatize both of those step?

Some documentation on this would be great also :)

[mumoshu]

@victornoel Hey!

installing some CRD prior to the chart

I believe this can be achieved by deploying incubator/raw chart as suggested in https://github.com/roboll/helmfile/issues/494#issuecomment-474697430

adding a label on the namespace

In the case that it's ok to create a dedicated, new namespace with the desired labels, I believe you can use incubator/raw chart here as well.

Otherwise, I'd suggest using a prepare hook. With that, you can inject any command, including kubectl create ns && kubectl label ns foo=bar, before the chart is installed.

Some documentation on this would be great also :)

I think so too!

Where would you like to have some documentation for this in?
A new section in README, or under docs/, or anywhere else..?

[victornoel]

@mumoshu thanks for the fast answer (and for helmfile to have all the needed features already ;).

For the hook, I should have read in more details, so no need to add more doc, but maybe if there is a best practice documentation, the use of the raw chart could be a good information to add.

Also, how sure are we of the order of execution of the release? Are they following the order of declaration in the file? I suppose it is the case, right?

[mumoshu]

@victornoel My pleasure :)

We do have a best-practices doc in the docs dir! Happy to add there. I'd also appreciate it if you could submit a PR.

Regarding the exec order, you should use sub-helmfies, so that helmfile execs sub-helmfiles in the order of definitions.

Note that you'll see helmfile executes "releases" in the order of definitions, but that's not always. Setting --concurrecy to greater than 1 makes the order undeterministic.

So you'd write your helmfile.yaml like this:

helmfiles:
- prereqs/cert-manager-crd.yaml

releases:
- name: labeled-ns
  chart: incubator/raw
  values:
   - resources:
     # or
     # templates:
     # ...
- name: cert-manager
  chart: stable?/cert-manager
  #snip

[victornoel]

@mumoshu one more question: is the hook executed even if calling helmfile with -i or if calling template for example?

I will see how I end up using the raw chart and will get back to you about a potential doc PR :)

[patrickmslatteryvt]

Using a prepare hook worked for me:

releases:
  - name: cert-manager
    namespace: clu-inf-all
    chart: stable/cert-manager
    wait: true
    version: {{ readFile "versions/infra/versions.yaml" | fromYaml | get "cert-manager" }}
    values:
      - values/infra/cert-manager.yaml.gotmpl
    hooks:
      - events: ["prepare"]
        command: "kubectl"
        args: ["apply", "--validate=true", "--filename=values/infra/cert-manager-crds.yaml"]
      - events: ["prepare"]
        command: "sh"
        args: ["../../scripts/dns-zone-client-secret.sh"]

but we did run into an issue with linting the file after adding hooks. I'll file a bug on that later today.

[andrewnazarov]

@patrickmslatteryvt, what version of helmfile are you using. I'm asking since I've experienced some troubles having several prepare hooks within a release. By the way, it was also a cert-manager).

[patrickmslatteryvt]

helmfile --version
helmfile version v0.53.0

[mumoshu]

Let's use presync hooks rather than prepare for anything that mutates your cluster state before syncing, as prepare triggers even for helmfile lint which should be a read-only operation.

See #552 for more info.

[andrewnazarov]

I successfully deployed cert-manager using prepare hooks in the past. Now I'm trying presync hook assuming to get the same result. But I cannot make it work with apply, although sync works well. I'm getting the application specific error during helm diff stage.

Worth to mention I've got tillerless enabled.

[andrewnazarov]

apply starts working after the successful sync. So in the beginnig, it should be a sync command.

[mumoshu]

@andrewnazarov Good point! That's probably due to that helm-diff needs the CRD to be already installed onto the cluster to successfully diff against the custom resources.

So my best bet is that you should use

helmfiles:
- install-crd-or-whatever.yaml

releases:
- name: yourapp
   chart: charts/your-chart-containing-custom-resources

So that helmfile apply firstly diff and applies the CRD, and then yourapp, without any problem.

[victornoel]

For the record, I have successfuly solved my problem both with the helmfiles field to setup crds and presync hook for labeling the namespace.

It looks as follow. The only small drawback I find with this is that I have to transform the crd resources file provided by cert-manager (https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml) into an array of resources in the helmfile. Maybe I'm missing a way to directly inject the file as-is with helmfile?

cert-manager-crds.helmfile.yaml

repositories:
  - name: incubator
    url: https://kubernetes-charts-incubator.storage.googleapis.com/

releases:
  - name: cert-manager-crds
    namespace: proxy
    chart: incubator/raw
    version: 0.2.3
    values:
    - resources:
      - ... resources here ...

helmfile.yaml

repositories:
  - name: jetstack
    url: https://charts.jetstack.io

helmfiles:
  - path: cert-manager-crds.helmfile.yaml

releases:
  - name: cert-manager
    namespace: proxy
    version: v0.8.1
    chart: jetstack/cert-manager
    hooks:
      - events: ["presync"]
        command: "kubectl"
        args: ["label", "--overwrite", "namespace", "{{`{{.Release.Namespace}}`}}", "certmanager.k8s.io/disable-validation=true"]

[aegershman]

@victornoel Thanks for sharing your solution. Want you to know I'm going to try something similar. The most displeasurable thing about cert-manager, in my view, is the requirement for CRDs. So I think this is an interesting solution.

I'm assuming this same technique of using the raw helm chart will also need to be used for creating Issuers and such.

Will try it out edit this comment with any feedback.

[zakkg3]

I find tricky deploy it with raw... at the end its a helm chart with the values resources on one yaml file inside templates.. so I am using a local chart crd-cert-manager with a single yaml file (crds.yaml) and my helmfile to make it deploy in order and it works just perfect :)

here is:

repositories:
  - name: jetstack
    url: https://charts.jetstack.io

helmfiles: # this will install the crds (issuer, certificate and so on)
  - path: crds-helmfile.yaml

releases:
  - name: cert-manager
    namespace: cert-manager
    version: v0.8.1
    chart: jetstack/cert-manager
    hooks: # here we add the annotation to namespace /disable-validation=true
      - events: ["presync"]
        command: "kubectl"
        args: ["label", "--overwrite", "namespace", "{{`{{.Release.Namespace}}`}}", "certmanager.k8s.io/disable-validation=true"]

crds-helmfile.yaml

releases:
  - name: crds-cert-manager
    namespace: cert-manager
    chart: ./crds-cert-manager
    version: v0.1.0

and inside ./certs-cert-manager:
.../templates/00-crds.yaml (wget https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml)
Chart.yaml

name: crds-cert-manager
version: v0.1.0

[zakkg3]

Update:

here is all you need https://github.com/zakkg3/cert-manager-installer

[zakkg3]

Final Update :D
Tested and merged into the official docs

https://github.com/jetstack/cert-manager/blob/master/docs/getting-started/install/kubernetes.rst#alternative-installation-methods

[mumoshu]

@zakkg3 Awesome! Thanks for sharing it ☺️

[zloeber]

I find tricky deploy it with raw... at the end its a helm chart with the values resources on one yaml file inside templates.. so I am using a local chart crd-cert-manager with a single yaml file (crds.yaml) and my helmfile to make it deploy in order and it works just perfect :)

I ran in the the same issues around the same time and came up with the same solution. It works but it forces you to double your support surface area (in a manner). It works but just feels wrong :)

[zakkg3]

@zloeber in terms of support you only need to bump version time to time and update the CRD's. i guess its what you mean...
Just run versionBump.py It will do everything for you :)
https://github.com/zakkg3/cert-manager-installer#bumping-version

[osterman]

Fwiw, here's how we've been deploying cert-manager.

https://github.com/cloudposse/helmfiles/blob/master/releases/cert-manager.yaml

[andrewnazarov]

@osterman is it working for helmfile apply? I guess it requires sync as well.