Helmfile: Failed to render chart and I don't know why

Created on 10 Apr 2020 · 7Comments · Source: roboll/helmfile

I get an error somewhere in the RAW Chart.

helmfile diff

in ./helmfile.yaml: failed processing release cert-cluster-issuer-letsencrypt: helm exited with status 1:
Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1alpha2"

I don't know where the problem is, the RAW part seems to be OK. I have spend the whole day on it but helmfile/helm/k8s doesn't provide any clues.

helmDefaults:
  wait: true
  verify: true

repositories:
  - name: jetstack
    url: https://charts.jetstack.io
  - name: incubator
    url: https://kubernetes-charts-incubator.storage.googleapis.com

releases:
  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    version: "v0.14.2"
    atomic: true
    hooks:
      - events: ["presync"]
        command: "/bin/sh"
        args: ["-c", "kubectl get namespace \"{{`{{ .Release.Namespace }}`}}\" >/dev/null 2>&1 ||
          kubectl create namespace \"{{`{{ .Release.Namespace }}`}}\" &&
          kubectl label namespace \"{{`{{ .Release.Namespace }}`}}\" certmanager.k8s.io/disable-validation=true --overwrite"]
      - events: ["presync"]
        command: "/bin/sh"
        args: ["-c", "kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/{{`{{ .Release.Version }}`}}/cert-manager.crds.yaml > /dev/null 2>&1 || true"]

  # The diff will fail here, if the CRD is not installed.
  - name: cert-cluster-issuer-letsencrypt
    chart: incubator/raw
    namespace: cert-manager
    atomic: true
    cleanupOnFail: true
    labels:
      component: cert-manager
    needs:
      - cert-manager/cert-manager
    values:
      - resources:
        - apiVersion: cert-manager.io/v1alpha2
          kind: ClusterIssuer
          metadata:
            name: letsencrypt-staging
            labels:
              app: cluster-issuer
              app.kubernetes.io/instance: cert-manager
          spec:
            acme:
              server: https://acme-staging-v02.api.letsencrypt.org/directory
              email: {{ .Values.acme_email }}
              privateKeySecretRef:
                name: letsencrypt-staging
              solvers:
                - selector: {}
                  http01:
                    ingress:
                      class: traefik
        - apiVersion: cert-manager.io/v1alpha2
          kind: ClusterIssuer
          metadata:
            name: letsencrypt-prod
          spec:
            acme:
              server: https://acme-v02.api.letsencrypt.org/directory
              email: {{ .Values.acme_email }}
              privateKeySecretRef:
                name: letsencrypt-prod
              solvers:
                - selector: {}
                  http01:
                    ingress:
                      class: traefik

Source

Vad1mo

Most helpful comment

Ok, I finally got a working solution.
@mumoshu maybe you have a better solution,

The CRD must be installed otherwise the RAW installation fails.
The solution is a bit hacky with exec, but I didn't know how to do it better to retrieve the state from k8s

The important piece is values.gotmpl that store the response of kubectl which checks if the CRD exists. If it doesn't exist, then it skips the RAW part entirely

environments:
  default:
    values:
      - values.gotmpl
      - values-default.yaml
  stage:
    values:
      - values.gotmpl
      - values-stage.yaml
  prod:

helmDefaults:
  wait: true
  verify: false

repositories:
  - name: jetstack
    url:  https://charts.jetstack.io
  - name: incubator
    url:  https://kubernetes-charts-incubator.storage.googleapis.com

releases:
  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    version: v0.14.2
    atomic: true
    hooks:
      - events: ["presync"]
        command: "/bin/sh"
        args: ["-c", "kubectl get namespace \"{{`{{ .Release.Namespace }}`}}\" >/dev/null 2>&1 ||
          kubectl create namespace \"{{`{{ .Release.Namespace }}`}}\" &&
          kubectl label namespace \"{{`{{ .Release.Namespace }}`}}\" certmanager.k8s.io/disable-validation=true --overwrite"]
      - events: ["presync"]
        command: "/bin/sh"
        args: ["-c", "kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/{{`{{ .Release.Version }}`}}/cert-manager.crds.yaml > /dev/null 2>&1 || true"]

{{ if .Values.crdInstalled }}
  - name: letsencrypt-cert-issuer
    namespace: cert-manager
    chart: incubator/raw
    atomic: true
    cleanupOnFail: true
    values:
      - resources:
          - apiVersion: cert-manager.io/v1alpha2
            kind: ClusterIssuer
            metadata:
              name: letsencrypt-staging
            spec:
              acme:
                server: https://acme-staging-v02.api.letsencrypt.org/directory
                email: {{ .Values.acme_email }}
                privateKeySecretRef:
                  name: letsencrypt-staging
                solvers:
                  - selector: {}
                    http01:
                      ingress:
                        class: traefik

          - apiVersion: cert-manager.io/v1alpha2
            kind: ClusterIssuer
            metadata:
              name: letsencrypt-prod
            spec:
              acme:
                server: https://acme-v02.api.letsencrypt.org/directory
                email: {{ .Values.acme_email }}
                privateKeySecretRef:
                  name: letsencrypt-prod
                solvers:
                  - selector: {}
                    http01:
                      ingress:
                        class: traefik
{{ end }}

values.gotmpl

crdInstalled: {{ exec "./check-crd.sh" (list "") | trim }}

check-crd.sh

#!/usr/bin/env sh
set -u # -e "Automatic exit from bash shell script on error"  -u "Treat unset variables and parameters as errors"

kubectl get crd issuers.cert-manager.io > /dev/null 2>&1

if [ $? -eq 0 ]
then
  echo "true"
  exit 0
else
  echo "false"
  exit 0
fi

Vad1mo on 11 Apr 2020

👍2

All 7 comments

@Vad1mo I think Helm 2 requires custom resources likeletsencrypt-staging and letsencrypt-prod in your example (1)to reside within the same release that installs ClusterIssue CRD OR (2) requires the cluster to already have ClusterIssuer CRD installed.

This works without any prob w/ Helm 3. So could you try upgrading to Helm 3?

Otherwise you can try forking the cert-manager chart with helm fetch and add the raw chart as a chart dependency, so that you can create CRDs and CRs in one chart, which would solve the issue.

The last possible workaround would be to add "helm.sh/hook": crd-instal annotation to your "custom resources", like letsencrypt-staging. Yes I know that annotation is for CRD, not for custom resources. But I found that it seems to make it work.

mumoshu on 10 Apr 2020

@mumoshu what makes you think I am using Helm2 ? We are on Helm "v3.1.2"

I will try you suggestions and report the result.

Vad1mo on 10 Apr 2020

here is the full error message when debug is enabled.
Same error when the namespace already exists.

worker 2/2 finished
exec: helm diff upgrade --reset-values --allow-unreleased cert-cluster-issuer-letsencrypt incubator/raw --namespace cert-manager --values /var/folders/5r/wh15_25d4nq0dpy5tywjwsfw0000gn/T/values560087137: ********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************

********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************

worker 1/2 finished
err: release "cert-cluster-issuer-letsencrypt" in "helmfile.yaml" failed: helm exited with status 1:
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1alpha2"

  Error: plugin "diff" exited with error
in ./helmfile.yaml: helm exited with status 1:
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1alpha2"

Vad1mo on 10 Apr 2020

Ok, I finally got a working solution.
@mumoshu maybe you have a better solution,

The CRD must be installed otherwise the RAW installation fails.
The solution is a bit hacky with exec, but I didn't know how to do it better to retrieve the state from k8s

The important piece is values.gotmpl that store the response of kubectl which checks if the CRD exists. If it doesn't exist, then it skips the RAW part entirely

environments:
  default:
    values:
      - values.gotmpl
      - values-default.yaml
  stage:
    values:
      - values.gotmpl
      - values-stage.yaml
  prod:

helmDefaults:
  wait: true
  verify: false

repositories:
  - name: jetstack
    url:  https://charts.jetstack.io
  - name: incubator
    url:  https://kubernetes-charts-incubator.storage.googleapis.com

releases:
  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    version: v0.14.2
    atomic: true
    hooks:
      - events: ["presync"]
        command: "/bin/sh"
        args: ["-c", "kubectl get namespace \"{{`{{ .Release.Namespace }}`}}\" >/dev/null 2>&1 ||
          kubectl create namespace \"{{`{{ .Release.Namespace }}`}}\" &&
          kubectl label namespace \"{{`{{ .Release.Namespace }}`}}\" certmanager.k8s.io/disable-validation=true --overwrite"]
      - events: ["presync"]
        command: "/bin/sh"
        args: ["-c", "kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/{{`{{ .Release.Version }}`}}/cert-manager.crds.yaml > /dev/null 2>&1 || true"]

{{ if .Values.crdInstalled }}
  - name: letsencrypt-cert-issuer
    namespace: cert-manager
    chart: incubator/raw
    atomic: true
    cleanupOnFail: true
    values:
      - resources:
          - apiVersion: cert-manager.io/v1alpha2
            kind: ClusterIssuer
            metadata:
              name: letsencrypt-staging
            spec:
              acme:
                server: https://acme-staging-v02.api.letsencrypt.org/directory
                email: {{ .Values.acme_email }}
                privateKeySecretRef:
                  name: letsencrypt-staging
                solvers:
                  - selector: {}
                    http01:
                      ingress:
                        class: traefik

          - apiVersion: cert-manager.io/v1alpha2
            kind: ClusterIssuer
            metadata:
              name: letsencrypt-prod
            spec:
              acme:
                server: https://acme-v02.api.letsencrypt.org/directory
                email: {{ .Values.acme_email }}
                privateKeySecretRef:
                  name: letsencrypt-prod
                solvers:
                  - selector: {}
                    http01:
                      ingress:
                        class: traefik
{{ end }}

values.gotmpl

crdInstalled: {{ exec "./check-crd.sh" (list "") | trim }}

check-crd.sh

#!/usr/bin/env sh
set -u # -e "Automatic exit from bash shell script on error"  -u "Treat unset variables and parameters as errors"

kubectl get crd issuers.cert-manager.io > /dev/null 2>&1

if [ $? -eq 0 ]
then
  echo "true"
  exit 0
else
  echo "false"
  exit 0
fi

Vad1mo on 11 Apr 2020

👍2

🤦 - Just found out a simple force: true in the chart spec would have worked as well.

Vad1mo on 12 Apr 2020

what makes you think I am using Helm2 ? We are on Helm "v3.1.2"

Just because I couldn't reproduce your issue with helm3 on my machine. Perhaps any issue in my helmfile.yaml content? 🤔

Anyways, awesome to hear that you've managed to find a workaround yourself!

mumoshu on 13 Apr 2020

@Vad1mo Okay so I think I better understand the issue now.

My recommendation would be just copy the cert-manager CRDs into a local chart, version-control them along with your helmfile.yaml, so that you don't need hooks.

releases:

- name: namespaces
  chart: incubator/raw
  namespace: kube-system
  values:
  - resources:
      apiVersion: core/v1
      kind: Namespace
      metadata:
        name: cert-manager

- name: crds
  namespace: cert-manager
  chart: ./charts/crds
  needs:
  - kube-system/namespaces

- name: cert-manager
  namespace: cert-manager
  chart: jetstack/cert-manager
  needs:
  - kube-system/namespaces

- name: cert-cluster-issuer-letsencrypt
  chart: incubator/raw
  namespace: cert-manager
  needs:
  - cert-manager/crds
  values:
  - resources:
    # ...

mumoshu on 13 Apr 2020

🚀1

Was this page helpful?

0 / 5 - 0 ratings