Cert-manager: helm install stable/cert-manager does not work anymore?

[tmontalbano]

Before I could install cert-manager via helm and since a helm repo update, it does not seem to work anymore:

Here is before:

helm upgrade --install cert-manager stable/cert-manager --set ingressShim.defaultIssuerName=letsencrypt-prod --set ingressShim.defaultIssuerKind=ClusterIssuer --kube-context --namespace default
Release "cert-manager" does not exist. Installing it now.
NAME: cert-manager
LAST DEPLOYED: Wed Jan 23 22:26:43 2019
NAMESPACE: default
STATUS: DEPLOYED

And now:

helm install stable/cert-manager --set ingressShim.defaultIssuerName=letsencrypt-prod --set ingressShim.defaultIssuerKind=ClusterIssuer --kube-context scheduled-button-cluster-admin --namespace default
Error: validation failed: [unable to recognize "": no matches for kind "Certificate" in version "certmanager.k8s.io/v1alpha1", unable to recognize "": no matches for kind "Certificate" in version "certmanager.k8s.io/v1alpha1", unable to recognize "": no matches for kind "Issuer" in version "certmanager.k8s.io/v1alpha1", unable to recognize "": no matches for kind "Issuer" in version "certmanager.k8s.io/v1alpha1"]

Describe the bug:

Here is my current helm version:

Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}

Expected behaviour:

cert-manager helm install should work

Steps to reproduce the bug:

Have helm last repo updates and run the following command:
helm install stable/cert-manager

Anything else we need to know?:

We create our clusters in Azure via aks.

/kind bug

[pag-r]

I've noticed this today. OpenStack installation.

[wrogrammer]

Please run:

kubectl apply \
    -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml

before your cert-manager installation process.

## IMPORTANT: you MUST install the cert-manager CRDs **before** installing the
## cert-manager Helm chart
$ kubectl apply \
    -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml

$ helm install --name my-release stable/cert-manager

Helm Chart -> cert-manager changed installation process.
https://github.com/helm/charts/pull/10830/files

https://cert-manager.readthedocs.io/en/latest/getting-started/2-installing.html#installing-cert-manager

[tmontalbano]

Thanks, this works, but, following this issue, I have troubles installing the cluster-issuer while it was working before.

Should I open a new issue for that?

Just in case, here is the error I get:

helm upgrade --install cluster-issuer chartmuseum/certificate-issuer --kube-context my_cluster --namespace default
Release "cluster-issuer" does not exist. Installing it now.
Error: release cluster-issuer failed: Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

I have tried and reading the doc, but, so far, I fail to find what I should do to make this work.

Here is my cluster-issuer chart:

certificate-issuer/templates/cluster-issuer.yaml:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: my_email
privateKeySecretRef:
name: letsencrypt-prod
http01: {}

Thanks for your help.

[Kusumoto]

I have a same problem in my AKS.

I workaround for this issue is downgrade to install v0.5.2.

helm install stable/cert-manager \
    --namespace kube-system \
    --set ingressShim.defaultIssuerName=letsencrypt-prod \
    --set ingressShim.defaultIssuerKind=ClusterIssuer \
    --version v0.5.2

[tmontalbano]

Thanks, the workaround to a previous version works, but I am wondering if there is a more permanent solution to this issue.

[loomsen]

I have problems installing 0.6.0 too.

Error: apiVersion "certmanager.k8s.io/v1alpha1" in cert-manager/charts/webhook/templates/pki.yaml is not available
Fresh installation on a newly created cluster, both methods - upgrade -i and install - fail with the same message.

0.5.2 works fine.

[Dynogic]

Same issue following the guide at: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes#step-4-%E2%80%94-installing-and-configuring-cert-manager

Resolving the issue by adding the version mentioned by @Kusumoto

helm install --name cert-manager --namespace kube-system --version v0.5.2 stable/cert-manager

[wstewartii]

I have the same issue. Can we reopen this?

[tmontalbano]

I don't understand why it has been closed. Has it ever been resolved? Using an anterior version does not sound very reliable in the long term.

[munnerz]

If you are running into issues with CRDs, please read the upgrade notes here which should resolve your problems: https://cert-manager.readthedocs.io/en/latest/tasks/upgrading/upgrading-0.5-0.6.html#upgrading-from-older-versions-using-helm

[wstewartii]

If you are running into issues with CRDs, please read the upgrade notes here which should resolve your problems: https://cert-manager.readthedocs.io/en/latest/tasks/upgrading/upgrading-0.5-0.6.html#upgrading-from-older-versions-using-helm

Following those instructions and specifying --version v0.6.1 causes the following error

Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

Running a helm repo update does not actually resolve this error.

Not specifying a version at allows the chart to install but causes the following error when trying to install a clusterissuer.

Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

This seems to also be where @tmontalbano is getting stuck at. The only thing that seems to work is passing --version v0.5.2

[wstewartii]

I've also done these steps on a newly installed eks cluster and I'm running into the same issue.

[pirtoo]

Seeing errors with 0.6.x on a brand new GKE cluster (so no upgrade issues), after doing the external kubectl apply command (that shouldn't be outside a chart) I'm getting:
Error: release cert-manager failed: customresourcedefinitions.apiextensions.k8s.io "certificates.certmanager.k8s.io" already exists

I've rolled back to v0.5.2 as well.

[victortrac]

This is how we've gotten 0.6.0 to work on EKS:

# Set up cert-manager as a dependency and install the ClusterIssuer as the same chart
$ cat requirements.yaml
dependencies:
- name: cert-manager
  version: "v0.6.0"
  repository: "@stable"

$ cat values.yaml
clusterissuer:
  name: letsencrypt-prod
  email: [email protected]
  server: https://acme-v02.api.letsencrypt.org/directory
  providers:
    - name: dns
      route53:
        region: us-west-2

cert-manager:
  ingressShim:
    defaultIssuerName: "letsencrypt-prod"
    defaultIssuerKind: "ClusterIssuer"

# https://docs.cert-manager.io/en/latest/getting-started/webhook.html#running-on-amazon-eks
webhook:
  enabled: false

# Delete all old certmanager CRDs
$ kubectl get -n cert-manager crd | grep certmanager
$ kubectl delete -n cert-manager crd <crds that you have already installed from older versions>

# Install the cert-manager CRDs
$ kubectl apply \
  -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml

$ kubectl create ns cert-manager
$ kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

$ helm upgrade cert-manager . --install --namespace cert-manager -f values.yaml

The important parts are that you need to set disable-validation=true on the namespace and set webhook.enabled=false.

[tonyfahrion]

Hi,

I think I found the related issue. It took me quite some time; but it seems to me, that kubernetes is unable to find the right endpoint for validation webhook. After I edited the resource and set the

  service:
    name: <service-name>
    namespace: <namespace of the service>

cert-manager worked like expected.

I also did a PR #1359. Hope this helps 🙂

Kind regards
Tony

[eaba]

I had this same issue with Azure DevOps
I needed to run helm init before ruining helm install to get it working

[deploymentking]

I have also had this issue with Azure AKS with v0.6.1
I ran helm init --upgrade and helm repo update
However instead of downgrading to 0.5.2 I downgraded to 0.6.0

helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.6.0 \
  stable/cert-manager

[kojidev]

Same issue...

[davi5e]

I also solved it by disabling webhook, as in

helm install \
    --name cert-manager \
    --namespace cert-manager \
    stable/cert-manager \
    --set webhook.enabled=false

[ghost]

There might be an issue with the contents of the stable charts repo.

I could neither install version 0.6.2:

$> helm install   --name cert-manager   --namespace cert-manager   --version v0.6.2   stable/cert-manager
Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

Nor version 0.6.1.:

$> helm install   --name cert-manager   --namespace cert-manager   --version v0.6.1   stable/cert-manager
Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

The index file https://kubernetes-charts.storage.googleapis.com/index.yaml contains no entry for version 0.6.2. There is an entry for appVersion: v0.6.1 that has a version: v0.6.5 key-value-pair.
When I tell helm to install 0.6.5, it works:

$> helm install   --name cert-manager   --namespace cert-manager   --version v0.6.5   stable/cert-manager
NAME:   cert-manager
LAST DEPLOYED: Thu Feb 21 14:56:10 2019
NAMESPACE: cert-manager
STATUS: DEPLOYED
[...]
cert-manager has been deployed successfully!
[...]

[Ogala]

@selobosi's solution works and is perhaps the most stable way to handle this issue.

[woodwardmatt]

Just followed @selobosi 's approach and used version --version v0.6.6 on Azure - deployed successfully!

[kieronlanning]

Confirmed too - v0.6.6 works for me.

[tsuna]

OK so for some reason the GitHub release version v0.6.2 is actually v0.6.6 in Helm — basically we're off by 4 since v0.6.1?

@munnerz can we realign things somehow to avoid the ongoing confusion? This issue should probably be reopened until this discrepancy is addressed.

[marcelloromani]

On the last comment by @tsuna , while trying different image.tag versions I noticed that if I specify v0.6.6 what gets actually deployed is v0.6.5.

Also, if I specify v0.5.2, I actually get v0.6.5. Something odd is going on in the chart repo...

[sww314]

On GKE (1.11.7-gke.6), this worked for me:
helm install --name cert-manager --namespace kube-system --version v0.6.0 stable/cert-manager --set webhook.enabled=false

Thanks to suggestion from @davi5e.
The newer versions did not work.

[davi5e]

In GCP, you have to add the port 6443 to the master's Firewall Rules in order for the webhook to work.

See more info here.

[stevenmunro]

I know this comment won't be particularly helpful, but this is incredibly frustrating.

[marcelloromani]

I was super frustrated by this bug as well, until I realised the documentation has the answer.
You need some manual steps (which I automated as null_resources in terraform)

https://docs.cert-manager.io/en/latest/getting-started/install.html#installing-with-helm

The section you want to follow is the green box under Step.

[tsuna]

v0.7.0 isn't present in the index either so can't upgrade to it now...

$ helm upgrade cert-manager --version v0.7.0 stable/cert-manager --set webhook.enabled=false
Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

[Majorem]

I ran into this issue with AKS.

I deleted everything and followed the instructions on their website to get it working.

Install the CustomResourceDefinition resources separately

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

Create the namespace for cert-manager

kubectl create namespace cert-manager

Label the cert-manager namespace to disable resource validation

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

Add the Jetstack Helm repository

helm repo add jetstack https://charts.jetstack.io

Update your local Helm chart repository cache

helm repo update

Install the cert-manager Helm chart

helm install
--name cert-manager
--namespace cert-manager
--version v0.7.0
jetstack/cert-manager

[ramkumarit2007]

I had the same issue, and fixed it by running the helm with the --version v0.5.2 with either Kind as ClusterIssuer or Issuer.

[dvdblk]

I have finally managed to install cert-manager v0.7.0 even though it didn't work via install or upgrade directly.

Here are the steps:

• remove the jetstack repo helm repo remove jetstack
• update helm repositories helm repo update
• fresh install helm install --name cert-manager --namespace cert-manager jetstack/cert-manager

[ggilley]

I still see this on virtually every install. Seems like some kind of race condition...

Error from server (InternalError): error when creating "/tmp/cert-manager-issuer.yml": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request Error from server (InternalError): error when creating "/tmp/cert-manager-issuer.yml": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

[saplla]

@ggilley I get that too. I created #1697 to track it.

[george-oakling]

Why there should be CRDs installed before helm chart? Isnt helm chart supposed to be complete solution for every deployment possible? I can't get this :)

[iorlas]

For the strangers with a problem like this:

Error: apiVersion "certmanager.k8s.io/v1alpha1" in cert-manager/charts/webhook/templates/pki.yaml is not available

Ensure you have applied CRD's

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml

Then, grab a cup of tea and wait for 10-15min, then try to install it again, for example:

// For helm3
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.9.0

I didn't dig it yet, but feels like CRDs are not available right after the kube apply command, it needs some time before making it available on the cluster.

And yeah, whole cert-manager thing is like a clusterfuck. It is not recommended yet as it is not 1.0 yet, but other solutions like kube-lego are not maintained anymore, referencing to this, not stable software, which is so wrong by any means...

[AmazingTurtle]

@DanielMorsing you have to wait for the apiservice for the CRD to come up. Looking up their docs it should be the following command (requires latest kubectl, because kubectl wait is a new experimental feature):

kubectl wait --for=condition=Available --timeout=300s \
    apiservice v1beta1.admission.certmanager.k8s.io

@george-oakling looking at https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html#installing-with-regular-manifests

This creates a chicken-and-egg problem, where cert-manager requires the webhook in order to create the resources, and the webhook requires cert-manager in order to run.

That's why the CRD has to be deployed first.