cert-manager could not find resources after upgrade

[g4s8]

Describe the bug:
After upgrading cert-manager using helm, cert-manager is not able to find any custom resources:
can't list or create issuer, clusterissuer, certificate.

cert-manager produces a lot of similar logs messages:

E1203 08:16:50.213380       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Issuer: the server could not find the requested resource (get issuers.certmanager.k8s.io)
E1203 08:16:51.213100       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.ClusterIssuer: the server could not find the requested resource (get clusterissuers.certmanager.k8s.io)
E1203 08:16:51.213929       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Certificate: the server could not find the requested resource (get certificates.certmanager.k8s.io)
E1203 08:16:51.214767       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Issuer: the server could not find the requested resource (get issuers.certmanager.k8s.io)

Expected behaviour:
cert-manager is able to list or create custom resources

Steps to reproduce the bug:

1. Install cert-manager-v0.4.0-dev.3 via helm install
2. Configure issuers and certs
3. Upgrade to cert-manager-v0.5.2 via helm upgrade

Anything else we need to know?:
All existing issuers and certificates was disappeared.

Environment details::

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"archive", BuildDate:"2018-10-12T16:56:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud-provider/provisioner: AWS KOPS
• cert-manager version: migration from 0.4.0-dev.3 to 0.5.2
• Install method: helm

/kind bug

[munnerz]

I've seen this reported in a few places, and it's due to a bug in Helm's handling of the crd-install hook: https://github.com/helm/helm/issues/4697

My current plan is to remove the crd-install annotation in v0.6, which will subsequently require that we ship the webhook Helm chart as a separate deployment manifest/chart. (i.e. we'll have to remove the webhook.enabled option from the cert-manager helm chart).

Removing the annotation will also cause users upgrading from v0.5 to receive an AlreadyExists error when upgrading to v0.6. In order to fix this, they'll need to backup all ClusterIssuer/Issuer/Certificate resources, delete the CRDs, and then perform the upgrade.

The other option would be to remove the CRDs from that chart altogether, and ship a simple YAML manifest containing CRDs only and require users kubectl apply -f this file before installing the cert-manager chart. Unfortunately, the side effect of this is that users upgrading from v0.4 or earlier to v0.6 will see the same behaviour as you've reported (i.e. the CRD, and thus all CRs, being deleted).

This is a really messy situation and I've been deliberating a while over the best way to handle this, burning the least number of users.

Thanks for opening an issue to track it, & I'm keen to hear any thoughts/ideas you may have on this (as we can't release 0.6 without a resolution!)

[sww314]

I ran into the same issue. To fix, I recreate everything?

[g4s8]

@sww314 yes, I stored all issuers and certifiates to yaml, deleted cert-manager using helm, then installed new version and restored issuers and certificates using kubectl apply -f

[munnerz]

I've opened #1138 to attempt to resolve this - we now require the CRD to be manually installed as part of the installation (with kubectl apply -f). This isn't the most ideal workaround, but it is the most robust until Helm has better support for CRDs!

[tsuna]

I just upgraded from v0.4.1 to v0.6.0 and somehow had to re-apply the CRDs after helm upgrade.

$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.6.0/deploy/manifests/00-crds.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io configured
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created

$ helm repo update
[...]
$ kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
Error from server (NotFound): namespaces "cert-manager" not found
$ helm upgrade --version v0.6.0 cert-manager stable/cert-manager
Release "cert-manager" has been upgraded. Happy Helming!
[...]



md5-941d19c1d346f5a352e2cc00e677cad5



$ kubectl -n kube-system logs -f cert-manager-6874795dc8-jn74r
[...]
E0127 08:33:17.908696       1 reflector.go:205] pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.ClusterIssuer: the server could not find the requested resource (get clusterissuers.certmanager.k8s.io)
E0127 08:33:17.908731       1 reflector.go:205] pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Issuer: the server could not find the requested resource (get issuers.certmanager.k8s.io)
E0127 08:33:17.909233       1 reflector.go:205] pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Certificate: the server could not find the requested resource (get certificates.certmanager.k8s.io)
[...]



md5-941d19c1d346f5a352e2cc00e677cad5



$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.6.0/deploy/manifests/00-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io unchanged

[tsuna]

bah, I missed this part in the manual, my bad: Upgrading from older versions using Helm