Cert-manager: No ConfigMap with the name "cert-manager-webhook-ca-sync" found

[mbaragiola]

Describe the bug:
When trying to upgrade from v0.5.0 to v0.6.0 using helm with RBAC disabled and namespace other than cert-manager, upgrade fails after following specified instructions on the very last command:

$ helm upgrade --version v0.6.0 cert-manager stable/cert-manager --set rbac.create=false
Error: UPGRADE FAILED: no ConfigMap with the name "cert-manager-webhook-ca-sync" found

Expected behaviour:

Get helm to upgrade from v0.5.0 to v0.6.0.

Steps to reproduce the bug:

1) Install cert-manager v0.5.0 with helm on namespace kube-system and RBAC disabled.

EDIT: Apparenty I have RBAC enabled on my cluster, but I have installed cert-manager with the RBAC disabled flag.

2) Follow upgrade instructions from v0.5.0 to v0.6.0.

Anything else we need to know?:

$ kubectl get pods --namespace kube-system 
NAME                                                          READY     STATUS    RESTARTS   AGE
cert-manager-7bb46cc6b-jwkkj                                  1/1       Running   8          80d
event-exporter-v0.2.3-799d485c7d-hjhgs                        1/1       Running   0          41d
fluentd-gcp-scaler-6d7bbc67c5-8l4dm                           1/1       Running   0          41d
fluentd-gcp-v3.1.0-6dqz7                                      1/1       Running   0          41d
fluentd-gcp-v3.1.0-94z9f                                      1/1       Running   0          41d
fluentd-gcp-v3.1.0-mxzwx                                      1/1       Running   0          41d
heapster-v1.5.3-7c79d479bb-k8gdx                              2/2       Running   0          141d
kube-dns-584df47cd9-7gxms                                     4/4       Running   0          83d
kube-dns-584df47cd9-g7zgm                                     4/4       Running   0          83d
kube-dns-autoscaler-79b4b844b9-kc9xd                          1/1       Running   0          141d
kube-proxy-gke-turnapp-cluster-1-default-pool-5e7b8041-703v   1/1       Running   0          141d
kube-proxy-gke-turnapp-cluster-1-default-pool-5e7b8041-8kkl   1/1       Running   0          141d
kube-proxy-gke-turnapp-cluster-1-default-pool-5e7b8041-kpvx   1/1       Running   0          141d
kubernetes-dashboard-74f855c8c6-tp9k2                         1/1       Running   0          141d
l7-default-backend-5d5b9874d5-twxkf                           1/1       Running   0          141d
metrics-server-v0.2.1-7486f5bd67-hgn25                        2/2       Running   0          141d
tiller-deploy-5577976c44-7jh4l                                1/1       Running   0          9m

Should I just rename the namespace?
I have successfully installed the new CRDs, but am still running v0.5.0. Is it on a incompatible state?

Thanks in advance!

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.10.11
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
• cert-manager version (e.g. v0.4.0): v0.5.0 trying to upgrade to v0.6.0
• Install method (e.g. helm or static manifests): helm

/kind bug

[kingdonb]

I have this issue too, following the upgrade guide with AKS and rbac disabled.

I also have used the kube-system namespace for my cert-manager installation. Perhaps that was unwise...

Tried to roll back to v0.4.1, but I needed help from this comment: https://github.com/jetstack/cert-manager/issues/870#issuecomment-427379552

My clusterissuer is gone, though. Looks like I had created that manually to begin with, my bad.

Oh boy, there were some important notes in upgrading from versions older than v0.5.0!

[kingdonb]

Well, I wasn't able to solve this myself yet. Azure/AKS#796 is more my issue, but I had these errors when I hadn't read the details in https://docs.cert-manager.io/en/latest/admin/upgrading/upgrading-0.5-0.6.html

Maybe that will help you. I also gave up on using the kube-system namespace, I'm using cert-manager as the namespace.

In fairness all of these APIs are marked alpha, so it's to be expected that there are some hiccups upgrading.

[kostyrev]

Yep, I had those errors as well.
Just read all upgrading notes for every release and execute commands as stated.
Doing this fixed installing 0.6.0 for me.

[kingdonb]

For me, it turned out that the problem preventing cert-manager-webhook-ca-sync from being created normally was that my cluster does not support RBAC, and rbac.create=false setting is not honored by cert-manager's new webhook sub-chart.

I was able to hack together some changes to the chart to make it work, but this definitely looks like a bug. The webhook attempts to create clusterroles and bindings even with rbac.create=false set in the parent chart. This makes normal installation impossible. After fixing that by hand with a kludge, it looks like everything is in good shape:

$ kubectl get clusterissuer,issuer,certificates --all-namespaces
NAME                                                AGE
clusterissuer.certmanager.k8s.io/letsencrypt-prod   14m

NAMESPACE      NAME                                                      AGE
cert-manager   issuer.certmanager.k8s.io/cert-manager-webhook-ca         15m
cert-manager   issuer.certmanager.k8s.io/cert-manager-webhook-selfsign   15m

NAMESPACE         NAME                                                              READY   SECRET                             AGE
bamboo-waxworks   certificate.certmanager.k8s.io/bamboo-waxworks.[redacted]         True    bamboo-waxworks.[redacted]         14m
blog              certificate.certmanager.k8s.io/blog.[redacted]                    True    blog.[redacted]                    14m
cert-manager      certificate.certmanager.k8s.io/cert-manager-webhook-ca            True    cert-manager-webhook-ca            15m
cert-manager      certificate.certmanager.k8s.io/cert-manager-webhook-webhook-tls   True    cert-manager-webhook-webhook-tls   15m
...

Glad your issue was just not having read the documentation fully. I think that Azure/AKS#796 is a real issue that should be opened on cert-manager though, for some time since it was made the default in kubernetes to have RBAC enabled, AKS clusters still do not come with RBAC enabled as far as I know.

My AKS cluster is on Kubernetes v1.11.5. It might also be time to upgrade again and enable RBAC.

I'll open an issue if I can't find one that says this already.

[kingdonb]

I opened #1283 for my issue. If you're satisfied that the documentation is correct and you were able to work through the upgrade notes, then you can probably close this issue @mbaragiola. Thanks @kostyrev for chiming in!

[mbaragiola]

@kingdonb thanks for participating. If I enable RBAC would that render my actual cert-manager invalid?

[kingdonb]

@mbaragiola As far as I can tell, if your helm release/upgrade fails, then you will see "Failed" next to that release in history, and the change is never applied/automatically rolled back.

If your cluster has rbac enabled, then you should use the RBAC settings. It's unlikely that enabling RBAC will work on clusters with no RBAC support, or disabling RBAC on clusters without it. The default on most clusters these days should be RBAC enabled.

I followed the directions to upgrade from v0.4.0 to v0.5.0, and v0.5.0 to v0.6.0, which were slightly more complicated and required uninstalling cert-manager and reinstalling it. I don't know if your issue is happening because the cert-manager is installed in a namespace other than cert-manager, but that could be it. If you find this link at the bottom of the v0.5.0-v0.6.0 upgrade instructions, it details the process of backup/purge/reinstalling cert-manager, so you could put it in the expected namespace.

https://docs.cert-manager.io/en/latest/tasks/upgrading/upgrading-0.5-0.6.html#upgrading-from-older-versions-using-helm

I think if you missed this step on the first go-around, you may have put your helm release into an inconsistent state:

# Install the cert-manager CustomResourceDefinition resources before
# upgrading the Helm chart
kubectl apply \
     -f https://raw.githubuser...

It was a non-disruptive and well-documented operation for me to purge and reinstall cert-manager. If you think your installation is hosed, you should try it. I don't think it's required to be in the cert-manager namespace, but it will complicate some of the instructions since you'll need to know the right places to substitute "your-namespace" and there's probably no good reason to put cert-manager anywhere else than in its own namespace.

Like this one:

# If you are upgrading from v0.5 or below, you should manually add this
# label to your cert-manager namespace to ensure the `webhook component`_
# can provision correctly.
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

It's not clear to me from reading this, is this the literal cert-manager namespace, or is it whatever namespace you have installed cert-manager into? I think it means "your-namespace" which is just assumed to be cert-manager.

Some other places I'd get confused, in the values.yaml:

# Override the namespace used to store DNS provider credentials etc. for ClusterIssuer
# resources. By default, the same namespace as cert-manager is deployed within is
# used. This namespace will not be automatically created by the Helm chart.
clusterResourceNamespace: ""

leaderElection:
  # Override the namespace used to store the ConfigMap for leader election
  namespace: ""
...
# Optional additional arguments
extraArgs: []
  # Use this flag to set a namespace that cert-manager will use to store
  # supporting resources required for each ClusterIssuer (default is kube-system)
  # - --cluster-resource-namespace=kube-system

It's kind of muddy, and I think it will be just a lot easier installing into cert-manager namespace like much of the documentation assumes you will have done.

[mbaragiola]

@mbaragiola As far as I can tell, if your helm release/upgrade fails, then you will see "Failed" next to that release in history, and the change is never applied/automatically rolled back.

I successfully rolled back using helm rollback cert-manager <REVISION>. So far so good.

If your cluster has rbac enabled, then you should use the RBAC settings. It's unlikely that enabling RBAC will work on clusters with no RBAC support, or disabling RBAC on clusters without it. The default on most clusters these days should be RBAC enabled.

I'm digging into the Kubernetes docs on this regard. I'll see what I can do. Thanks!

I think if you missed this step on the first go-around, you may have put your helm release into an inconsistent state:

Installing v0.6.0 CRDs on v0.5.0 don't leave you on a inconsistent state, or so I've been told on the Slack channel. Don't know about the rest of the steps, though.

# If you are upgrading from v0.5 or below, you should manually add this
# label to your cert-manager namespace to ensure the `webhook component`_
# can provision correctly.
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

It's not clear to me from reading this, is this the literal cert-manager namespace, or is it whatever namespace you have installed cert-manager into? I think it means "your-namespace" which is just assumed to be cert-manager.

That's right, it's exactly "your-namespace". They've been discussing that on the Slack channel.

It's kind of muddy, and I think it will be just a lot easier installing into cert-manager namespace like much of the documentation assumes you will have done.

Indeed it is. I don't recall why I chose kube-system, probably was some trouble creating a new namespace and I already had kube-lego running at that namespace.

[munnerz]

Closing in favour of #1283 - thanks for all the discussion here 😄

/close

[jetstack-bot]

@munnerz: Closing this issue.

In response to this:

Closing in favour of #1283 - thanks for all the discussion here 😄

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.