Cert-manager: Manual Removal - Namespace Stuck at Terminating Status

[woodwardmatt]

Describe the bug:
I was trying a fresh install with the manifest file, but have a number of errors appear to do with the webhook. I following the troubleshooting but couldn't get a resolution. I decided to manually remove all objects and retry with a helm install instead. All objects within the namespace "cert-manager" are deleted, but I am unable to delete the "cert-manager" namespace itself - it's stuck at the "Terminating" status.

Expected behaviour:
Deleting the empty namespace should remove the namespace from the cluster

Steps to reproduce the bug:
Run a fresh install with the manifest file. Delete all objects within the namespace manually after install is "complete". Then try to delete the namespace.

Anything else we need to know?:

Command message:

# What I see when trying to delete the namespace
kubectl delete namespace cert-manager
Error from server (Conflict): Operation cannot be fulfilled on namespaces "cert-manager": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

# What I see when listing objects in the namespace
kubectl get all --namespace cert-manager
No resources found.

# What I see when inspecting the namespace object
kubectl get namespace cert-manager -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"certmanager.k8s.io/disable-validation":"true"},"name":"cert-manager","namespace":""}}
  creationTimestamp: 2019-02-20T10:09:06Z
  deletionTimestamp: 2019-02-22T09:02:27Z
  labels:
    certmanager.k8s.io/disable-validation: "true"
  name: cert-manager
  resourceVersion: "13191191"
  selfLink: /api/v1/namespaces/cert-manager
  uid: 8ea8e65e-34f7-11e9-9519-d2c2f38b7628
spec:
  finalizers:
  - kubernetes
status:
  phase: Terminating

Environment details::

• Kubernetes version (e.g. v1.11.2): v1.11.3
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): Azure
• cert-manager version (e.g. v0.4.0): 0.6
• Install method (e.g. helm or static manifests): Static Manifests

/kind bug

[elbaro]

Experienced the same. There is an upstream issue about zombie namespace. kubectl get all --namespace=cert-manager does not list all objects, so try delete -f ~~.yaml. For me kubectl delete apiservice [cert-manager-related-apiservices] and reboot fixed it.

[woodwardmatt]

Thanks @elbaro that makes sense, so would I just use the original manifest file in that delete command? :)

[woodwardmatt]

Just to resolve this issue. I followed @elbaro's advice above and ran the delete command with the original manifest file. This cleared up any left over objects from the original install and a few moments later the namespace was removed by the K8s.

Now to try the Helm install! :)

[kopaygorodsky]

Status of webhook apiservice is false.
v1beta1.webhook.cert-manager.io cert-manager/cert-manager-webhook False (ServiceNotFound)

This is the fix: kubectl delete apiservice v1beta1.webhook.cert-manager.io

[seunroava]

Even that command didn't work for me @kopaygorodsky

nonroot@cloudshell:~/spinnaker$ kubectl delete apiservice v1beta1.webhook.cert-manager.io
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.webhook.cert-manager.io" not found

[kopaygorodsky]

are there any other hung apiservices from cert-manager ?

[seunroava]

As far as I knew, nothing. But here is a solution that eventually worked for me:

NAMESPACE=cert-manager

kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

I had implemented the first two steps earlier which didn't work. The third command finished it off for me.

Credit to Stackoverflow. Can't remember the link right now.