Cert-manager: Ingress not cleaned up after certificate is issued

[sfriquet]

Describe the bug:
Ingress is not cleaned up after certificate is issued, as seen through the following warning:

$ kubectl describe -f ngress.yml 
...
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason  Message
  --------- --------    -----   ----            -------------   --------    ------  -------
...
  18m       7m      19  loadbalancer-controller         Warning     Sync    Error during sync: error while evaluating the ingress spec: could not find service "default/cm-acme-http-solver-j7ljq"

Note that the certificate is issued:

  FirstSeen LastSeen    Count   From        SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----        -------------   --------    ------      -------
...
  39m       39m     1   cert-manager            Normal      IssueCert   Issuing certificate...
  39m       39m     1   cert-manager            Normal      CertObtained    Obtained certificate from ACME server
  39m       39m     1   cert-manager            Normal      CertIssued  Certificate issued successfully

Expected behaviour:
Ingress should no longer reference services created by cert-manager after certificate is issued.

Steps to reproduce the bug:
I followed the instructions here: https://github.com/ahmetb/gke-letsencrypt/blob/master/40-deploy-an-app.md (with the exception that cert-manager version was latest)

Anything else we need to know?:

Environment details::

• Kubernetes version: 1.10.5-gke.4
• Cloud-provider/provisioner: GKE
• cert-manager version: v0.4.1
• Install method: helm install --name cert-manager --namespace kube-system stable/cert-manager

/kind bug

[munnerz]

Are you able to provide your logs from around the time of the issuance to help debug?

[sfriquet]

Sure, logs for cert-manager: https://pastebin.com/G2S7MdFx

This is how the Certificate resource looked like:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: mydomain-com-tls
  namespace: default
spec:
  secretName: mydomain-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: sub.domain.mydomain.com
  dnsNames:
  - sub.domain.mydomain.com
  - subdomain.mydomain.com
  acme:
    config:
    - http01:
        ingress: helloweb
      domains:
      - sub.domain.mydomain.com
      - subdomain.mydomain.com

Note that there were 2 domains being validated, I was trying with both 1 and 2 levels of subdomain, if that matters.

[munnerz]

Thanks 😄 could you also share a copy of your Ingress resource? I've found this error in your log that looks like the culprit: E certificates controller: Re-queuing item "default/mydomain-com-tls" due to error processing: [Ingress.extensions "helloweb" is invalid: spec.rules[0].http.paths: Required value, Ingress.extensions "helloweb" is invalid: spec.rules[0].http.paths: Required value]

[sfriquet]

The ingress (prior to adding tls rule):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloweb
  annotations:
    kubernetes.io/ingress.global-static-ip-name: cert-test-1-ip
  labels:
    app: hello
spec:
  backend:
    serviceName: helloweb-backend
    servicePort: 8080

[sfriquet]

I've tried reproducing using a simpler Certificate

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: mydomain-com-tls
  namespace: default
spec:
  secretName: mydomain-com-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: sub.domain.mydomain.com
  dnsNames:
  - sub.domain.mydomain.com
  acme:
    config:
    - http01:
        ingress: helloweb
      domains:
      - sub.domain.mydomain.com

Here's what the Ingress looks like in full:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ...
    kubernetes.io/ingress.global-static-ip-name: cert-test-1-ip
  creationTimestamp: 2018-08-13T15:51:29Z
  generation: 6
  labels:
    app: hello
  name: helloweb
  namespace: default
  resourceVersion: "38732"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/helloweb
  uid: be037b21-9f10-11e8-b48d-42010a8e00d3
spec:
  backend:
    serviceName: helloweb-backend
    servicePort: 8080
  rules:
  - host: sub.domain.mydomain.com
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-9gdw7
          servicePort: 8089
        path: /.well-known/acme-challenge/7uBaUxj-5-OEx9_hpQLSXG8W57umwbp6MomKPYMSX1E
status:
  loadBalancer:
    ingress:
    - ip: 35.190.17.203

And indeed doing kubectl describe certificate.yml shows

Message:            Ingress.extensions "helloweb" is invalid: spec.rules[0].http.paths: Required value

[munnerz]

Thanks for the comprehensive response 😄

I have opened #831 which adds a unit test for your particular configuration, and includes a patch to fix the failing test!

[munnerz]

If you can try running the :canary image of cert-manager and report back, hopefully your issue is fixed 😄

[sfriquet]

Thanks for the quick turnaround!
Do you mean install cert-manager from https://github.com/jetstack/cert-manager/blob/master/contrib/manifests/cert-manager/with-rbac.yaml ?

[munnerz]

Yep that should do it!

[sfriquet]

I can confirm the ingress is properly cleaned up with the canary image. Thanks a lot.