Cert-manager: Order stuck in pending

[jaitaiwan]

Describe the bug:
A fresh installation of v0.8.0 of cert-manager via helm in it's own namespace "cert-manager". We used the following values to enable the ingress-shim behaviour:

---
ingressShim.defaultIssuerName: letsencrypt
ingressShim.defaultIssuerKind: ClusterIssuer

A letsencrypt ClusterIssuer is installed:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt
  namepace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt
    https01: {}

Note we previously were running 0.5.2 (I think) with a staging ClusterIssuer to check this was all working and it was fine. So we installed the production version and it worked fine.

An ingress is automatically created when we deploy our application via helm:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/acme-challenge-type: http01
    certmanager.k8s.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:80;
      proxy_hide_header l5d-remote-ip;
      proxy_hide_header l5d-server-id;
    nginx.ingress.kubernetes.io/cors-allow-headers: '*'
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: quiz-calculator
    chart: quiz-calculator-0.1.0
    heritage: Tiller
    release: feature-staging-branch-quiz-calculator
  name: feature-staging-branch-quiz-calculator
  namespace: my-app
spec:
  rules:
  - host: feature-staging-branch-calculator.staging.example.com
    http:
      paths:
      - backend:
          serviceName: feature-staging-branch-quiz-calculator
          servicePort: http
        path: /
  tls:
  - hosts:
    - feature-staging-branch-calculator.staging.example.com
    secretName: feature-staging-branch-calculator-api-cert

A pending order can be found in the my-app namespace as pending as expected. However the order is never updated.

The cert-manager pods show this in the logs:

I0522 21:34:30.560547       1 sync.go:181] Certificate "feature-staging-branch-calculator-api-cert" for ingress "feature-staging-branch-quiz-calculator" already exists
I0522 21:34:30.560614       1 sync.go:184] Certificate "feature-staging-branch-calculator-api-cert" for ingress "feature-staging-branch-quiz-calculator" is up to date
I0522 21:34:30.560631       1 controller.go:184] ingress-shim controller: Finished processing work item "my-app/feature-staging-branch-quiz-calculator"

Expected behaviour:
A certificate is created and the ingress is updated to use the letsencrypt certificate.

Steps to reproduce the bug:
As above.

Anything else we need to know?:
Events shows as <none> when using kubectl describe -n my-app order

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.13.5-gke.10
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
• cert-manager version (e.g. v0.4.0): v0.8.0
• nginx-ingress version: v0.24.1 (my-app namespace)
• Install method (e.g. helm or static manifests): helm

/kind bug

[PabloCastellano]

I've found the same behaviour in 0.7.2. Previously we were using 0.5.0.

[jaitaiwan]

@PabloCastellano I think maybe we were using 0.7.0 and it worked fine? Maybe give that a go in the meantime. 0.7.0 seemed pretty buggy still tho

[PabloCastellano]

@jaitaiwan Turns out our customer had a misconfiguration. He was trying to issue a certificate for two domains, one of them had georouting policy for US, which affected Let's Encrypt validation and the resolved host didn't properly forward port 80 so Let's Encrypt timed out while trying to resolve the challenge. When I isolated this domain alone it was when I started to see the error message.

I could provide more information and try to reproduce it in the smallest configuration but I don't think it's worth it. cert-manager could probably handle this situation better but it was working fine :+1:

[brettcat]

I had this same issue with a bunch of versions from 0.7.0 up to 0.8.1 to canary. Turns out the annotation on the ingress doesn't work as I believe it's supposed to. I'm using route53 DNS01 with IAM (no access/secret key in the cluster issuer object)

I had to manually add the cert with the same secretName that the ingress object wants to use

This is the ingress object. The certmanager.k8s.io/cluster-issuer annotation will not create the challenges k get challenges and the orders k get orders stay in pending state. The secret portal-wildcard is only self-signed.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: ""
  labels:
    app: fe-app
    chart: microservice-base-0.1.1
    heritage: Tiller
    release: purple-fe-app
  name: fe-app-portal
spec:
  rules:
  - host: '*.some.domain.com'
    http:
      paths:
      - backend:
          serviceName: fe-app
          servicePort: 81
        path: /
  tls:
  - hosts:
    - '*.some.domain.com'
    secretName: portal-wildcard

If I add the certificate object, The DNS01 challenge happens and the certificate is valid.

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: wildcard-secure-domain-com
spec:
  secretName: portal-wildcard
  dnsNames:
  - '*.some.domain.com'
  acme:
    config:
    - dns01:
        provider: route53
      domains:
      - '*.some.domain.com'
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Hope this helps anyone

[jaitaiwan]

For anyone who comes here later... I should have written http01 not https01