Cert-manager: Unable to get wildcard cert working

[pms1969]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
although it could just be me not understanding how the dns provider works with route53

What happened:
trying to update cert-manager to 0.3.0-alpha2 in order to enable wildcard certificate support.

NB: names changed to protect the innocent in all samples and logs.

I directed example.com to the kubernetes ingress, as well as *.example.com

What you expected to happen:
Certificate to show up in secrets.

How to reproduce it (as minimally and precisely as possible):
ClusterIssuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-v2
  namespace: ingress
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    dns01:
      providers:
      - name: route53
        route53:
          region: us-east-1

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: redirect-wildcard-ingress
  namespace: prod-tools
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging-v2
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
spec:
  tls:
  - hosts:
    - "*.example.com"
    secretName: redirect-wildcard-tls
  rules:
  - host: "*.example.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: redirector
          servicePort: 80

Anything else we need to know?:

log output:

I0521 11:09:42.363687       1 controller.go:177] certificates controller: syncing item 'tools/redirect-wildcard-tls'
I0521 11:09:42.363778       1 sync.go:239] Preparing certificate tools/redirect-wildcard-tls with issuer
I0521 11:09:42.363788       1 acme.go:159] getting private key (letsencrypt-staging->tls.key) for acme issuer ingress/letsencrypt-staging-v2
I0521 11:09:42.364123       1 prepare.go:247] Cleaning up previous order for certificate tools/redirect-wildcard-tls
I0521 11:09:42.364132       1 prepare.go:263] Cleaning up old/expired challenges for Certificate tools/redirect-wildcard-tls
I0521 11:09:42.364138       1 logger.go:22] Calling CreateOrder
I0521 11:09:43.193474       1 acme.go:193] Created order for domains: [{dns *.example.com}]
I0521 11:09:43.193525       1 logger.go:52] Calling GetAuthorization
I0521 11:09:43.390599       1 prepare.go:263] Cleaning up old/expired challenges for Certificate tools/redirect-wildcard-tls
I0521 11:09:43.390622       1 sync.go:241] Error preparing issuer for certificate tools/redirect-wildcard-tls: ACME server does not allow selected challenge type for domain "example.com"
I0521 11:09:43.396717       1 controller.go:152] ingress-shim controller: syncing item 'tools/redirect-wildcard-ingress'
I0521 11:09:43.396738       1 sync.go:123] Certificate "redirect-wildcard-tls" for ingress "redirect-wildcard-ingress" already exists
I0521 11:09:43.396746       1 sync.go:126] Certificate "redirect-wildcard-tls" for ingress "redirect-wildcard-ingress" is up to date
I0521 11:09:43.396776       1 controller.go:166] ingress-shim controller: Finished processing work item "tools/redirect-wildcard-ingress"
E0521 11:09:43.397085       1 sync.go:168] [tools/redirect-wildcard-tls] Error getting certificate 'redirect-wildcard-tls': secret "redirect-wildcard-tls" not found
E0521 11:09:43.397106       1 controller.go:186] certificates controller: Re-queuing item "tools/redirect-wildcard-tls" due to error processing: ACME server does not allow selected challenge type for domain "example.com"
I0521 11:09:47.013317       1 controller.go:138] clusterissuers controller: syncing item 'letsencrypt-staging-v2'
I0521 11:09:47.013370       1 acme.go:159] getting private key (letsencrypt-staging->tls.key) for acme issuer ingress/letsencrypt-staging-v2
I0521 11:09:47.013699       1 logger.go:67] Calling GetAccount
I0521 11:09:47.909386       1 setup.go:66] letsencrypt-staging-v2: verified existing registration with ACME server
I0521 11:09:47.909411       1 helpers.go:119] Found status change for ClusterIssuer "letsencrypt-staging-v2" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2018-05-21 11:09:47.909405087 +0000 UTC m=+2729.243859956
I0521 11:09:47.916043       1 controller.go:152] clusterissuers controller: Finished processing work item "letsencrypt-staging-v2"
I0521 11:09:47.921378       1 controller.go:138] clusterissuers controller: syncing item 'letsencrypt-staging-v2'
I0521 11:09:47.921413       1 acme.go:159] getting private key (letsencrypt-staging->tls.key) for acme issuer ingress/letsencrypt-staging-v2
I0521 11:09:47.921747       1 logger.go:67] Calling GetAccount
I0521 11:09:48.720910       1 setup.go:66] letsencrypt-staging-v2: verified existing registration with ACME server
I0521 11:09:48.725106       1 controller.go:152] clusterissuers controller: Finished processing work item "letsencrypt-staging-v2"

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**: AWS
• Install tools: terraform / kubectl / helm
• Others:

[Evesy]

@pms1969 I don't believe ingress shim currently supports wildcard certificates, for now you need to create the Certificate resource manually (As I understand it)

[pms1969]

@Evesy Thanks. That makes some sense, although it did look like it was trying. Thought my config might be out in some way, and I did spot something that I need to try (can't quite remember what that is right now tho).

Should I leave this open for tracking?

[munnerz]

It is possible to obtain wildcard certificates using ingress-shim. You will need to deploy cert-manager with flags something like:

helm upgrade --install cm \
        stable/cert-manager \
        --namespace cert-manager \
        --set ingressShim.defaultIssuerName=letsencrypt-prod \
        --set ingressShim.defaultIssuerKind=ClusterIssuer \
        --set ingressShim.defaultACMEChallengeType=dns01 \
        --set ingressShim.defaultACMEDNS01ChallengeProvider=cloudflare

Notice that we are setting a default DNS challenge provider so that we don't have to specify it each time in annotations.

Otherwise, you could use the annotations specified here: https://cert-manager.readthedocs.io/en/latest/reference/ingress-shim.html to achieve the same thing without setting DNS01 as the default challenge mechanism 😄

[pms1969]

thanks @munnerz Will give it a look in the next couple of days.

[munnerz]

Going to close this issue for now - feel free to re-open if the issue persists

[ekryski]

@munnerz thanks for your advice.

For the most part I got this working using the ingress-shim! Provisions the cert, updates ingress, etc. 👌

However there is one issue:

If I only specify the wildcard domain in the tls section of my ingress the default Fake Kubernetes cert is used. If I specify the wildcard domain and the full domain in the ingress nginx-ingress resolves the correct TLS cert, but then cert-manager complains about the domain name being redundant.

Specs

Ingress:

spec:
  rules:
  - host: hello.staging.mydomain.com
    http:
      paths:
      - backend:
          serviceName: hello-service
          servicePort: 80
        path: /
  tls:
  - hosts:
    - '*.staging.mydomain.com'
    secretName: staging-tls

This provisions certs just fine and ingress and generated certs seem to be included by nginx properly when looking at logs but nginx-ingress serves up the default Fake Kubernetes cert.

Error

If I add both like this:

spec:
  rules:
  - host: hello.staging.mydomain.com
    http:
      paths:
      - backend:
          serviceName: hello-service
          servicePort: 80
        path: /
  tls:
  - hosts:
    - '*.staging.mydomain.com'
    - hello.staging.mydomain.com
    secretName: staging-tls

I get this error:

Error preparing issuer for certificate default/staging-tls: acme: urn:ietf:params:acme:error:malformed: Error creating new order :: Domain name "hello.staging.mydomain.com" is redundant with a wildcard domain in the same request. Remove one or the other from the certificate request.

Versions

nginx-ingress-controller: 0.15.0
cert-manager: 0.3.0

---

Any insight? This seems like it is a defect with nginx-ingress not properly mapping a host to a wildcard domain defined in tls. Not sure if that is actually the case. There seem to be a bunch of issues with wildcards in there.

However, a safe way to guard against any issues here would be that we could ignore ignore domains that are already covered by the wildcard cert?

[gaelleiadvize]

Hi :)

I got the same issue using Heptio Contour ingress ( on GKE)
When I specify the subdomain I get the redondant subdomain log message

To temporarily fix it I comment those annotations (disable auto)

ingress-rs.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: contour-rs-ingress namespace: production annotations: kubernetes.io/ingress.class: "contour" ingress.kubernetes.io/force-ssl-redirect: "true" # certmanager.k8s.io/cluster-issuer: letsencrypt-prod # certmanager.k8s.io/acme-challenge-type: dns01 # certmanager.k8s.io/acme-dns01-provider: clouddns # kubernetes.io/tls-acme: "true"

And I update my certificate (to remote redondant subdomains) :
certificate-prod.yaml

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  clusterName: ""
  name: tls-prod
  namespace: production
spec:
  secretName: tls-prod
  commonName: '*.mydomain.com'
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  dnsNames:
  - mydomain.com
  acme:
    config:
    - dns01:
        provider: clouddns
      domains:
      - '*.mydomain.com'
      - mydomain.com'

I'm not sure I'm going in the right way by doing this ?