Cert-manager: TLS secret is not being created when using ClusterIssuer and ingress-shim

[novakg]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:
I have chart "cert-manager-0.2.4" installed.
I've installed it without rbac:

helm install --namespace kube-system --set rbac.create=false stable/cert-manager

I've created a cluster wide issuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-cluster-issuer
spec:
  acme:
    server: https://acme-staging.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    http01: {}

I have the following ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging-cluster-issuer
    ingress.kubernetes.io/proxy-body-size: 2048m
    nginx.org/proxy-connect-timeout: 30s
    nginx.org/proxy-read-timeout: 300s
  creationTimestamp: 2018-03-22T10:43:00Z
  generation: 16
  labels:
    app: goodly-kangaroo-letsencrypt-test
    chart: letsencrypt-test-0.1.5
    heritage: Tiller
    release: goodly-kangaroo
  name: goodly-kangaroo-httpserver
  namespace: test
  resourceVersion: "33812443"
  selfLink: /apis/extensions/v1beta1/namespaces/test/ingresses/goodly-kangaroo-httpserver
  uid: ca5b839c-2dbd-11e8-b51c-fa163e7911c3
spec:
  rules:
  - host: cert-test.mydomain.eu
    http:
      paths:
      - backend:
          serviceName: goodly-kangaroo-httpserver
          servicePort: 8080
        path: /
      - backend:
          serviceName: cm-test-letsencrypt-ssl-secret-enasc
          servicePort: 8089
        path: /.well-known/acme-challenge/IEpp9_YzfkasRZP8dLQVUf51YbN-Bgqo2x_yNv1Kc4g
  tls:
  - hosts:
    - cert-test.mydomain.eu
    secretName: test-letsencrypt-ssl-secret
status:
  loadBalancer:
    ingress:
    - ip: 192.168.0.220

http://cert-test.mydomain.eu works fine.
http://cert-test.mydomain.eu/.well-known/acme-challenge/IEpp9_YzfkasRZP8dLQVUf51YbN-Bgqo2x_yNv1Kc4g also works fine
BUT https://cert-test.mydomain.eu/ does not work (have no certificate). I can see the followings in the logs.

nginx ingress pod:

W0322 13:35:02.170376   25961 backend_ssl.go:46] error obtaining PEM from secret test/test-letsencrypt-ssl-secret: secret named test/test-letsencrypt-ssl-secret does not exist

cert-manager container:

I0322 13:35:38.565077       7 sync.go:242] Error preparing issuer for certificate: error waiting for key to be available for domain "cert-test.mydomain.eu": context deadline exceeded
E0322 13:35:38.621398       7 sync.go:190] [test/test-letsencrypt-ssl-secret] Error getting certificate 'test-letsencrypt-ssl-secret': secret "test-letsencrypt-ssl-secret" not found
E0322 13:35:38.621476       7 controller.go:196] certificates controller: Re-queuing item "test/test-letsencrypt-ssl-secret" due to error processing: error waiting for key to be available for domain "cert-test.mydomain.eu": context deadline exceeded
I0322 13:35:38.621555       7 controller.go:187] certificates controller: syncing item 'test/test-letsencrypt-ssl-secret'
I0322 13:35:38.621661       7 sync.go:107] Error checking existing TLS certificate: secret "test-letsencrypt-ssl-secret" not found
I0322 13:35:38.621734       7 sync.go:238] Preparing certificate with issuer
I0322 13:35:38.622728       7 prepare.go:239] Compare "" with "https://acme-staging.api.letsencrypt.org/acme/reg/5581886"

ingress-shim container:

I0322 13:35:38.567766      19 controller.go:147] ingress-shim controller: syncing item 'test/goodly-kangaroo-httpserver'
I0322 13:35:38.569372      19 sync.go:85] Certificate "test-letsencrypt-ssl-secret" for ingress "goodly-kangaroo-httpserver" already exists, not re-creating
I0322 13:35:38.569407      19 controller.go:161] ingress-shim controller: Finished processing work item "test/goodly-kangaroo-httpserver"
I0322 13:35:39.342743      19 controller.go:147] ingress-shim controller: syncing item 'test/goodly-kangaroo-httpserver'
I0322 13:35:39.342807      19 sync.go:85] Certificate "test-letsencrypt-ssl-secret" for ingress "goodly-kangaroo-httpserver" already exists, not re-creating
I0322 13:35:39.342829      19 controller.go:161] ingress-shim controller: Finished processing work item "test/goodly-kangaroo-httpserver"

Am I doing something wrong?
My understanding is that the "test-letsencrypt-ssl-secret" secret should be created by the cert-manager/shim. Is that correct?

I couldn't find instructions in the documentation about how to define the ingress. First I tried without the tls clause as I thought that the cert manager will add that part automatically. Then I added the tls part, but without the secretName definition... I think it would be good to have an example ingress definition in the doc.

What you expected to happen:

Certificate is automatically created through Let's Encrypt for each new ingress definition that has the certmanager.k8s.io/cluster-issuer: "letsencrypt-staging-cluster-issuer" annotation.

How to reproduce it (as minimally and precisely as possible):

Set up a ClusterIssuer and use it through shim.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3+coreos.0", GitCommit:"42de91f04e456f7625941a6c4aaedaa69708be1b", GitTreeState:"clean", BuildDate:"2017-08-07T19:44:31Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**:
  OpenStack Newton
• Install tools:
  kubeadm, helm v2.8.1
• Others:
  The httpserver I'm using in the test chart is: adejonge/helloworld:latest

[opaolini]

Bumping,

I have the exact same problem using ClusterIssuer and ingress-shim.

The Certificate resource is created, but the resulting tls secret is not.

[bananaspliff]

check if cert-manager container can retrieve certificate from your https service like outside the cluster.
otherwise the certificate remains in status Ready False
i have solve this problem just with a simple NAT rule on my host

[thearrow]

Likewise, I would be grateful for an example of a working TLS ingress definition using ingress-shim and a clusterissuer.

[opaolini]

I have solved this problem. As @vdbxavier correctly pointed out, you should make sure that cert-manager is able to connect to your service.

In my case it was that AWS Route53 had a Public and a Private Hosted zone (managed via TerraForm) that shared the domain example.com. When trying to do a challenge for test.example.com it was bouncing against the private one not correctly resolving the address.

[novakg]

@vdbxavier could you please explain how to check that the container can retrieve the certificate?
What I did is that I went into the container that run quay.io/jetstack/cert-manager-controller:v0.2.3 image and did:

wget http://cert-test.mydomain.eu/.well-known/acme-challenge/MQFtVTw-jHLjuu68-DRZGYmMVGHdcNgdTY_NHuWhu8Q

That gave me the same response as what I get when I enter the that url in the browser. So the cert manager container can access that challenge fine.
Did you mean this when you talked about retrieving the certificate, or something else?

[novakg]

Hi @opaolini,
How did you find it out that "it was bouncing against the private one not correctly resolving the address"?

[bananaspliff]

@novakg just attach docker container and check with ping / wget / curl
do not check http, check if https service target is OK (wget https://cert-test.mydomain.eu with insecure option)

[opaolini]

@novakg What I did is is kubectl exec -it -n <namespace> <cert-manager-pod> cert-manager sh and tried nslookup test.example.com which in my case yielded an unknown / bad address. After fixing that DNS issue I did the same and coupled that with a wget and all was :ok_hand:

[balboah]

Another hint that happened for me was that cert manager didn't observe that I changed the secretName in the ingress tls section. Had to manually delete the pod

[novakg]

I can confirm that the problem was in my network set-up, not in cert manager: the cert manager was trying to check the exposed token through the external IP of our system which was routed incorrectly from inside the network. Cert manager works fine after I defined some DNS views in our DNS server with correct routings that make cert manager see the exposed tokens correctly.

Thanks everyone for the help.

[mikesparr]

@balboah and @novakg - I'm facing same issue when upgrading to cert manager from kube-lego. Traffic still encrypted fine but I notice the cert returned is the old one produced by kube-lego and not the new one. I updated the secretName in the Ingress, deleted pods and let deployment recreate them, and re apply -f ingress-file.yml all without errors but still not serving up the new certs.

In GCloud console under Services tab, I see this error below. I see some people speaking about DNS updates but unsure what/where to look so specific solutions would be useful.

Confirmed dns resolve issue

I have confirmed that dns doesn't resolve in the cert-manager pod, but other pods in the same cluster do resolve the domain. They all share the same resolv.conf values so what DNS config did you guys do to solve this?

[goloxxly]

You should check for errors in the logs of the cert-manager pod, and also check the status of the cert.

You might also could try to (backup and then) delete the "letsencrypt-private-key". Cert-manager will create the secret once it's able to update the cert from Let's Encrypt.

Do you say in your DNS related comments that you cannot curl the test..../.well-known/acme-challenge.... url from the cert-manager pod?

[mikesparr]

Hi thanks for the reply. I cannot curl the test from within the cert-manager pod and it says nslookup: can't resolve '(null)': Name does not resolve

However in another pod in same cluster with same resolv.conf it resolves without issue. Unsure what is different about the helm chart that prevents it and I saw others above say they solved with DNS config but no one actually said what they did.

One person on Kubernetes Slack workspace suggested adding this to helm chart:
--set extraArgs={"--dns01-recursive-nameservers=8.8.8.8:53,4.4.4.4:53"}

[mikesparr]

I found the issue and the DNS was just a smokescreen. In the ingress file I mistakenly named the secretName value to the same secret as the issuer in the annotations, and that was created in a non-default namespace. I deleted all and fixed and seems to be working (at least staging. trying production next). thanks again for reply and suggestion. Hope this helps others.

Summary of solution

• https://github.com/ahmetb/gke-letsencrypt/blob/master/30-setup-letsencrypt.md

• give tiller role for cluster the cluster-admin privileges after your user (if GKE, then via gcloud) has elevated privileges

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

1. helm init --service-account tiller
2. Install Cert Manager

# Install the CustomResourceDefinition resources separately
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml

# Create the namespace for cert-manager
kubectl create namespace cert-manager

# Label the cert-manager namespace to disable resource validation
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install \
  --name cert-manager \
  --namespace cert-manager \
  stable/cert-manager

1. Create ServiceIssuer (or ClusterIssuer) like link above yaml file
2. Create or edit Ingress with annotation secret same name as Issuer secret key, then secretName some arbitrary name where you store credentials after Lets Encrypt issues them

WAIT 15-20 minutes

• kubectl describe certificate and events at end show status

[thecodeassassin]

Hi thanks for the reply. I cannot curl the test from within the cert-manager pod and it says nslookup: can't resolve '(null)': Name does not resolve

However in another pod in same cluster with same resolv.conf it resolves without issue. Unsure what is different about the helm chart that prevents it and I saw others above say they solved with DNS config but no one actually said what they did.

One person on Kubernetes Slack workspace suggested adding this to helm chart:
--set extraArgs={"--dns01-recursive-nameservers=8.8.8.8:53,4.4.4.4:53"}

Small note here, I think it should be

--set extraArgs={"--dns01-recursive-nameservers=8.8.8.8:53,8.8.4.4:53"}

This did solve my issue, but it's 8.8.4.4 and not 4.4.4.4