Cert-manager: Error getting secret for Route53 access

[paulbdavis]

/kind bug

What happened:

Got an error trying to set up the Route53 DNS verification

E0508 22:30:23.777194       1 prepare.go:167] Error cleaning up solver: error getting route53 secret access key: secret "route53-credentials" not found

What you expected to happen:

Cert to be issued via Route53 DNS verification

How to reproduce it (as minimally and precisely as possible):

I installed the chart with helm upgrade --install --namespace cert-manager cert-manager stable/cert-manager

Set up a ClusterIssuer as so:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  # Adjust the name here accordingly
  name: letsencrypt
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: {{ .Values.acme.url }}
    # Email address used for ACME registration
    email: {{ .Values.acme.email }}
    # Name of a secret used to store the ACME account private key from step 3
    privateKeySecretRef:
      name: letsencrypt-issuer-key
    dns01:
      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:
      - name: prod-dns
        route53:
          region: {{ .Values.aws.region }}
          # optional if ambient credentials are available; see ambient credentials documentation
          accessKeyID: {{ .Values.aws.accessKeyID }}
          secretAccessKeySecretRef:
            name: route53-credentials
            key: secret-access-key

I created a secret in the same namespace

kubectl --namespace cert-manager create secret generic route53-credentials --from-literal="secret-access-key=$aws_secret"

Then issued a cert

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: foobar-ltd
  namespace: default
spec:
  secretName: tls-foobar-ltd
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt
  commonName: foobar.ltd
  dnsNames: 
  - www.foobar.ltd
  acme:
    config:
    - dns01:
        provider: prod-dns
      domains:
      - foobar.ltd
      - www.foobar.ltd

Anything else we need to know?:

Environment:

• Kubernetes version: 1.9.3
• Cloud provider: AWS
• Install tools: kops, helm
• Others: [email protected]

[pgold30]

Same issue here , im getting : [cert-manager-cert-manager-656fc7477f-r6pvd cert-manager] E0518 13:09:43.720975 1 controller.go:196] certificates controller: Re-queuing item "default/echo-sandbox-cluster-XXXX-com" due to error processing: error presenting acme authorization for domain "echo.sandbox.cluster.XXXX.com": error getting route53 secret access key: secret "route53" not found
also i created the secret manually but its not working either

[dhawton]

Same thing but for cloudflare.

[darkyat]

Did you make sure you deployed the secret into the right namespace? I struggled with the error above due the fact that I deployed the secret into the default namespace, whereas cert-manager expected it to be in kube-system.

[dhawton]

I put mine in all namespaces.

[holmesb]

Me too on route53, created secret in all namespaces:

Warning  ErrorPrepareCertificate  <invalid> (x6 over 0s)  cert-manager-controller  Error preparing issuer for certificate: error presenting acme authorization for domain "foo.example.com": error getting route53 secret access key: key 'route53-credentials-secret' not found in secret

user@hostname~/$ kubectl get secret --all-namespaces |grep route53-credentials-secret
default        route53-credentials-secret              Opaque                                1         3m
kube-public    route53-credentials-secret              Opaque                                1         3m
kube-system    route53-credentials-secret              Opaque                                1         2m

user@hostname~/$ kubectl get namespace
NAME           STATUS    AGE
default        Active    12d
kube-public    Active    12d
kube-system    Active    12d

[euank]

@holmesb Your error is different from the others here. It was able to find the secret, but couldn't find the key in it, e.g. if your issuer has:

dns01:
  providers:
  - name: myroute53provider
    route53:
      secretAccessKeySecretRef;
        name: "name-of-secret"
        key: "name-of-key-in-secret"

the error message is saying it can find the secret, but it can't find a key named "route53-credentials-secret" inside it.

If you do kubectl describe secret route53-credentials-secret it should have fields under Data, and the error message is saying that there isn't a field there named "route53-credentials-secret".

My guess is you want to change your "key" value in your issuer to whatever the data field's name currently is.

[holmesb]

Thanks @euank, yes had the wrong key name. Working now :-)

[paulbdavis]

I was able to work around the issue by deploying the cert manager chart in kube-system and putting the secret there as well.

[Tarick]

In my case creating secret in the same namespace as the Issuer helped, hope it will save somebody some time.

[munnerz]

Sounds like this was down to placing the secret resources into the wrong namespace.

The docs do lightly mention this here: https://cert-manager.readthedocs.io/en/latest/reference/clusterissuers.html

When referencing a Secret resource in ClusterIssuer resources (eg apiKeySecretRef) the Secret needs to be in the same namespace as the cert-manager controller pod. You can optionally override this by using the --cluster-resource-namespace argument to the controller.

[paulbdavis]

@munnerz In the original issue, I installed all elements into the same namespace (except the cert itself, as the issuer is a ClusterIssuer). I am going to be deploying it again so I will double check and see if this has actually been fixed (or at least get more info from the logging changes)

[mkxxq]

Sounds like this was down to placing the secret resources into the wrong namespace.

The docs do lightly mention this here: https://cert-manager.readthedocs.io/en/latest/reference/clusterissuers.html

When referencing a Secret resource in ClusterIssuer resources (eg apiKeySecretRef) the Secret needs to be in the same namespace as the cert-manager controller pod. You can optionally override this by using the --cluster-resource-namespace argument to the controller.

the document link is here: https://cert-manager.io/docs/faq/cluster-resource/