Cert-manager: Failed to determine Route 53 hosted zone ID: SignatureDoesNotMatch

[jbartus]

/kind bug

What happened:
cert-manager gets a 403 SignatureDoesNotMatch error when trying to interact with the AWS route53 API.

# kubectl logs `kubectl get pods | grep cert-manager | awk '{print $1}'` | tail
I0621 01:29:53.288113       1 logger.go:47] Calling GetChallenge
I0621 01:29:53.335912       1 dns.go:78] Checking DNS propagation for "example.com" using name servers: [10.39.240.10:53]
I0621 01:29:53.414515       1 dns.go:85] DNS record for "example.com" not yet propagated
I0621 01:29:53.415040       1 dns.go:72] Presenting DNS01 challenge for domain "example.com"
I0621 01:29:53.583171       1 helpers.go:162] Found status change for Certificate "wildcard-example-com" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-06-21 01:29:53.583142985 +0000 UTC m=+9532.608235815
I0621 01:29:53.583404       1 sync.go:241] Error preparing issuer for certificate default/wildcard-example-com: Failed to determine Route 53 hosted zone ID: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
        status code: 403, request id: 992ef882-74f2-11e8-8fbe-453cacca4e3a
E0621 01:29:53.594554       1 sync.go:168] [default/wildcard-example-com] Error getting certificate 'wildcard-example-com-tls': secret "wildcard-example-com-tls" not found
E0621 01:29:53.594937       1 controller.go:186] certificates controller: Re-queuing item "default/wildcard-example-com" due to error processing: Failed to determine Route 53 hosted zone ID: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
        status code: 403, request id: 992ef882-74f2-11e8-8fbe-453cacca4e3a

What you expected to happen:
It would use the defined credentials to make the API calls successfully.

How to reproduce it (as minimally and precisely as possible):
on a fresh gke 1.10 cluster with helm

# helm install --name cert-manager stable/cert-manager  --version v0.3.2

# cat example-com-r53-creds.yaml
apiVersion: v1
kind: Secret
metadata:
  name: example-com-r53-creds
type: Opaque
data:
  secret-access-key: <redacted>

# kubectl apply -f example-com-r53-creds.yaml

# cat letsencrypt-clusterissuer.yaml
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    dns01:
      providers:
        - name: example-com-r53
          route53:
            region: us-east-1
            accessKeyID: AKIAABCDEFGHIJKLMNO
            secretAccessKeySecretRef:
              name: example-com-r53-creds
              key: secret-access-key

# kubectl apply -f letsencrypt-clusterissuer.yaml

# cat wildcard-example-com-cert.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: default
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: '*.example.com'
  acme:
    config:
    - dns01:
        provider: example-com-r53
      domains:
      - '*.example.com'

# kubectl apply -f wildcard-example-com-cert.yaml

# kubectl logs `kubectl get pods | grep cert-manager | awk '{print $1}'` | tail -f

Anything else we need to know?:
I wrote a test to prove that the values stored in kubernetes work to make the route53 api calls

#!/bin/bash

export AWS_ACCESS_KEY_ID=$(kubectl get clusterissuer letsencrypt-staging -o jsonpath='{.spec.acme.dns01.providers[].route53.accessKeyID}')
SECRET_NAME=$(kubectl get clusterissuer letsencrypt-staging -o jsonpath='{.spec.acme.dns01.providers[].route53.secretAccessKeySecretRef.name}')
SECRET_KEY=$(kubectl get clusterissuer letsencrypt-staging -o jsonpath='{.spec.acme.dns01.providers[].route53.secretAccessKeySecretRef.key}')
export AWS_SECRET_ACCESS_KEY=$(kubectl get secret $SECRET_NAME -o jsonpath="{.data.$SECRET_KEY}")
ZONE=$(kubectl get cert wildcard-example-com -o jsonpath='{.status.acme.order.challenges[].domain}')
ZONE_ID=$(aws route53 list-hosted-zones | jq -r --arg ZONE "$ZONE." '.HostedZones[] | select(.Name==$ZONE) | .Id' | awk -F\/ '{ print $3}')
RAND_NUM=$(cat /dev/urandom | tr -dc '0-9' | fold -w 256 | head -n 1 | head --bytes 3)

echo "current value: $(dig +short txt $ZONE)"
echo "updating to: \"$RAND_NUM\""

CHANGE_ID=$(aws route53 change-resource-record-sets \
                --hosted-zone-id $ZONE_ID \
                --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"'$ZONE.'","Type":"TXT","TTL":1,"ResourceRecords":[{"Value":"\"'$RAND_NUM'\""}]}}]}' \
                | jq -r '.ChangeInfo.Id')

aws route53 wait resource-record-sets-changed --id $CHANGE_ID

echo "after value: $(dig +short txt $ZONE)"

and it works

# ./test-r53-update.sh
current value: "066"
updating to: "486"
after value: "486"

Environment:

• Kubernetes version (use kubectl version): v1.10.2-gke.3
• Cloud provider or hardware configuration**: GKE 1.10
• Install tools: helm
• Others: just a note, I'm not actually trying to use example.com I've obfuscated my domain.

[ghost]

I found this problem also.

My domain name is the same but it's separated into 2 types, "Public" and "Private". The HostzonedIDs are start with "Z32" and "Z20" as you can see in the following picture.

When I create certificate for my domain, I specify Public HostzonedID but I get the error related to ns-0.awsdns-00.com. The nameserver is NS value of Private domain not the domain that I specify HostzonedID

[munnerz]

I'm not particularly familiar with the AWS APIs workings here - that said, @korkeat the issue you describe is why we have the issuer.acme.dns01.providers.route53.hostedZoneID parameter (to help cert-manager disambiguate which zone to update if there is more than one listed in the route53 account).

@jbartus did you make any progress with this issue/manage to resolve it? If so, would you be able to share any findings? 😄

[jbartus]

@munnerz I just tried again with the 0.4.1 helm chart but I get the exact same error. I tried adding the hostedZoneID to the issuer, and I get

controller.go:190] certificates controller: Re-queuing item "default/wildcard-example-com" due to error processing: Failed to change Route 53 record set: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

because the credentials work in an aws-cli profile, and because the error "moves" to whichever api call happens first, it seems that the problem is client side in generating/formatting the request. I'm not sure how to debug further.

[jbartus]

A very kind stranger (@awsashish) hit me up on the Kubernetes slack with the solution:

I had just pasted my aws secret access key into the yaml (the part that says <redacted> in my post above), but it needs to be base64 encoded!

https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually