Cert-manager: authentication error on route53

[acartapanis]

/kind bug

What happened:
cert-manager fails to create the aws route53 records.

What you expected to happen:
cert-manager create an aws route53 record for DNS-01 challenge.

How to reproduce it (as minimally and precisely as possible):
try to create a certificate with dns01 on route53.

Anything else we need to know?:
The cert-manager give me the following error :

E1128 19:20:50.525952       1 controller.go:196] certificates controller: Re-queuing item "xxxxx/my-domain-com-cert" due to error processing: error presenting acme authorization for domain "XXX": Failed to change Route 53 record set: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
status code: 403, request id:xxx

The credentials are OK, i can use the aws cli with them.

The issuer :
```apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: letsencrypt-staging
namespace: xxxxx
spec:
acme:
server: https://acme-staging.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-staging
dns01:
# Here we define a list of DNS-01 providers that can solve DNS challenges
providers:
- name: route53
route53:
# The Route53 access key ID
accessKeyID: XXXXXXXX
# A secretKeyRef to a the route53 secret access key
secretAccessKeySecretRef:
key: password.txt
name: cert-manager-secret-access-key
# The Route53 region
region: eu-west-1
hostedZoneID: XXXXXX

The certificate request :
```apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: my-domain-com-cert
  namespace: xxxxx
spec:
  secretName: my-domain-com-sample-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: my.domain.com
  dnsNames:
  - my.domain.com
  acme:
    config:
    - dns01:
        provider: route53
      domains:
      - my.domain.com

I can say that i have a hosted zone "domain.com" declared in route53, but no record for "my.domain.com".

Environment:

• Kubernetes version (use kubectl version): 1.8.3
• Cloud provider or hardware configuration**: aws
• Install tools: helm charts
• Others:

[acartapanis]

Juste to add information : the secret is generated with the command :

kubectl create secret generic cert-manager-secret-access-key --from-file=./password.txt -n xxxxx

where the password.txt file only contains the aws secret access key as plain text.

[euank]

Random shot in the dark, but can you double check you don't have a trailing newline in the file?

This command should print a character, not a space or newline:

$ kubectl get secret cert-manager-secret-access-key -o=go-template --template '{{index .data "password.txt"}}' | base64 -d | tail -c 1

[acartapanis]

I checked the password file and there is not trailing newline nor any other blank character, and I still have the same error...

[acartapanis]

Is there someone that use cert-manager on an aws cluster (created with kops) ? Maybe i miss a specific IAM policy or something like this ?

[euank]

Since cert-manager configures the aws sdk explicitly with the access/secret key given and doesn't use the default config/credential chain, the environment (aws/kops/whatever) shouldn't make a difference.

That error also won't be an IAM policy thing since it's not "permission denied", but "signature mismatch".
The only things I can think of that would cause that error are:

1. typo in secret key
2. time on the computer making the request being off by over 5 minutes
3. Bug in the code (here or the sdk).

I've successfully used the route53 issuer for my certs and my configuration isn't significantly different, so there's not an obvious / overt bug in the code.

To rule out number 1 and 2, you've used the aws cli to successfully make a request with those credentials on the same machine?

[acartapanis]

Yes absolutely, I made requests with those credentials, not on the same machine but with an other machine in the same cluster.
I will try to do so on the same machine ASAP.

[baldeynz]

Did anyone get any further with this one? i have the same thing happening as well with a very similar setup except im on kubernetes 1.9.2

status code: 403, request id: 88de6b5b-0d07-11e8-90e8-1bbdfd297ccb, error presenting acme authorization for domain "mydomain.": Failed to change Route 53 record set: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

[baldeynz]

Please ignore my last comment, my issue was due to a whitespace at the end of the secret.

[roshan-mr]

I am also experiencing the same issue. Not sure what is going wrong here

_Warning ErrorPrepareCertificate Error preparing issuer for certificate: error presenting acme authorization for domain "domain.com": Failed to change Route 53 record set: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
status code: 403, request id: 10440bad-16d5-11e8-bb0d-f371b11ccc4f_

[gianrubio]

Same issue here, reading the comment from @euank solved my issue.

If you are generating the credentials on Mac use echo with -n option, ex: echo -n KEY|base64

This issue could be closed!

[mcfedr]

I had a similar issue to this, I had created the creds with a newline, but even after updated the secret it didn't solve the problem until i restarted the cert manager (i deleted the pod and it was recreated)

[munnerz]

Closing this as the issue seems to be alleviated by #391.