Cert-manager: Support AWS session tokens

[ChienHuey]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Is your feature request related to a problem? Please describe.:
The applications in our cluster use Route53 zones that are hosted in multiple AWS accounts. While we can use AWS static access keys with Issuers to handle this issue, we prefer to use temporary keys provided by AWS STS. Ideally we would use AWS STS temporary access keys and write automation to rotate them as they expire.

Describe the solution you'd like:
Support temporary access keys issued by AWS STS consisting of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.

Describe alternatives you've considered:
We attempted to use kube2iam. We wrote a sidecar to rotate through kube2iam roles on the cert-manager pod annotation. But that solution leads to delays in the certificate being issued because of the wait as the roles are rotated.

Anything else we need to know?:

/kind feature

[munnerz]

This would be great to support better 😄

I don't use AWS day-to-day, but if anyone is able to pick this up that'd be great 😄

• Does this require additional/alternative configure beyond what we already expose on the Issuer resource?
  
  
  
  • If so, how do you think we should best represent this?
  
  
  • Are there other auth methods we may want to support in future?
  
  

[ChienHuey]

The quick way to implement is to add the session token field (as optional) to the Issuer resource route53 config. I already have a pod that can reacquire new temporary credentials and swap them out in k8s secrets to have this work.

The better solution is to add a solution that will use ambient credentials to acquire the temporary credentials, use them and renew them when the temporary credentials expire. This would be more involved to implement.

[munnerz]

The quick way to implement is to add the session token field (as optional) to the Issuer resource route53 config

We currently do not re-use the route53 client between calls to the AWS API - would this be problematic for rotating?

Would you be able to provide a config example for what this might look like? I'm unsure whether this is a property that applies only to the current token based auth, or if it applies to more auth types we may support in future (I'm also not certain of the variety of different types of AWS auth available, so it'd be great if we can enumerate those so we have an idea of what we're working towards).

The better solution is to add a solution that will use ambient credentials

We do currently supporting using 'ambient credentials' (i.e. metadata service) to acquire credentials - is this affected by the session token handling too?

[ChienHuey]

Example config with session tokens:

spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-fido
    dns01:
      providers:
      - name: route53
        route53:
          region: us-east-1
          accessKeyID: MYDOGLOVESBONES
          secretAccessKeySecretRef:
            name: my-dog-secret-access-key
            key: secret-access-key
          sessionToken: kjfdslkdfskfjslkdfjsalkdfjslkfjlsadjgflkgjlkfsdjflasdjfdjaslfjaslkdfjlasdkfj

The current setup with the AWS access key ID and AWS secret access key is one way of using AWS IAM user credentials. Downside with this method is that the keys are static and should be rotated for secret. Another way is to obtain AWS temporary credentials which include the same two keys and adds a third which is a session token. These credentials are valid for a maximum of 1 hour so after that point, need credentials would have to be obtained. We already have a utility to obtain these credentials as a Kubernetes cronjob. Slight modification would be required to update the Issuer.

Perhaps my company is unique. Many of our developer teams have their own accounts (so we have dozens) where they host their own zone's Route 53 records. Ambient credentials don't work for us because kube2iam and cert-manager only support one role at any given time - so we can only update one account's Route53 for dns01 at a time. We tried rotating the kube2iam annotation but that makes our teams wait for certificate provisioning on new applications which is less than ideal.

[whereisaaron]

Hi @ChienHuey doesn't that just shift the credentials issue to your CronJob? Or that CronJob using a kube2iam role to obtain the other temporary credentials?

Would it be better if in cert-manager we could name a role ARN in the Issuer/ClusterIssuer, and cert-manager could use its ambient credentials assigned by kub2iam to the assume the role named in the Issuer/ClusterIssurer? Then the CronJob would be unnecessary.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[awwithro]

/remove-lifecycle rotten
I have a similar use case as @ChienHuey multiple DNS zones in multiple accounts.
I agree with @whereisaaron that ideally a role arn would be specified instead of the access/secret key fields and certmanager would attempt to assume the role before calling into route53. A tool like kiam for kube2iam is probably in use for users of k8s on aws

[moolen]

I have the same use-case as @awwithro and @ChienHuey.

I agree specifying a role would be the best option. It should be compatible with kiam, kube2iam and using the accessKeyID / secretAccessKeySecretRefopts.

kind: ClusterIssuer # this should work for namspaced Issuer, too
metadata:
  name: letsencrypt-prod
spec:
  acme:
    dns01:
      providers:
      - name: xxxx-account
        route53:
          region: eu-central-1
          hostedZoneID: xxxx-account-zone-id
          role: arn:aws:iam::xxxxxxxxxxxx:role/my-role# <---- add this
      - name: yyyy-account
        route53:
          region: eu-central-1
          hostedZoneID: yyyy-account-zone-id
          role: arn:aws:iam::yyyyyyyyyyyy/my-role# <---- add this
# [...]

I'm working on a prototype of the above proposal.

@ChienHuey would a role-based access model work for you, too?