Atlantis: [Help Needed] AWS iam role to manipulate EKS cluster

[kennethtxytqw]

I am trying to get atlantis to manage our EKS cluster. Following the instructions here https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

I added the following code to the configmap under mapRoles

- rolearn: ${instance_role_arn}
  username: atlantis
  groups:
    - system:masters

I still get this error

Error: Unauthorized

Does anyone know of any solution?

[kennethtxytqw]

Fixed. I have no idea how but changing the username helped.

[andridzi]

hi @kennethtxytqw
could you please share what change did help?
thanks

[llamahunter]

I am experiencing a similar problem. I have added the atlantis task role arn to the EKS aws-auth configmap, but when the atlantis launched terraform task tries to operate on the EKS cluster, it fails:

Error: Unauthorized

  on .terraform/modules/prometheus_operator/modules/prometheus-operator/main.tf line 36, in resource "kubernetes_namespace" "this":
  36: resource "kubernetes_namespace" "this" {

And looking at the EKS authorization logs I see this:

time="2020-05-06T05:17:59Z" level=warning msg="access denied" client="127.0.0.1:55512" error="input token was not properly formatted: X-Amz-Date parameter is expired (15 minute expiration) 2020-05-06 01:09:00 +0000 UTC" method=POST path=/authenticate

It appears that atlantis, or terraform via atlantis, is trying to use a several hour old token to auth to EKS?

[rverma-jm]

@llamahunter did you found a solution for this?

[llamahunter]

Well, not really. The problem seems to be that the terraform plan caches the eks auth token, so that when you go to apply it later, the tokens are expired. We have to re-plan right before apply, and even then, it's possible that for complex terraform that there will be eks timeouts midway through the apply. We then need to re-plan and re-apply to finish applying the terraform.
See https://github.com/terraform-providers/terraform-provider-aws/issues/13189 and https://github.com/hashicorp/terraform/issues/24886

[kennethtxytqw]

I think @llamahunter is right. We (team at my workplace) have an internal rule that states if

1. the plan involves kubernetes resources, whether via helm or kubernetes provider
2. it is more than 5 minutes old

Always re-plan and apply.

[trallnag]

@kennethtxytqw, so performing a plan does recreate the token if the saved one has expired?

[llamahunter]

@kennethtxytqw, so performing a plan does recreate the token if the saved one has expired?

In my experience, yes. However, you can still run into problems if you have a LONG running operation and the token expires in the middle of it. You will need to re-plan and re-apply to pick up from where you left off.