Terraform: Add support for assuming role via AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables when running on EKS /w IRSA

[salekseev]

Terraform Version

Terraform v0.12.9

Terraform Configuration Files

terraform {
  # ... potentially other configuration ...

  backend "s3" {
    # ... other configuration ...
  }
}

Debug Output

Crash Output

Error: Failed to get existing workspaces: AccessDenied: Access Denied

status code: 403, request id: E649377A72E6FD52, host id: alDHq5WWI81ZAV67FDwAl/P9x5GKOd36FVZompUEUDHgM9C0hbbogV7Y98rJsP0zFNviwPYIYXY=

Expected Behavior

Assumed role via web identity token via the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables.

Actual Behavior

Used EC2 metadata service credentials.

Steps to Reproduce

1. terraform init when running inside EKS pod /w service account

Additional Context

References

• https://github.com/hashicorp/terraform/pull/22253 claims to have addressed it but it did not.
• https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
• https://github.com/hashicorp/aws-sdk-go-base/pull/5 potentially addresses this issue but new release tag had not yet been produced.

[mildwonkey]

PR #22992 does not fully fix this issue, reopening!

[lacatus]

+1 ! This would be great to be able to run Atlantis in EKS with serviceaccounts auth

[apenney]

Just a bump, I ran into this just now and stumbled over this issue after a bunch of searching. This currently doesn't work and stops me running atlantis in EKS, just like @lacatus

[charandas]

This need not stop anyone from actually using EKS/ISRA feature. For S3 access, one can temporarily provide an IAM user.

// ci-backend.tfvars for backend config, not sure if plan and apply need this, but init sure does
access_key = "access key here"
secret_key = "secret key here"

terraform init -backend-config=./test/ci-backend.tfvars

And the aws provider itself should work for plan, validate and apply so long as you prefix with AWS_SDK_LOAD_CONFIG=1 when you have the service account properly configured.

[burakovsky]

Adding skip_metadata_api_check = true to backend configuration solved this issue.

[dmitryax]

As a workaround you can get temporary old style permissions putting this before your terraform script:

$ aws sts assume-role-with-web-identity \
 --role-arn $AWS_ROLE_ARN \
 --role-session-name mh9test \
 --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
 --duration-seconds 1000 > /tmp/irp-cred.txt
$ export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")"
$ export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")"
$ export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")"
$ rm /tmp/irp-cred.txt

[abc11h]

Is that planned to be fixed in the next release?

[xanderflood]

https://github.com/aws/aws-sdk-go/issues/3101

I believe this can be resolved by bumping the aws-sdk-go from 1.25 to 1.29

[r-nasiri]

still happening in version v0.12.25

bash-4.2# ./terraform --version
Terraform v0.12.25
bash-4.2# ./terraform init

Initializing the backend...
Error refreshing state: AccessDenied: Access Denied
        status code: 403, request id: D22E52DFE348214C, host id: Gza/tgaOTAhyjoFZiRXagjlcojoQ5WzsvzEkI1OX3GVr51ORKl2q86nMdBuN7oSMchQKaiV52KY=

[bflad]

Multiple fixes for credential ordering, automatically using the AWS shared configuration file if present, and profile configuration handling of the S3 Backend have been merged and will release with version 0.13.0-beta2 of Terraform. In particular, this functionality is now specifically verified to work as expected, see also https://github.com/hashicorp/aws-sdk-go-base/issues/33.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.