Terraform: Add "credential_source" variable to S3 backend

[vanniszsu]

Terraform Version

Terraform v0.11.7

Problem description

according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
I've setup instance profile and attached a role to this instance profile, and also associate my EC2 instance to this instance profile
the role has been attached all the necessary policies, for the trusted relationship of this role, we allow ec2 service to assume it.

in AWS configuration, we can use credential_source=Ec2InstanceMetadata under [default] profile to make use of ec2 auto assumed https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#using-aws-iam-roles

but in Terraform S3 backend Configuration variables , there are only role_arn,   profile,   shared_credentials_file , but missing credential_source

Request Terraform S3 backend to have credential_source variable to match all the AWS authentication options

Expected Behavior

Terraform can read the meta data of the EC2 instance use the temporary access key and token from http://169.254.169.254/latest/meta-data/iam/security-credentials/ to access the S3

Actual Behavior

No credential_source

have to setup aws configuration as below first before I can run the terraform init

[default]
output = json
region = cn-north-1
credential_source = Ec2InstanceMetadata

Steps to Reproduce

terraform init

Additional Context

[jbardin]

Hi @vanniszsu,

Thanks for filing the issue. This would need to be a feature request for the aws provider first, since we use their authentication code to ensure the behavior is kept sin sync.

Note that it's not necessary to put credential_source in your [default] profile, since the configuration has a profile option, and also honors the AWS_PROFILE environment variable.

I'll defer to the aws provider developers' opinion on adding another configuration field, when the standard mechanisms for providing credentials already support this.

[mootpt]

@jbardin running into similar issue with the following setup:

main.tf:

terraform {
  backend "s3" {
    bucket         = "lorem-example-prod-terraform"
    dynamodb_table = "lorem-example-prod-terraform"
    key            = "foo/terraform.tfstate"
    region         = "us-west-2"
    profile        = "prod"
  }
}

provider "aws" {
  region  = "us-west-2"
  profile = "prod"
}

~/.aws/config:

[profile staging]
role_arn=arn:aws:iam::999999999:role/tf.staging.role
credential_source=Ec2InstanceMetadata

[profile prod]
role_arn=arn:aws:iam::888888888:role/tf.prod.role
credential_source=Ec2InstanceMetadata

What's wrong here? I would expect Terraform to attempt to use the profile and retrieve the appropriate credentials from instance metadata.

I did find the following issue: https://github.com/terraform-providers/terraform-provider-aws/issues/5018, which seems to suggest the aws go sdk only recently added support for credential_source: https://github.com/aws/aws-sdk-go/pull/2201

Edit:
Reread your comment. Looks like we the work needs to be done on the aws provider

[bflad]

Version 1.41.0 of the AWS provider should support credential_source via an AWS configuration file (it may require AWS_SDK_LOAD_CONFIG=1 as well) since the AWS Go SDK version was updated beyond v1.15.54 prior to release. We can submit the dependency update upstream here so the S3 backend can utilize this enhancement too, but it will likely be part of the Terraform 0.12 releases at this point.

Outside the built-in support from an AWS configuration file, we'll need to decide if adding the ability to configure the provider and/or backend with an extra credential_source argument is valuable. If there are use cases that suggest this is beneficial, it would be great to hear about them.

[bflad]

Dependency update pull request submitted: #19190

[mootpt]

@bflad thanks a bunch! I appreciate the quick response. I will say that I tried in a variety of ways to get this working with no success.

My setup is a bit complicated, so I imagine I am missing something somewhere. Essentially, I have a terraform worker running in kubernetes that uses kube2iam to initially assume some role, let's call that roleA.

roleA has the ability to assume 3 roles across different AWS accounts, Role1, Role2, Role3. For my ~/.aws/config on the worker, Role1, Role2, Role3 are defined as profiles with the appropriate role_arn and a credential_source=Ec2InstanceMetadata.

Terraform configuration uses said profiles, so they can easily be reused for local plans and applies. Worth noting the above hasn't worked to date, so I am using a assume_role block for the aws provider and role_arn for the s3 backend.

I have attempted with the latest version of terraform and the aws terraform provider, playing with my test quite a bit. In one test I attempted just using the profile for the aws provider and role_arn for the backend, one where both the backend and provider use profiles, and one where I used profile for the backend and assume_role with the provider. All of these cases failed. Even attempted all with AWS_SDK_LOAD_CONFIG=1 per your suggestion. The only case that seems to work is when I use assume_role for the provider and role_arn for the backend, but this breaks the hybrid approach where engineers can use the same terraform config locally.

Looking at Terraform debug, it appears the provider is attempting to use roleA when using profile. Seems to not be respecting the profile specified:

2018-10-25T18:24:43.017Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] No assume_role block read from configuration
2018-10-25T18:24:43.017Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] Building AWS region structure
2018-10-25T18:24:43.017Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] Building AWS auth structure
2018-10-25T18:24:43.017Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] Setting AWS metadata API timeout to 100ms
2018-10-25T18:24:43.018Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] AWS EC2 instance detected via default metadata API endpoint, EC2RoleProvider added to the auth chain
2018-10-25T18:24:43.020Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] AWS Auth provider used: "EC2RoleProvider"
2018-10-25T18:24:43.020Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [INFO] Initializing DeviceFarm SDK connection
2018-10-25T18:24:43.020Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2018-10-25T18:24:43.020Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:43 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:

And the actual get callerid call from the provider:

2018-10-25T18:24:43.020Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: -----------------------------------------------------
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:44 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: ---[ RESPONSE ]--------------------------------------
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: HTTP/1.1 200 OK
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: Connection: close
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: Content-Length: 557
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: Content-Type: text/xml
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: Date: Thu, 25 Oct 2018 18:24:43 GMT
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: X-Amzn-Requestid: XXXXXXX
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: -----------------------------------------------------
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:44 [DEBUG] [aws-sdk-go] <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:   <GetCallerIdentityResult>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:     <Arn>arn:aws:sts::XXXXXXXX:assumed-role/ROLEA/7XXXXXX-ROLEA</Arn>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:     <UserId>REDACTED:7XXXXX-ROLEA</UserId>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:     <Account>XXXXXXXX</Account>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:   </GetCallerIdentityResult>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:   <ResponseMetadata>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:     <RequestId>XXXXXXXXXXXXXXXXXX</RequestId>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4:   </ResponseMetadata>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: </GetCallerIdentityResponse>
2018-10-25T18:24:44.180Z [DEBUG] plugin.terraform-provider-aws_v1.41.0_x4: 2018/10/25 18:24:44 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeAccountAttributes Details:

I am scratching my head. Sorry for the rant, just not sure others have been successful in a similar approach or if there are any tests in place that capture such a scenario.

[kipkoan]

@bflad - I'm running into the same problem @mootpt describes. I'm using Terraform v0.11.11 and provider.aws v1.57.0. I wasn't sure if you were saying earlier that this won't be fixed until v0.12? Can you confirm?

[lifeofguenter]

This is also still an issue with terraform 0.12.13. The ~/.aws/credentials works perfectly fine with the aws-cli but does not work with terraform

[jiashucheniress]

This is also still an issue with terraform 0.12.18. aws-cli works perfectly with our configuration, however, terrafrom is not able to assume the correct role