Terraform: S3 backend not working with credential_process + source_profile + role_arn combination

[reegnz]

Terraform Version

➜ terraform -v
Terraform v0.12.7

Terraform Configuration Files

A minimal example that breaks:

terraform {
  backend "s3" {
    bucket = "terraform-state.example.com"
    key    = "state.tf"
    region = "us-east-1"
  }
}

The AWS config file:

[profile test]
region = us-west-2
credential_process = cat ./cred.json

[profile testassume]
source_profile = test
role_arn = arn:aws:iam::222222222222:role/OrganizationAccountAccessRole

The AWS_PROFILE env variable is set to 'testassume'.

Debug Output

➜ TF_LOG=trace /usr/local/bin/terraform init
2019/08/30 12:54:48 [INFO] Terraform version: 0.12.7
2019/08/30 12:54:48 [INFO] Go runtime version: go1.12.9
2019/08/30 12:54:48 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init"}
2019/08/30 12:54:48 [DEBUG] Attempting to open CLI config file: /Users/reegnz/.terraformrc
2019/08/30 12:54:48 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/08/30 12:54:48 [INFO] CLI command args: []string{"init"}

Initializing the backend...
2019/08/30 12:54:48 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 1998351999
2019/08/30 12:54:48 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2019/08/30 12:54:48 [DEBUG] New state was assigned lineage "f6e6b5ac-7632-cba4-31b1-3f0b1c1e8781"
2019/08/30 12:54:48 [TRACE] Meta.Backend: moving from default local state only to "s3" backend
2019/08/30 12:54:48 [INFO] Setting AWS metadata API timeout to 100ms
2019/08/30 12:54:48 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id

Error: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

Crash Output

Expected Behavior

When using the credential_process in a profile and in another profile you use source_profile with role_arn to assume a role, terraform should connect properly with assumerole.

Actual Behavior

Terraform crashes with the above message.

Steps to Reproduce

1. configure aws with the above config file
2. export AWS_PROFILE=testassume
3. aws sts get-caller-identity should return the assumed role
4. terraform init fails

Additional Context

I ran into the same kind of error with aws-nuke and there upgrading the sdk seems to have solved the issue.
So maybe upgrading the SDK here as well will also solve the issue.

References

https://github.com/rebuy-de/aws-nuke/issues/397

22381

22377

21908

21120

[reegnz]

I have tried upgrading to the latest sdk to see if it's a quick fix or not:

go get -u github.com/aws/aws-sdk-go
make dev

I still get the error, so there's something deeper wrong in the s3 backend. :(

➜ TF_LOG=trace ~/go/bin/terraform init
2019/08/30 13:22:19 [INFO] Terraform version: 0.12.8 dev
2019/08/30 13:22:19 [INFO] Go runtime version: go1.12.9
2019/08/30 13:22:19 [INFO] CLI args: []string{"/Users/reegnz/go/bin/terraform", "init"}
2019/08/30 13:22:19 [DEBUG] Attempting to open CLI config file: /Users/reegnz/.terraformrc
2019/08/30 13:22:19 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/08/30 13:22:19 [INFO] CLI command args: []string{"init"}

Initializing the backend...
2019/08/30 13:22:19 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 3864323679
2019/08/30 13:22:19 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2019/08/30 13:22:19 [DEBUG] New state was assigned lineage "95e31768-fe4c-31e8-dbe6-fd66e64ab596"
2019/08/30 13:22:19 [TRACE] Meta.Backend: moving from default local state only to "s3" backend
2019/08/30 13:22:19 [INFO] Setting AWS metadata API timeout to 100ms
2019/08/30 13:22:19 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id

Error: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

[mmack-innio]

Same here.

[teamterraform]

Hi @reegnz,

The commands you've tried did not build Terraform with the latest sdk release. You would either need to vendor the updated dependency, or use go install directly with modules. However, I have a feeling that the latest sdk release is going to have the same issue.

While the AWS client code does seem to be reading the profiles from the credentials file, why the use of source_profile isn't working will take further investigation. This is probably best handled in the https://github.com/hashicorp/aws-sdk-go-base repo, and may even be something that should be wrapped into the https://github.com/hashicorp/aws-sdk-go-base/pull/5 PR.

[scalp42]

I'm still seeing the issue either way with 0.12.8, see #22377

[scalp42]

Just tested with 0.12.5 and can confirm it works again.

[grimm26]

@scalp42 @reegnz Do you have AWS_SDK_LOAD_CONFIG=1 set in your environment? If not, try it. I use an s3 backend with assumed roles and have no issue.

[scalp42]

I do @grimm26 see https://github.com/terraform-providers/terraform-provider-aws/issues/9999

[reegnz]

@teamterraform the SDK doesn't have an issue with that this use-case works with the latest SDK, see https://github.com/rebuy-de/aws-nuke/issues/397.
The issue to me seems to be with the s3 backend code itself:

1. The backend code uses an outdated SDK
2. Even with the latest SDK the backend code does some funky stuff instead of loading SDK configuration in a standard way.

@grimm26 the issue is not simply assuming a role, that works fine for me as well. The problem is when you try to run terraform with a profile that:

1. relies on a source_profile that makes use of a credential_process
2. assumes a role with role_arn

It's kind-of an edge case. I found this scenario by combining these two use-cases of aws-vault:

• https://github.com/99designs/aws-vault/blob/master/USAGE.md#using-credential-helper
• https://github.com/99designs/aws-vault/blob/master/USAGE.md#example-awsconfig

The CLI works perfectly with such a configuration, also with another tool that relies on the SDK (aws-nuke) a plain SDK upgrade fixed the issue. So my only guess is that the SDK isn't used the way it's meant to be used in the S3 backend code.

Sometime in the next month I'll dive into the code of all three projects (SDK, aws-nuke, terraform) to see how the SDK API should be used compared to how it is used now. If somebody else can have a look before then I'd appreciate their effort.

[lorengordon]

Can you build terraform and the aws provider with this patch and see if it works for you, https://github.com/hashicorp/aws-sdk-go-base/pull/5?

[reegnz]

Sure, I'll see if I can get around to it next week.

[mmack-innio]

any update on this?

[scalp42]

I believe it's related to https://github.com/hashicorp/aws-sdk-go-base/pull/5 but Terraform needs to be updated to pull in the changes.

[reegnz]

I see that that PR was merged and released as 0.4.0, and also merged into terraform now in the context of https://github.com/hashicorp/terraform/issues/22992.
https://github.com/hashicorp/terraform/commit/6656104f8fb4da54bc5b4df8849691357be72e95

It got released with 0.12.11, so I'll do a quick check if this works now.

[reegnz]

Works!

Things to keep in mind:

You need to instruct the SDK explicitly to load the config file, and you also need to specify the region and profile in env variables to be able to use the above config in the issue description:

export AWS_SDK_LOAD_CONFIG=1
export AWS_REGION=us-east-1
export AWS_PROFILE=testassume

Then:

terraform init
terraform apply

[mhvelplund]

Tested.
Works in 0.12.14
Broken in < 0.12.10

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.