Terraform-provider-aws: "No valid credential sources found for AWS Provider" scenario

[dmrzzz]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform version
Terraform v0.12.6
+ provider.aws v2.24.0

Affected Resource(s)

• provider.aws

Terraform Configuration Files

Very simple main.tf which just prints out VPC ids:

provider "aws" {
  # for historical note
  #version = "1.41"
  region  = "us-east-2"
}

data "aws_vpcs" "myvpcs" {
}

output "myvpcs" {
  value = "${data.aws_vpcs.myvpcs.ids}"
}

Description

I'm using [https://pypi.org/project/awscli-login/] to obtain temporary STS credentials (via assume_role_with_saml) and save them under separate profiles for different AWS accounts. The result looks like this:

~/.aws/config:

[profile foo]
region = us-east-2
output = json
[profile bar]
region = us-east-2
output = json

~/.aws/credentials:

[foo]
aws_access_key_id = AAA
aws_secret_access_key = BBB
aws_session_token = CCC
[bar]
aws_access_key_id = XXX
aws_secret_access_key = YYY
aws_session_token = ZZZ

and generally works fine with both awscli and Terraform (using AWS_PROFILE to specify which one I want to use).

However, I've discovered a scenario which causes Terraform to fail while awscli still works.

Steps to Reproduce

1. Demonstrate that both profiles' credentials are working. Since they belong to different accounts, we get different outputs:

$ export AWS_PROFILE=foo
$ terraform apply
data.aws_vpcs.myvpcs: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

myvpcs = [
  "vpc-5007f539",
  "vpc-94f389fd",
]

$ export AWS_PROFILE=bar
$ terraform apply
data.aws_vpcs.myvpcs: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

myvpcs = [
  "vpc-2b0e8843",
  "vpc-590fe130",
  "vpc-5c9f5d35",
  "vpc-647dfb0c",
  "vpc-a024d0c9",
  "vpc-ecff7184",
]

1. Unset the credentials for profile foo, leaving them blank in ~/.aws/credentials:

[foo]
aws_access_key_id = 
aws_secret_access_key = 
aws_session_token = 
[bar]
aws_access_key_id = XXX
aws_secret_access_key = YYY
aws_session_token = ZZZ

1. Demonstrate that Terraform can no longer use profile bar, but awscli still can.

$ export AWS_PROFILE=bar
$ terraform apply

Error: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

  on main.tf line 1, in provider "aws":
   1: provider "aws" {

$ aws ec2 describe-vpcs --output text --query 'Vpcs[*].[VpcId]'
vpc-5c9f5d35
vpc-647dfb0c
vpc-590fe130
vpc-2b0e8843
vpc-ecff7184
vpc-a024d0c9

Historical Note

This issue is present today in provider.aws v2.24.0, but seems to date back all the way to v1.42.

If I uncomment version = "1.41" (and use an older compatible terraform core), then profile bar works fine:

$ rm -rf .terraform/ terraform.tfstate*
$ terraform-0.11.8 init
$ terraform-0.11.8 version
Terraform v0.11.8
+ provider.aws v1.41.0
$ terraform-0.11.8 apply
data.aws_vpcs.myvpcs: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

myvpcs = [
    vpc-590fe130,
    vpc-a024d0c9,
    vpc-647dfb0c,
    vpc-5c9f5d35,
    vpc-2b0e8843,
    vpc-ecff7184
]

If I change it to version = "1.42", profile bar fails:

$ terraform-0.11.8 init
$ terraform-0.11.8 version
Terraform v0.11.8
+ provider.aws v1.42.0
$ terraform-0.11.8 apply

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

References

• possibly related to #6320

[bflad]

I'm able to confirm this issue, but it seems to only affect the credentials immediately after the errant missing right hand value in the AWS credentials file, e.g. given this ~/.aws/credentials

[foo]
aws_access_key_id = 
aws_secret_access_key = 
aws_session_token = 
[bar]
aws_access_key_id = XXX
aws_secret_access_key = YYY
aws_session_token = ZZZ
[baz]
aws_access_key_id = AAA
aws_secret_access_key = BBB
aws_session_token = CCC

Using bar fails while using baz works, interestingly enough.

The logic for reading the AWS credentials INI file is handled upstream in the AWS Go SDK and Terraform AWS Provider version 1.42.0 contained an upgrade past AWS Go SDK v1.15.59 where they switched from a third-party INI library to an internal implementation (https://github.com/terraform-providers/terraform-provider-aws/pull/6252, https://github.com/aws/aws-sdk-go/pull/2210).

I've filed an upstream AWS Go SDK issue here: https://github.com/aws/aws-sdk-go/issues/2800

In the meantime, there are a few fixes/workarounds available:

• Removing the errant INI section completely
• Adding right hand values in the errant INI section
• Adding a new line above the [bar] section header

[bixu]

I'm seeing similar behavior even though all keys are populated (I have 2 different AWS profiles in the config). In my case I can only get a working terraform plan setting AWS_PROFILE=<profile> terraform plan.

[ewbankkit]

@bflad The fix for https://github.com/aws/aws-sdk-go/issues/2800 was released in AWS SDK v1.25.4, merged via https://github.com/terraform-providers/terraform-provider-aws/commit/747ebb80404b174eb363b3ced000a1e73ca89c14.

[bflad]

Hi folks 👋

As mentioned above, the AWS Go SDK fix for this was merged and is included with version 2.31.0 of the Terraform AWS Provider, released earlier today.

Given the following ~/.aws/credentials file:

[buggy]
aws_access_key_id =
aws_secret_access_key =
[test]
aws_access_key_id = AKIA...
aws_secret_access_key = ...

The Terraform AWS Provider was able to successfully initialize with an example configuration:

$ export AWS_PROFILE=test
$ terraform plan
...
Plan: 1 to add, 0 to change, 0 to destroy.

For further bug reports or feature requests with the Terraform AWS Provider credentials initialization process, please create a new GitHub issue following one of the issue templates. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!