Terraform-provider-aws: No valid credential sources found for AWS Provider when using remote backend - Windows.

[mbailey-eh]

Description

When using the terraform cloud backend with AWS provider, terraform AWS provider fails to find AWS credentials/profile. Setting the profile name and shared credentials file in the provider or environment variable results in the same error. Removing the complete terraform block allow terraform plan to succeed. Configuration with backend passes terraform validate.

Verified that AWS CLI works correctly via random AWS commands such as "aws s3 ls". Also deleted AWS profile/credentials and reset up using "aws configure".

Terraform Version

C:\Users\username\Documents\GitHub\OPS-Terraform\AWS\Consumer\BU\Dev>terraform version
2019/10/21 11:24:14 [INFO] Terraform version: 0.12.12
2019/10/21 11:24:14 [INFO] Go runtime version: go1.12.9
2019/10/21 11:24:14 [INFO] CLI args: []string{"C:\Terraform\terraform.exe", "version"}
2019/10/21 11:24:14 [DEBUG] Attempting to open CLI config file: C:\Users\username\AppData\Roaming\terraform.rc
T2019/10/21 11:24:14 Loading CLI configuration from C:\Users\username\AppData\Roaming\terraform.rc
2019/10/21 11:24:14 [DEBUG] checking for credentials in "C:\Users\username\AppData\Roaming\terraform.d\plugins"
2019/10/21 11:24:14 [INFO] CLI command args: []string{"version"}
2019/10/21 11:24:14 [DEBUG] checking for provider in "."
2019/10/21 11:24:14 [DEBUG] checking for provider in "C:\Terraform"
e2019/10/21 11:24:14 [DEBUG] checking for provider in ".terraform\plugins\windows_amd64"
2019/10/21 11:24:14 [DEBUG] found provider "terraform-provider-aws_v2.33.0_x4.exe"
2019/10/21 11:24:14 [DEBUG] checking for provider in "C:\Users\username\AppData\Roaming\terraform.d\plugins"
2019/10/21 11:24:14 [DEBUG] found valid plugin: "aws", "2.33.0", "C:\Users\username\Documents\GitHub\OPS-Terraform\AWS\Consumer\BU\Dev\.terraform\plugins\windows_amd64\terraform-provider-aws_v2.33.0_x4.exe"
rraform v0.12.12

• provider.aws v2.33

Affected Resource(s)

provider.aws v2.33
terraform 0.12.9 -> 0.12.12

Terraform Configuration Files

Environment Variable Set

AWS_SDK_LOAD_CONFIG=1
AWS_PROFILE =default
AWS_SHARED_CREDENTIALS_FILE=C:\Users\username\.aws\credentials or C:/Users/username/.aws/credentials or %userprofile%.aws\credentials or %useprofile%/.aws/credentials or not set at all
TF_LOG=TRACE

Non-working configuration

terraform {
  backend "remote" {
    organization = "Org-name"
    hostname     = "app.terraform.io" ###also tried omitting like some documentation
    workspaces {
      name = "Workspace"
    }
  }
}
provider "aws" {
  region  = "us-east-1"
  profile = "default" ### Also attempted omitting
}

data "aws_caller_identity" "current" {}

output "account_id" {
  value = "${data.aws_caller_identity.current.account_id}"
}

output "caller_arn" {
  value = "${data.aws_caller_identity.current.arn}"
}

output "caller_user" {
  value = "${data.aws_caller_identity.current.user_id}"
}

working configuration

```hcl
provider "aws" {
region = "us-east-1"
profile = "default" ### Also attempted omitting
}

data "aws_caller_identity" "current" {}

output "account_id" {
value = "${data.aws_caller_identity.current.account_id}"
}

output "caller_arn" {
value = "${data.aws_caller_identity.current.arn}"
}

output "caller_user" {
value = "${data.aws_caller_identity.current.user_id}"
}
````

Debug Output

C:\Users\username\Documents\GitHub\OPS-Terraform\AWS\Consumer\BU\Dev>terraform plan
2019/10/21 11:57:36 [INFO] Terraform version: 0.12.12
2019/10/21 11:57:36 [INFO] Go runtime version: go1.12.9
2019/10/21 11:57:36 [INFO] CLI args: []string{"C:\Terraform\terraform.exe", "plan"}
2019/10/21 11:57:36 [DEBUG] Attempting to open CLI config file: C:\Users\username\AppData\Roaming\terraform.rc
2019/10/21 11:57:36 Loading CLI configuration from C:\Users\username\AppData\Roaming\terraform.rc
2019/10/21 11:57:36 [DEBUG] checking for credentials in "C:\Users\username\AppData\Roaming\terraform.d\plugins"
2019/10/21 11:57:36 [INFO] CLI command args: []string{"plan"}
2019/10/21 11:57:36 [TRACE] Meta.Backend: built configuration for "remote" backend with hash value 3926588858
2019/10/21 11:57:36 [TRACE] Preserving existing state lineage "8950802a-5353-11c8-d233-19183dd9b31a"
2019/10/21 11:57:36 [TRACE] Preserving existing state lineage "8950802a-5353-11c8-d233-19183dd9b31a"
2019/10/21 11:57:36 [TRACE] Meta.Backend: working directory was previously initialized for "remote" backend
2019/10/21 11:57:36 [TRACE] Meta.Backend: using already-initialized, unchanged "remote" backend configuration
2019/10/21 11:57:36 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2019/10/21 11:57:36 [TRACE] HTTP client GET request to https://app.terraform.io/.well-known/terraform.json
2019/10/21 11:57:36 [DEBUG] Retrieve version constraints for service tfe.v2.1 and product terraform
2019/10/21 11:57:36 [TRACE] HTTP client GET request to https://checkpoint-api.hashicorp.com/v1/versions/tfe.v2.1?product=terraform
2019/10/21 11:57:37 [TRACE] Meta.Backend: instantiated backend of type *remote.Remote
2019/10/21 11:57:37 [DEBUG] checking for provider in "."
2019/10/21 11:57:37 [DEBUG] checking for provider in "C:\Terraform"
2019/10/21 11:57:37 [DEBUG] checking for provider in ".terraform\plugins\windows_amd64"
2019/10/21 11:57:37 [DEBUG] found provider "terraform-provider-aws_v2.33.0_x4.exe"
2019/10/21 11:57:37 [DEBUG] checking for provider in "C:\Users\username\AppData\Roaming\terraform.d\plugins"
2019/10/21 11:57:37 [DEBUG] found valid plugin: "aws", "2.33.0", "C:\Users\username\Documents\GitHub\OPS-Terraform\AWS\Consumer\BU\Dev\.terraform\plugins\windows_amd64\terraform-provider-aws_v2.33.0_x4.exe"
2019/10/21 11:57:37 [DEBUG] checking for provisioner in "."
2019/10/21 11:57:37 [DEBUG] checking for provisioner in "C:\Terraform"
2019/10/21 11:57:37 [DEBUG] checking for provisioner in ".terraform\plugins\windows_amd64"
2019/10/21 11:57:37 [DEBUG] checking for provisioner in "C:\Users\username\AppData\Roaming\terraform.d\plugins"
2019/10/21 11:57:37 [TRACE] Meta.Backend: backend *remote.Remote supports operations
2019/10/21 11:57:41 [INFO] backend/remote: starting Plan operation
Running plan in the remote backend. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.

Preparing the remote plan...

To view this run in a browser, visit:
https://app.terraform.io/app/BUops/Consumer-BU-Dev/runs/run-rm3Us7cUfEn3f3rT

Waiting for the plan to start...

Terraform v0.12.9
Configuring remote state backend...
Initializing Terraform configuration...
2019/10/21 15:58:02 [DEBUG] Using modified User-Agent: Terraform/0.12.9 TFC/fc19dcc488
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

Error: No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider

on main.tf line 10, in provider "aws":
10: provider "aws" {

Expected Behavior

Output AWS account and user info as such
C:\Users\username\Documents\GitHub\OPS-Terraform\AWS\Consumer\BU\Dev>terraform apply
2019/10/21 11:29:39 Loading CLI configuration from C:\Users\username\AppData\Roaming\terraform.rc
data.aws_caller_identity.current: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

account_id = 123456789987
caller_arn = arn:aws:sts::123456789987:assumed-role/CrossAccountAccess/987654321123
caller_user = ARFFFFFFHHHHHHJJJJJJJ:15555555555555

Actual Behavior

Error: No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider

on main.tf line 10, in provider "aws":
10: provider "aws" {

Steps to Reproduce

Run terraform plan or terraform apply

• #0000

[alvarodelvalle]

I too am seeing this issue. Initiated a new tf cloud remote backend. My creds are set in ~/.aws/credentials. I've tried

1. export AWS_PROFILE=terraform-test && terraform plan with the aws provider section configured to accept the named profile and a default region
2. I've also tried exporting the aws environment variables
   export AWS_ACCESS_KEY_ID=$(aws configure get terraform-test.aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get terraform-test.aws_secret_access_key)
export AWS_DEFAULT_REGION=us-east-1

If I remove the remote configuration (switch to local state management), terraform plan works

UPDATE: I found out that if you have your tf cloud workspace set up for Local Execution Mode, it will use your local aws creds, if you have your cloud workspace set up for Remote Execution Mode, you will need to set up the aws credentials in your workspace's environment variables .../workspaces/your_workspace/variables.

[shanedroid]

I also hit this and @alvarodelvalle's suggestion of setting ENV VARS for the workspace in Terraform Cloud worked.

→ sw_vers && tf -v
ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G103
Terraform v0.12.13

terraform {
  required_version = ">= 0.12.0"

  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "org"

    workspaces {
      prefix = "test-"
    }
  }
}

provider "aws" {
  version                 = "~> 2.0"
  region                  = var.region
  allowed_account_ids     = var.allowed_account_ids
}

There is kind of a reference of this in the docs...?

Remote applies use the configuration code from the local working directory, but will use variable values from the specified workspace.

This makes more sense to me now that I encountered this myself.

[DenisBY]

Does it mean that I should store my AWS credentials at app.terraform.io? Why not to upload it temporary during plan execution?

[SudoBasher]

@DenisBY's comment seems inline with my thinking here too. is there a way to facilitate transmitting my local aws creds to terraform.io to perform remote backend state plan/apply commands, but not have to have them explicitly stored in the app.terraform.io platform?

[SudoBasher]

so after some more poking around, i found a radio button selection on the app.terraform.io site:

• go to your workspace
• go to settings
• change execution mode to local

then app.terraform.io will only be used for syncronizing states and state locking, and everything else about the terraform cli commands like the env vars will be sourced from your local env. sweet. this is working for me.

[DenisBY]

so after some more poking around, i found a radio button selection on the app.terraform.io site:

* go to your workspace

* go to settings

* change execution mode to local

then app.terraform.io will only be used for syncronizing states and state locking, and everything else about the terraform cli commands like the env vars will be sourced from your local env. sweet. this is working for me.

this is workaround, not a solution. time to time we need to deploy 10-20 new states and for every state we have to go to setting and change to local.

[ypadlyak]

There is also a problem with using an ENV variables to authenticate when using multiple provider profiles. Do you guys have any solution to this?

[DavidGereb]

I'm experiencing the same, init fails when remote backend is set, env vars didn't help, when removing the backend block, init succeeds

macOS Catalina Version 10.15.4 (19E287)
Terraform v0.12.26
+ provider.aws v2.66.0

[chrisj-au]

My issue is that I had no default profile configured. Having created a new windows login account recently I created a named profile using aws configure --profile so there was no 'default' profile. I could not do an init until after I created a default profile, after which time I could use and refer to my named profile in the aws provider.

[bflad]

Hi folks 👋 It looks like there are a few things going on in this issue, so I will try to cover them all.

If you are using Terraform Cloud or Terraform Enterprise with the remote backend, where your AWS credentials live depends on the execution mode for the workspace. In either case, these are sourced from where Terraform is physically executing the run.

• Local: Terraform calls to the AWS APIs are executed on your workstation/server, while storing the state remotely. AWS credentials will be sourced from your workstation/server.
• Remote: Terraform calls to the AWS APIs are executed on the Terraform Cloud/Enterprise workers. AWS credentials will be sourced from your Terraform Cloud/Enterprise workspace configuration (e.g. Terraform Cloud/Enterprise variables or Terraform Cloud/Enterprise environment variables)

If you are looking to manage Terraform Cloud/Enterprise (environment) variables en masse, there is the Terraform TFE Provider. There are also other solutions available for providing a more holistic solution for your Terraform run credentials, such as Vault, AWS Secrets Manager, etc.

If you are interesting in the remote backend and Terraform Cloud/Enterprise supporting uploading and using local credentials automatically during runs, this is a feature request that will need to be submitted through their respective support channels. Any implementation there is likely to be outside anything the Terraform AWS Provider codebase; the Terraform AWS Provider will automatically support the credentials once they are in the expected location.

For any other lingering reports here about default profile oddities, please do note that we are in the process of making some large improvements and bug fixes in the authentication handling for the Terraform AWS Provider, which will release in a few weeks with version 3.0.0.

Given that nothing here looks actionable within this codebase, I'm going to close this out. Thanks again for the report and hopefully the above clears up some things.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!