Terraform-provider-aws: Terraform S3 Remote State using only credential profiles (no default credentials)

[hashibot]

_This issue was originally opened by @mikereinhold as hashicorp/terraform#8911. It was migrated here as part of the provider split. The original body of the issue is below._

---

Terraform remote state S3 backend does not properly retrieve remote state using the profile config option alone if there is no default AWS credentials.

Example, consider a multiple account configuration using assumed administrative roles across accounts. I may not want to have a [default] profile configured in ~/.aws/credentials, set in Terraform, or in environment variables. For example, maybe because I want to prevent accidentally creating resources in the account that has the IAM account when the correct provider is not specified for each resource.

If I only specify the S3 config profile, the data source fails to find AWS credentials. If I add the provider line, terraform complains that the provider does not support terraform_remote_state.

Terraform Version

Terraform v0.7.3

Affected Resource(s)

• data source terraform_remote_state
• provider/aws
  

  Terraform Configuration Files

data "terraform_remote_state" "shared-state"{
    provider = "aws.admin"
    backend = "s3"
    config {
        bucket = "${var.state-bucket}"
        key = "${var.group}/${var.project}/${var.environment}.tfstate"
        region = "${var.state-region}"
        profile = "admin"
    }
}

Expected Behavior

Terraform should use the specified provider for the Terraform remote state data source.

Alternatively, in order to avoid needing to specify the provider at all, the data source should consult the specific config (in this case S3 config) in order to determine the correct provider. This would probably be preferable for DRY.

Actual Behavior

If no provider is specified, just using the S3 config profile:

• No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

If the provider is specified (with or without the S3 config profile):

• data.terraform_remote_state.devyse-domains: Provider doesn't support data source: terraform_remote_state

Steps to Reproduce

Configure a terraform_remote_state as described above, with an S3 config profile (with or without the provider line, pending consideration for which approach is "correct").
Ensure an AWS provider using alias admin exists, referencing an AWS Config profile admin. No default AWS credentials should be found (no env vars, no default AWS config profile / credentials, no default Terraform provider that supplies the creds.

1. terraform plan

[ascendantlogic]

I am seeing this behavior in version 0.9.8 so it is still present.

[ncs-msmith]

Still a problem as of 0.10.5

[sysadmiral]

I updated to 0.10.6 and immediately hit this issue on our build/deploy server and my local laptop.

Steps to reproduce:

• delete default credentials in ~/.aws/credentials
• set profile in s3 backend configuration
• checkout a repo of terraform code that is known to work
• run terraform init

[in4mer]

This currently exists in v0.11.1, which is latest as of this writing.

[JackNeto]

I'm having the same problem in v0.11.2

[ghost]

Same problem in 0.10.8, too.

It actually beggars belief that this bug has been known to exist since at least Sept of 2016 and it is still not fixed. Does hashicorp just simply not care that people cannot actually use the tool that they are building? This bug precludes any kind of cross-account access for anything. 3rd party oauth or saml authentication all use assumed roles instead of IAM users and also require temporary credentials with a token, or any other access that doesn’t depend on IAM users directly. This makes it impossible to use terraform with a team of people, since the only way to do so, reasonably, would be to share an access key amongst many users if they are otherwise unwilling or unable to transition to using IAM users for everything.

And really, how hard can this be to fix? All of amazon’s APIs and cli tools honor a specified profile name. All hashicorp has to do is fix the provider so that it specifies it correctly when calling API methods or running aws commands. And what's the point of allowing a profile name to be specified if it isn't going to honor it? Shouldn't the most basic of unit testing be able to validate that that is not working correctly?

You'd think there'd at least be some kind of workaround. Or does everyone doing any kind of real user management around AWS or cross-account configuration just abandon terraform at this point and find a tool that actually works?

I've tried multiple providers, each with a profile, and specifying the provider instance in the terraform_remote_state data items. That didn't work. I can set the AWS_PROFILE environment variable and it honors that, but that forces me to then have terraform_remote_state and all resources in the same AWS account, since cross-account access would require different aws providers to use different profiles, and that doesn't work.

[ghost]

OK, I don't know a thing about Go, but I took a few minutes to grep through the source code and I'm hopeful that I've figured out the core problem, and it doesn't look to be a particularly difficult fix.

In terraform-core, there is backend/remote-state/s3/backend.go, there is a function called configure() which configures an AWS provider for use by the terraform provider - it does not use the provider declared in the template. It configures one based on the values in the config block of terraform_remote_state.

But that aws provider is VERY completely specified, with lots of default values for things that are not usually specified at all when declaring a provider in a template.

My template providers look like this:

        provider "aws" {
                profile = "terraform"
                region = "${var.some_var}"
        }

But the provider that is configured by the terraform provider in code looks like this:
https://github.com/hashicorp/terraform/blob/fb6b349e580b1cbb9053c09f97f8424d63d0716f/backend/remote-state/s3/backend.go

        cfg := &terraformAWS.Config{
        AccessKey:               data.Get("access_key").(string),
        AssumeRoleARN:           data.Get("role_arn").(string),
        AssumeRoleExternalID:    data.Get("external_id").(string),
        AssumeRolePolicy:        data.Get("assume_role_policy").(string),
        AssumeRoleSessionName:   data.Get("session_name").(string),
        CredsFilename:           data.Get("shared_credentials_file").(string),
        Profile:                 data.Get("profile").(string),
        Region:                  data.Get("region").(string),
        S3Endpoint:              data.Get("endpoint").(string),
        SecretKey:               data.Get("secret_key").(string),
        Token:                   data.Get("token").(string),
        SkipCredsValidation:     data.Get("skip_credentials_validation").(bool),
        SkipGetEC2Platforms:     data.Get("skip_get_ec2_platforms").(bool),
        SkipRegionValidation:    data.Get("skip_region_validation").(bool),
        SkipRequestingAccountId: data.Get("skip_requesting_account_id").(bool),
        SkipMetadataApiCheck:    data.Get("skip_metadata_api_check").(bool),
        S3ForcePathStyle:        data.Get("force_path_style").(bool),
    }

Note that it gets a default value for token, secret_key, and access_key, among other things, which would never be configured in a template if the author is expecting to use a shared credentials file. That was the clue that led me to find where that config struct is turned into credentials for talking to amazon. That appears to be in terraform-provider-aws aws/auth_helpers.go, where an array of credentials providers is created in the following code:
https://github.com/terraform-providers/terraform-provider-aws/blob/e3959651092864925045a6044961a73137095798/aws/auth_helpers.go

        providers := []awsCredentials.Provider{
        &awsCredentials.StaticProvider{Value: awsCredentials.Value{
            AccessKeyID:     c.AccessKey,
            SecretAccessKey: c.SecretKey,
            SessionToken:    c.Token,
        }},
        &awsCredentials.EnvProvider{},
        &awsCredentials.SharedCredentialsProvider{
            Filename: c.CredsFilename,
            Profile:  c.Profile,
        },
    }

Which is eventually followed by this code:

        // This is the "normal" flow (i.e. not assuming a role)
    if c.AssumeRoleARN == "" {
        return awsCredentials.NewChainCredentials(providers), nil
    }

I didn't dig any farther than that because my suspicion is that the first awsCredentials.provider, which takes access_key, secret_key, and token, is returning a credential which is, for all intents and purposes, empty rather than non-existent, so the other two awsCredentials.providers in that array never get accessed. Whereas wth a less fully-specified provider, the first awsCredentials.provider in the array will be correctly detected to be useless and it will fall through to the 3rd one, which looks up creds in a file, applying the profile name if specified. We know that functionality works, because normal aws resources that use an aws provider in the template do honor the profile name. So my guess is that the code needs to be modified to correctly deduce that the first awsCredentials.provider is invalid when the params are empty strings rather than null (or whatever the differentiation is in Go).

I'll attempt to figure out how to build the code from scratch tomorrow and put a conditional around the chain of credentials providers in order to test my hypothesis. I'm hopeful that someone with more understanding of Go and the terraform codebase can use my investigation to make a very quick patch to fix this glaring bug in the codebase.

And while I'm at it - the codebase issues the EXACT SAME error message from at least 3 separate locations related to authenticating with AWS so that it is impossible to determine which one of those 3 failure cases is the one that actually fired. 1 of the locations isn't even in the same file or the same git repo, yet it uses identical language. That's not exactly aiding in debugging, which is surely what error messages are actually for, no?

The message in question is the following: No valid credential sources found for AWS Provider. Please see https://terraform.io/docs/providers/aws/index.html for more information on providing credentials for the AWS Provider

[reza]

I reached to the same piece of code for another reason when aws provider profile settings where ignored and default profile was referenced. I deleted default profile of my aws credentials to debug so I also get this error message:

* provider.aws: No valid credential sources found for AWS Provider. Please see https://terraform.io/docs/providers/aws/index.html for more information on providing credentials for the AWS Provider

The problem is that terraform is sending an empty string as profile to aws-sdk-go, whatever the reason it happens at this section of the code where the config is passed to the GetCredentials and default profile is returned:

https://github.com/terraform-providers/terraform-provider-aws/blob/4a8e67ad477f19769447ee60a426d93269e4a98c/aws/config.go#L255

    log.Println("[INFO] Building AWS auth structure")
    creds, err := GetCredentials(c)
    if err != nil {
        return nil, err
    }

At the aws-sdk-go side, it receives an empty profile and switches to default profile unless AWS_PROFILE is already defined (that's why if you set that env var terraform will work):

https://github.com/aws/aws-sdk-go/blob/master/aws/credentials/shared_credentials_provider.go#L141

func (p *SharedCredentialsProvider) profile() string {
    if p.Profile == "" {
        p.Profile = os.Getenv("AWS_PROFILE")
    }
    if p.Profile == "" {
        p.Profile = "default"
    }

    return p.Profile
}

[danbudris]

This is still a problem as of 0.11.7 as far as I can tell. If I have a default credential specified in the credentials file, the backend uses it no matter what -- confirmed through traces. If I set the default credential to the credential I wish to use, it works right. No matter what I define in the remote state block it does not behave as both expected and advertised.

[ghost]

Clearly, it isn't going to get fixed until one of us submits a pull request. I'm not quite sure how, but apparently no one at hashicorp thinks fully supporting authentication is important. It's coming up on 2 years that this has been a known issue. My templates and modules are finally compatible with recent builds of terraform, so maybe I'll try fixing this.

[Laxman-SM]

this is Still problem with terraform 0.11.7

data "terraform_remote_state" "state" {
provider = "aws.some region"
backend = "s3"
config = {
//region = "us-west-2"
region = "${data.aws_region.current.name}"
//provider = "aws.oragon"
buknet = "qa-qa-dockerswarm"
key = "laxman1/terraform.tfstate"
//skip_region_validation = "true"
profile = "default"
encrypt = true
dynamodb_table = "tfstates1-swarm"
kms_key_id = "arn:aws:kms:us-west-2:xxxxxxxxxx:key/c4ba3saxz-efbb-430d-b9f3-b3602bc3a14a"
//workspace = "${terraform.workspace}"
}
}

terraform plan showering error

/usr/local/bin/terraform plan -out terraform.out
Acquiring state lock. This may take a few moments...
var.environment
Enter a value: qa

var.project
Enter a value: qa

Releasing state lock. This may take a few moments...

Error: data.terraform_remote_state.state: Provider doesn't support data source: terraform_remote_state

Process finished with exit code 1,

[ghost]

It occurs to me that perhps one of the reasons this bug has languished unfixed for so long is because it is actually in the wrong place. terraform_remote_state is part of the terraform provider, not the aws provider. The buggy code is in the terraform provider, where it instantiates its own connection to AWS rather than using the aws provider already declared or the default aws provider. Probably the correct fix isn't so much to get the terraform provider to specify the profile and region correctly, it is to get the terraform provider to utilize an already existing AWS provider for its connection, instead, if that is even possible. I'm not sure if provider to provider dependencies can be resolved. We would also need the ability to specify the alias of the AWS provider in the terraform provider declaration or else in the terraform provider resources, like terraform_remote_state.

Regardless, the bug should definitely be moved from terraform-provider-aws to https://github.com/terraform-providers/terraform-provider-terraform. I'm going to create an issue in there and see if I can't get someone to mark this as a duplicate and then actually assign someone to the fix.

[t3hmrman]

For anyone running into this, note that you must include profile in your (mentioning it twice). For example:

provider.tf:

provider "aws" {
  profile = "your_project_staging"
}

terraform {
  backend "s3" {
    bucket = "your-project-staging"
    key    = "staging.tfstate"
    region = "ap-northeast-1"
    profile = "your_project_staging"
  }
}

Prefer underscores (_) to hypens (-) for AWS profile names.

[edtan]

This may be related to hashicorp/terraform#16710 (in particular, this comment).

I ran into this issue on v0.12.12. Initially, I created a resource several months ago using my default AWS profile - I did not use static credentials, environment variables, or specify the shared_credentials_file or profile attributes, so it used the default profile in my .aws/config and .aws/credentials files. Recently, I changed the AWS profile name from default to test, and this issue started happening.

Upon inspecting .terraform/terraform.tfstate, backend.config does not have shared_credentials_file or profile defined, so it looks like this is where the problem is happening. After manually inserting the correct profile into terraform.tfstate, terraform finds my correct profile and starts working again.

Update: I think the issue that I'm running into is that terraform is trying to read the existing remote state to compare it with the new backend configuration. Doing a terraform init -reconfigure seems to also have worked for me.

[bflad]

Hi folks 👋 The original issue regarding the Terraform S3 Backend (and theoretically the terraform_remote_state data source) not using the profile argument correctly was fixed in Terraform CLI v0.13.0-beta2. Please do note that the Terraform S3 Backend uses similar authentication mechanisms for consistency, but _separate_ configuration from the Terraform AWS Provider as mentioned above.

Any further bug reports or feature requests with the Terraform S3 Backend or terraform_remote_state data source will need to filed in the hashicorp/terraform repository for further triage.

While the profile bug was not present in the Terraform AWS Provider, we do intend to make similar other authentication updates here as part of our version 3.0.0 work in the next few weeks. If there are lingering Terraform AWS Provider authentication issues after the version 3.0.0 release, please file a new bug report in this repository and we will take a fresh look. Thanks.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!