Terraform-provider-aws: RFC Supporting credential_process for assuming roles using external credential providers

[daxroc]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

As part of our workflow, we use third-party tooling to assume roles within multiple accounts.
My specific use case is with Okta - https://github.com/oktadeveloper/okta-aws-cli-assume-role

There are probably more than two ways to implement this

Defer to AWS - credential_process for a profile

With this configuration in ~/.aws/config no credentials file exists - as credentials are fetched during execution

[profile TestAccount_A]
credential_process = okta-credential_process arn:aws:iam::1234567890:role/terraform-role
region = us-east-1

[profile TestAccount_B]
credential_process = okta-credential_process arn:aws:iam::987654321:role/terraform-state-role
region = us-west-1

This would lead to no configuration change in the provider definitions if transparent

Support credential_process internally

This would require enhancing the default methods of credential lookup to include

• Static credentials
• Environment variables
• Shared credentials file
• EC2 Role
• AWS Config - credential_process (NEW)

See below for the provider configuration example

New or Affected Resource(s)

• aws_provider
• terraform backend

Potential Terraform Configuration

This could have two modes of operation

Explicit - configuration within the provider & terraform backends

provider "aws" {
  ...
  credential_process = "okta-credential_process arn:aws:iam::1234567890:role/terraform-role"
}

terraform {
    backend "s3" {
        bucket = "tf-state"
        ...
        credential_process = "okta-credential_process arn:aws:iam::9876543210:role/terraform-state-role"
    }
}

Implicit - through the aws ~/.aws/config profiles with no change to the provider definitions

provider "aws" {
  ...
  profile = "TestAccount_A"
}

terraform {
    backend "s3" {
        bucket = "tf-state"
        ...
        profile = "TestAccount_B"
    }
}

References

• https://github.com/oktadeveloper/okta-aws-cli-assume-role
• https://docs.aws.amazon.com/cli/latest/topic/config-vars.html

[bflad]

Hi @daxroc 👋 Quick question -- does the implicit support for this already work with version 1.52.0 of the AWS provider? It may require setting the environment variable AWS_SDK_LOAD_CONFIG=true as well. Upstream support in the AWS Go SDK was added via https://github.com/aws/aws-sdk-go/pull/2217 and version 1.52.0 of the AWS provider should include that update.

[daxroc]

Hi @bflad, Initial testing with the AWS_SDK_LOAD_CONFIG=true didn't work. Will dig in when I get some free time.

[YakDriver]

I've done some digging. Here's what I've found:

Credential files

~/.aws/credentials (All tests use the same file):

[default]
credential_process = cat ~/.aws/credproc.json

[credproc]
credential_process = cat ~/.aws/credproc.json

[giltig]
aws_secret_key_id = <valid secret key id>
aws_secret_access_key = <valid secret access key>

~/.aws/credproc.json (The credential_process expects a JSON, which could come from Okta. I'm just catting a local file for simplicity):

{
  "Version": 1,
  "AccessKeyId": "<valid secret key id>",
  "SecretAccessKey": "<valid secret access key>"
}

1. Default profile, local credential_process works ✅

This HCL and default profile using credential_process works since v1.52.

provider "aws" {
  version = "<= 1.52"
  region  = "us-east-1"
}

resource "aws_s3_bucket_object" "object1" {
  bucket  = "yak-forsok"
  key     = "arch/three_gossips/turret"
  content = "Delicate"
}

2. Non-default profile, local credential_process works ✅

This HCL and non-default profile using credential_process works since v1.52.

provider "aws" {
  version = "<= 1.52"
  region  = "us-east-1"
  profile = "credproc"
}

resource "aws_s3_bucket_object" "object1" {
  bucket  = "yak-forsok"
  key     = "arch/three_gossips/turret"
  content = "Delicate"
}

3. S3 backend credential_process does not work ❌

This HCL with default profile (which is credential_process) doesn't work.

terraform {
  backend "s3" {
    bucket = "yak-forsok"
    key    = "forsok-1.tfstate"
    region = "us-east-1"
  }
}

provider "aws" {
  version = "<= 1.52"
  region  = "us-east-1"
}

resource "aws_s3_bucket_object" "object1" {
  bucket  = "yak-forsok"
  key     = "arch/three_gossips/turret"
  content = "Delicate"
}

4. Normal creds S3 backend with local credential_process does work ✅

This HCL shows valid, normal credentials in S3 backend working with credential_process AWS provider.

terraform {
  backend "s3" {
    bucket = "yak-forsok"
    key    = "forsok-1.tfstate"
    region = "us-east-1"
    profile = "giltig"
  }
}

provider "aws" {
  version = "<= 1.52"
  region  = "us-east-1"
  profile = "credproc"
}

resource "aws_s3_bucket_object" "object1" {
  bucket  = "yak-forsok"
  key     = "arch/three_gossips/turret"
  content = "Delicate"
}

5. terraform_remote_state data source does not work with credential_process ❌

Assuming a remote state already exists, the terraform_remote_state data source doesn't work with credential_process.

provider "aws" {
  version = "<= 1.52"
  region  = "us-east-1"
  profile = "credproc"
}

data "terraform_remote_state" "objects_bucket" {
  backend = "s3"

  config {
    bucket  = "yak-forsok"
    key     = "forsok-1.tfstate"
    region  = "us-east-1"
    profile = "credproc"
  }
}

6. terraform_remote_state with local credential_process does work ✅

Assuming a remote state already exists, the terraform_remote_state works using normal credentials even if the local AWS provider uses credential_process.

provider "aws" {
  version = "<= 1.52"
  region  = "us-east-1"
  profile = "credproc"
}

data "terraform_remote_state" "objects_bucket" {
  backend = "s3"

  config {
    bucket  = "yak-forsok"
    key     = "forsok-1.tfstate"
    region  = "us-east-1"
    profile = "giltig"
  }
}

Conclusion

Using a credential_process in the config of the terraform_remote_state data source will not work. Also, using a credential_process with a Terraform S3 backend will not work.

However, using a credential_process with the AWS provider itself will work.

[YakDriver]

I built the latest Terraform v0.12-dev (v0.12@b217624d8) core and the AWS provider, and the credential_process works in terraform_remote_state and s3 backend situations.

The credential_process became available in AWS SDK Go v1.16.0. The issue is simply that the Terraform core is currently vendored/pinned to AWS SDK Go v1.14.31 and since terraform_remote_state and the s3 backend use core rather than the AWS provider for credentials, credential_process is not available. When a Terraform core release uses v1.16+, all will be fine.

[llamahunter]

Note that this appears to have been addressed for v0.11 in https://github.com/hashicorp/terraform/commit/102c78c343998bb27ddb01b2f24c12f675cbfc4e

There hasn't been a v0.11 release since then, tho. Whenever v0.11.14 comes out, it should have credential_process support.

[lorengordon]

One thing that would be pretty awesome about a credential_process argument to the provider would be the ability to interpolate vars in the command. Would make it easier/possible to get creds that way for different accounts and roles directly from a federated identity provider (without requiring assume role).

[pikeas]

With TF 0.11.14:

Error configuring the backend "s3": No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

credential_process works generally (for example, with AWS_PROFILE=foo aws sts get-caller-identity), so it looks like the s3 backend does not yet support this feature.

[lorengordon]

@pikeas The s3 backend in tf v0.11.14 does support profiles that use credential_process. There are some configurations that do not work though, such as using credential_process in the profile, and the assume_role block in the backend or provider config. A PR has been opened to address that though...

[pikeas]

@lorengordon I am not using assume_role in the backend or provider.

terraform {
  required_version = ">= 0.11.14"

  backend "s3" {
    region         = "..."
    bucket         = "..."
    key            = "..."
    dynamodb_table = "..."
    encrypt        = false
  }
}

provider "aws" {
  version = "~> 2.17"
  region  = "..."
}

[profile foo]
region = ...
mfa_serial = arn:aws:iam::12345:mfa/pikeas
credential_process = aws-vault exec -j foo

$ aws-vault exec -j foo # works as expected
{"Version":1,"AccessKeyId":"...","SecretAccessKey":"...","SessionToken":"...","Expiration":"..."}

$ AWS_PROFILE=foo TF_LOG=debug terraform plan
2019/07/03 16:08:53 [INFO] Terraform version: 0.11.14
2019/07/03 16:08:53 [INFO] Go runtime version: go1.12.4
2019/07/03 16:08:53 [INFO] CLI args: []string{"/usr/local/Cellar/tfenv/0.6.0/versions/0.11.14/terraform", "plan"}
2019/07/03 16:08:53 [DEBUG] Attempting to open CLI config file: /Users/pikeas/.terraformrc
2019/07/03 16:08:53 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/07/03 16:08:53 [INFO] CLI command args: []string{"plan"}
2019/07/03 16:08:53 [INFO] Setting AWS metadata API timeout to 100ms
2019/07/03 16:08:53 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2019/07/03 16:08:58 [DEBUG] plugin: waiting for all plugin processes to complete...
Failed to load backend:
Error configuring the backend "s3": No valid credential sources found for AWS Provider.
    Please see https://terraform.io/docs/providers/aws/index.html for more information on
    providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing
the values in your configuration, run "terraform init"

[lorengordon]

@pikeas I get that the config posted is obfuscated, so maybe this is a cut/paste/edit type of error, but I do see in your config that your credential_process is associated with the profile foo and that nowhere in your backend or provider do you specify the profile argument. So, if that's _not_ a posting error, maybe try adding profile = "foo" to your configs.

[pikeas]

@lorengordon Your comment helped me solve this!

For anyone else having trouble with this, https://www.terraform.io/docs/backends/types/s3.html#configuration-variables says the following:

profile - (Optional) This is the AWS profile name as set in the shared credentials file. It can also be sourced from the AWS_PROFILE environment variable if AWS_SDK_LOAD_CONFIG is set to a truthy value, e.g. AWS_SDK_LOAD_CONFIG=

Adding profile to the provider and/or backend does not help, this fails with the same error as above.

However, AWS_PROFILE=foo AWS_SDK_LOAD_CONFIG=1 terraform plan (no profile specified in TF file) works perfectly!

I'm glad this works, but I find it surprising and unexpected that AWS_SDK_LOAD_CONFIG=1 is necessary. Why doesn't TF (provider and s3 backend) read the config automatically when ~/.aws/config is present?

[lorengordon]

Ahh yes, attempting to load the profile from the env is a little wonky, currently. It would be nice if terraform setup the session so that it was not necessary to use AWS_SDK_LOAD_CONFIG (it is supported it the SDK, just need to pass the config option when creating the session).

[mungojam]

Maybe this needs a new issue. I've found that credential_process works fine unless I have spaces in the path which then leads to it not passing in the arguments. It works fine with CLI. The underlying issue seems to be the presence of the speech marks

So this works ✅:

[profile devDeveloperRole]
region = eu-west-1
credential_process = C:\SomePath\CredentialsProcess.exe 12345678901 MY_ROLE

But this doesn't ❌:

[profile devDeveloperRole]
region = eu-west-1
credential_process = "C:\SomePath With Spaces\CredentialsProcess.exe" 12345678901 MY_ROLE

And neither does the original with speech marks❌:

[profile devDeveloperRole]
region = eu-west-1
credential_process = "C:\SomePath\CredentialsProcess.exe" 12345678901 MY_ROLE

In the failing cases, the parameters 12345678901 and MY_ROLE don't get passed to CredentialsProcess

[bflad]

@mungojam FYI the AWS configuration file parsing occurs upstream in the AWS Go SDK, so any potential fixes for that particular issue would need to be done there. 👍

[ngrilly]

https://github.com/hashicorp/aws-sdk-go-base/pull/5 has been merged into the AWS Go SDK: this issue seems solved now?

[llamahunter]

With 0.12.20, still seems to require setting export AWS_SDK_LOAD_CONFIG=1 in the shell to work correctly.

[spenczar]

I believe that https://github.com/hashicorp/aws-sdk-go-base/pull/38 fixes this. Once a v0.5.0 release is cut for that package, then updating to it here will fully resolve this issue.

[bflad]

Hi folks 👋 Version 3.0 of the Terraform AWS Provider will include a few authentication changes that should help in this case including:

• Enabling the AWS shared configuration file (e.g. ~/.aws/config) by default
• Ensuring the AWS shared configuration and credential files are read before falling back to the EC2 Instance Metadata Service (should no longer need workarounds to disable it via AWS_METADATA_URL)

Similar fixes were applied to the Terraform S3 Backend (part of Terraform CLI) in version 0.13.0-beta2.

The Terraform AWS Provider major version update will release in the next two weeks or so. Please follow the v3.0.0 milestone for tracking the progress of that release. If you are still having trouble after updating when its released, please file a new issue. Thanks!

[hashibot[bot]]

This has been released in version 3.0.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!