Terraform-provider-vault: Dynamic Secrets within China (cn-northwest-1)

[rbracewell]

I have followed the documentation Using Dynamic Secrets in Terraform and everything works as documented when configured against us-east-1.

When I configure Vault to use the AWS Secrets Engine configured against cn-northwest-1 Terraform constantly fails with the below error

Error: error checking if credentials are valid: InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id:

on main.tf line 9, in data “vault_aws_access_credentials” “creds”:
9: data “vault_aws_access_credentials” “creds” {

Even though Terraform errors I can see that Vault is successfully creating and deleting leased users within the Chinese account when running plan or apply commands. Terraform seems unable to validate the credentials received back from Vault.

Below is a simplified code snippet which exhibits this issue

terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
}

provider "vault" {}

data "vault_aws_access_credentials" "creds" {
  backend = "backend"
  role    = "role"
}

provider "aws" {
  region     = "cn-northwest-1"
  access_key = "${data.vault_aws_access_credentials.creds.access_key}"
  secret_key = "${data.vault_aws_access_credentials.creds.secret_key}"
}

resource "aws_s3_bucket" "tf_testbucket" {
  bucket = "tf-testbucket-blah"
  acl    = "private"
  tags   = local.common_tags
}

[bflad]

Hi @rbracewell 👋 Sorry you are running into trouble here.

The error message seems to be indicating an issue with the Vault data source:

on main.tf line 9, in data “vault_aws_access_credentials” “creds”:

So I'm going to transfer this issue to the Terraform Vault Provider repository for triage.

[bflad]

Hi Vault folks 👋 This appears to be an issue with the vault_aws_access_credentials data source, but please let me know if its not. From my experience with InvalidClientTokenId: The security token included in the request is invalid. errors, it usually indicates that the credentials being used are accessing the wrong AWS partition (e.g. Commercial instead of China).

[bflad]

Briefly looking at the vault_aws_access_credentials code, I'm not sure if anything allows operators to configure the region/partition of the IAM and STS clients:

https://github.com/terraform-providers/terraform-provider-vault/blob/5da0170b5530aa775a1ddd81a5ccca198e0d5996/vault/data_source_aws_access_credentials.go#L158-L168

Which would explain why this error is happening (since the clients will default to Commercial). The data source likely requires reading the region from the backend configuration and passing it through to the AWS clients. 👍 (It seems like it might also be worthwhile to also get the iam_endpoint and sts_endpoint if available as well.)

[jvanan24]

Hi,

It looks like I encountered the same problem but for cn-north-1 region : aws secret backend (STS) + terraform is not working for AWS China.

With the help of Terraform Enterprise support, we firstly thought it was a problem linked to vault, but know I think it's more a terraform / terraform vault provider one.

To give a little bit more information, if I "strace terraform plan" I can see this in logs :

read(14, "\250m\201\200\0\1\0\0\0\1\0\0\3**sts\tamazonaws\3com**0\0"..., 512) = 90
epoll_ctl(8, EPOLL_CTL_DEL, 14, 0xc0001b09fc) = 0
<... futex resumed>) = 0
close(14) = 0
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
epoll_pwait(8, <unfinished ...>
<... socket resumed>) = 14
<... nanosleep resumed>NULL) = 0
setsockopt(14, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("**52.46.134.192"**)}, 16 <unfinished ...>

It seems the endpoint used to validate the token is the global one "sts.amazonaws.com" not the one for cn-north-1 (sts.cn-north-1.amazonaws.com.cn according to this doc ) I suspect that's why we get a 403 error.

[dedamico]

based on testing, this also occurs with AWS GovCloud (US) when trying to use the us-gov-west-1 region. I tried setting the sts_endpoint on the AWS Secret Engine (outside of Terraform, once the secret engine was provisioned) to sts.us-gov-west-1.amazonaws.com but it made no difference in behavior, I got the same error.

[auzroz]

Looks like the region needs to be passed through to the awsConfig
Pseudocode:

rootPath := backend + "/config/root"
regionData, err := client.Logical().ReadWithData(rootPath, data)
region := regionData.Data["region"].(string)

awsConfig := &aws.Config{
        Credentials: credentials.NewStaticCredentials(accessKey, secretKey, securityToken),
        HTTPClient:  cleanhttp.DefaultClient(),
                Region:  region, 
}

References:
https://github.com/aws/aws-sdk-go/blob/master/aws/config.go#L70
https://github.com/aws/aws-sdk-go/blob/master/service/sts/service.go#L48