Terraform-provider-vault: Cannot create child token without root priviliges

[tylersmith34]

With the release of 1.6, we're trying to use namespaces when we connect with the Vault provider. Our user in Vault only has access in the namespace, it does not have any root level access. When we execute a run in Terraform Enterprise and hit our Vault Enterprise instance, we get an error about creating child tokens. It seems that namespace support did not get extended to that level. If we add access to the root for our user, it works. Our Vault admins do not want to do this and gave us a short window in which this is acceptable for our testing.

Terraform Version

$ terraform --version
Terraform v0.11.13

+ provider.aws v2.3.0
+ provider.template v2.1.0
+ provider.vault v1.6.0

Affected Resource(s)

-vault_aws_access_credentials

Terraform Configuration Files

provider "vault" {
  address   = "${var.vault_address}"
  namespace = "${var.vault_namespace}"
}

data "vault_aws_access_credentials" "creds" {
  backend = "${var.vault_aws_secret_backend_path}"
  role    = "${var.vault_aws_secret_backend_role}"
}

provider "aws" {
  access_key = "${data.vault_aws_access_credentials.creds.access_key}"
  secret_key = "${data.vault_aws_access_credentials.creds.secret_key}"
  region     = "${var.aws_region}"
}

Debug Output

https://gist.github.com/tylersmith34/51611f2971e3032112465be926292b3c

Error: Error refreshing state: 1 error(s) occurred:

* provider.vault: failed to create limited child token: Error making API request.

URL: POST https://vaulturl.company.com/csg/processing/AIS/Vault/general/prod/v2/v1/auth/token/create
Code: 403. Errors:

* permission denied

Panic Output

No panic output

Expected Behavior

I expect the Vault provider to create a child token in the namespace declared in the provider's configuration (or set through VAULT_NAMESPACE env var). After creating the token, the Vault provider and the vault_aws_access_credentials resource should provide us access to AWS in the declared Vault role.

Actual Behavior

The call to create a child token fails with an authorization error.

Steps to Reproduce

1. terraform plan

Important Factoids

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:
Not exactly the same thing as GH-289, we're not using an admin token in Vault E.

[tyrannosaurus-becks]

@jtcressy was this the issue you want to solve? If not, can you open a separate issue with steps to reproduce what you're seeing so we're on the same page about what we're trying to solve?

There is a general example of how to write enterprise tests here. The way I test in real life is, I run Vault enterprise locally, and set the following env test variables:

TF_ACC_ENTERPRISE=1
TF_ACC=1
VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=<something>

In the original version of the test, we would expect to see a failure using the token given in the steps to reproduce. Then of course, after the code edits, the test would pass. Let me know if you have more questions.

[tyrannosaurus-becks]

FWIW this is regarding the conversation here: https://github.com/terraform-providers/terraform-provider-vault/pull/289#issuecomment-479966767

[jtcressy]

@tyrannosaurus-becks Yes, this is the issue we are experiencing.

[HridoyRoy]

@tyrannosaurus-becks Hi, I noticed that PR 415 was merged. How will we pull in this fix? Is it packaged with version 1.8.0? Thanks!

[tyrannosaurus-becks]

Fix has been merged and will be in the next release, which is imminent.

[trodemaster]

I built the plugin from master branch today and it appears to solve this problem for me. Thanks!

[tyrannosaurus-becks]

@tylersmith34 can you confirm this is also fixed for you?

[tylersmith34]

Please connect with Steve Burns (TAM for my company). We are awaiting on other fixes with the provider to work properly with namespaces, our issues have not been fixed yet.

[tyrannosaurus-becks]

Hi! Actually, I'm able to get this to work on the present release of the Terraform Vault Provider. I believe the child token merely needs a policy attached to it that allows it to create a child token.

Here's an example of how I can successfully get this to work.

First, I create an "education" namespace as demonstrated in this guide. Then, I export VAULT_NAMESPACE=education before executing the following steps.

# Set up an arbitrary Vault secrets back end for testing
vault secrets enable -path jenkins-aws-secrets aws
vault write jenkins-aws-secrets/config/root \
    access_key=redacted \
    secret_key=redacted \
    region=us-east-1
vault write jenkins-aws-secrets/roles/cloud-engineering-jenkins-assume-role \
    credential_type=iam_user \
    policy_document=-<<EOF
{                         
  "Version": "2012-10-17",
  "Statement": [
    {                   
      "Effect": "Allow",
      "Action": "iam:*",
      "Resource": "*"
    }
  ]
}
EOF

Create a policy file that's slightly over-privileged but will work for illustration:

$ cat tokencreatepol.hcl 
path "auth/token/create" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "jenkins-aws-secrets/creds/cloud-engineering-jenkins-assume-role" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Add that policy to Vault:

vault policy write create-token tokencreatepol.hcl

Use that policy to create a child token that will be used for running Terraform with a non-root token:

vault token create -policy=create-token

Then I export the resulting token, and run tf init and tf plan successfully, against the following Terraform config:

provider "vault" {
  namespace = "education"
}

data "vault_aws_access_credentials" "creds" {
  backend = "jenkins-aws-secrets"
  role    = "cloud-engineering-jenkins-assume-role"
}

provider "aws" {
  region     = "us-east-1"
  access_key = "${data.vault_aws_access_credentials.creds.access_key}"
  secret_key = "${data.vault_aws_access_credentials.creds.secret_key}"
}

If you want to use a non-root token for Terraform, you should be able to do it, it will simply need access to the auth/token/create endpoint, as well as all the endpoints that Terraform will consume in Vault.

I believe this could be closed as written, but I'll wait in case there's further comment.

[tylersmith34]

I talked with our account rep and was told that the root problem has not been fixed and will be addressed in 1.3.0 of Vault.