Terraform-provider-vault: Vault provider is unable to access data with a non-root token

[hashibot]

_This issue was originally opened by @mperriere as hashicorp/terraform#16457. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Hi there,

Terraform Version

0.10.6

Terraform Configuration Files

cat main.tf
provider "vault" {
  address = "http://${var.vault_host}:${var.vault_port}"
  skip_tls_verify = true
  token = "dc985ea7-57eb-77b5-17c9-ae86fe019c82"
}

data "vault_generic_secret" "key" {
  path = "secret/dev/onenext/db_user01"
}

output pwd {
 value = "${data.vault_generic_secret.key.data["pwd"]}"
}

cat variable.tf
variable vault_host {
 #default="consuldev.pipeline.aws"
 default="10.196.14.160"
}
variable vault_port {
 default="8200"
}
variable env {
 default="dev"
}
variable project_name {
 default="onenext"
}

Debug Output

terraform-0.10.6 apply
data.external.read_token: Refreshing state...
data.external.fetch_token: Refreshing state...
Error refreshing state: 1 error(s) occurred:

• provider.vault: failed to create limited child token: Error making API request.

URL: POST http://10.196.14.160:8200/v1/auth/token/create
Code: 403. Errors:

• permission denied

Expected Behavior

I'm trying to create a policy based for every project and env: we need a token for every project, in every environnment (DEV, PREP, PROD).
Then i attach a token to it, write a secret inside the path defined in the policy, and finally i'm trying to read it from Terraform. The token should allow read access to the credentials, but it does not.

the rules are defined in a ansible playbook (outside terraform scope)
path "secret/{{env}}/{{project_name}}/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/{{env}}/{{project_name}}" {
capabilities = ["list"]
}
path "auth/token/lookup-self" {
capabilities = ["read"]
}

Next step, i'm creating a token attached to that policy:
(sorry, ansible code too):

• hashivault_token_create:
  display_name: "{{project_name}}-{{env}}"
  policies: ["{{project_name}}-{{env}}"]
  renewable: True
  token: "{{root_token}}"

This token is available, and can be queried on the Vault server:
VAULT_TOKEN=dc985ea7-57eb-77b5-17c9-ae86fe019c82 vault token-lookup --address=http://127.0.0.1:8200
Key Value
--- -----
accessor ec4ccd86-c6b9-b0e5-b337-5504ced231cd
creation_time 1508927130
creation_ttl 2764800
display_name token-onenext-dev
explicit_max_ttl 0
id dc985ea7-57eb-77b5-17c9-ae86fe019c82
meta
num_uses 0
orphan false
path auth/token/create
policies [default onenext-dev]
renewable true
ttl 2761029

Next step. i'm writing a secret in the policy path. No issue there.

Now i'm trying to read it from the vault cli: this is ok, i can read the secret. This proves that the secret, policy and token are correctly defined:

VAULT_TOKEN=dc985ea7-57eb-77b5-17c9-ae86fe019c82 vault read --address=http://127.0.0.1:8200/ secret/dev/onenext/db_user01
Key Value
--- -----
refresh_interval 768h0m0s
pwd tai.s0dohM5r

Now i'm trying to do the same thing with terraform.
I get that error:

• provider.vault: failed to create limited child token: Error making API request.
  URL: POST http://10.196.14.160:8200/v1/auth/token/create
  Code: 403. Errors:

PS: if i switch back to the root token in the provider vault definition, i can access to the secret.

It seems that terraform is trying to create a temporary token instead of using the one newly created to just read the secret key with the provided token.

Perhaps i did not understand the documentation? I did not find full usage examples on the internet, i don't know what's wrong here.

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

1. terraform apply

[mperriere]

after a day of various other tests, il really don't understand how we can use Vault Token from terraform.
if i provide another token than the root token in the provider "vault", i'm unable to read vault entries.

Do we conclude that actually it's note possible to use non-root token with terraform?
Or do you have an working example where a policy limited access with a path restriction can be use from Terraform?

[mroark-exp]

Hi @mperriere I ran into the same issue as this in my local install and I'm trying to understand all the implications of this issue. In the prior issue, hashicorp/terraform#16457, you mentioned, "opening all rights on auth/token/create, i give too much rights to that policy".

What do you think the implications are of granting that right to the policy that makes it unacceptable? I'm honestly curious about this, because I haven't been able to come up with any particular problem with it.

As far as I can understand, auth/token/create only allows you to create child tokens of the token you already have, which will still be revoked when the parent is revoked.

[mperriere]

Hello,
after a bunch of new tests i realized that i have 2 different issues: find a working policy and provide the token stored on S3:
First i found a policy that worked:

path "secret/dev/onenext/*" {
capabilities = ["read", "create", "update", "list"]
}

path "auth/token/create" {
capabilities = ["create", "update"]
}
As terraform will create a temporary token based on the provided token, we need to provide auth/token/create rights with "create" and "update". I didn't want to allow other capabilities than access to the secrets, but allowing temporary token creation is finally not a big deal.

Warning: the tfstate is caching the results, access to vault is only made once.

from TF doc, i found:
_"Terraform will issue itself a new token that is a child of the one given, with a short TTL to limit the exposure of any requested secrets."_
This suggest that my initial policy has not enough rights to provide auth/token/create.

-> yes, confirmed.

Second issue: how to get the token without writing it inside the TF code.

Using an external provider to define token variable fails:
provider "vault" {
address = "http://${var.consul_host}:8200"
skip_tls_verify = true
token = "${data.external.read_token.result["token"]}"
}

Error refreshing state: 1 error(s) occurred:
provider.vault: failed to create limited child token: Error making API request.
URL: POST http://consuldev.pipeline.aws:8200/v1/auth/token/create
Code: 403. Errors:
permission denied

-> opened the case https://github.com/hashicorp/terraform/issues/16478

[sidewinder12s]

@mperriere Your correct about the child token creation, I ran into that as well when I was using LDAP based Auth.

As far as where to keep the token, generally they recommend you keep it as an ENV var and the Vault provider will automatically pick it up. If you use an env var you don't need to specify anything in the provider config.

[leftathome]

If you have concerns about tokens being sniffed from the environment by other processes on the same system, you could also write it to the filesystem with an outboard process just before the Terraform run and read it into the provider config with file() ...

[mperriere]

@sidewinder12s yes, i exported the Token as an ENV var as workaround. Thanks.

[martin-andrew-ellis]

I solved this by granting the update capability on the auth/token/create path to users that need to run terraform commands. It seems safer then digging out the root token in terms of security, accidents and audibility.

But I'm left wondering: Does create new security hole?

It also occurs to me that that ephemeral data like auth tokens are not really part of the terraform state. It's different each time. The vault address is part of the state, changing it changes the outcome, and, unfortunately, the secrets fetched from vault are normally part of the state, but the auth token is not logically. (Even if it is treated as such in practice)

I think that terraform needs a way of dealing with such data and lot of the problems go away.

[weihj1999]

I meet same issue with my dev server. @martin-andrew-ellis could you share more detail how you update capability on auth/token/create path to allow child token created.?

[cvbarros]

This is clearly documented:
https://www.terraform.io/docs/providers/vault/index.html#using-vault-credentials-in-terraform-configuration

The "automation" persona that will be used to execute Terraform in Vault should be capable of update on auth/token/create. They can only create tokens that have the same or lower permissions, never more.

Usually, there are a lot more permissions involved to grant when automating Vault configuration with Terraform and not use a root token. For that, what I do is set an "automation" policy that can perform all these actions.

[martin-andrew-ellis]

@weihj1999

I create a policy HCL file in the form

path "auth/token/create" {
  capabilities = [ "update" ]
}

Then create the policy with vault policy write token-create token-create.hcl (using the root auth token)

This can be applied LDAP group representing privileged users or as @cvbarros indicated an automation or terraform AppRole.

Having said that, given that I am adding a single capability normally reserved for root I wonder if I'm trading once security flaw with another.

[martin-andrew-ellis]

As I said before, I think the _real_ solution is for terraform to be more controlled as to the contents of the state.

I would love it, if when defining attributes in resources or providers, that they get marked with one of three terms:

• stateful - the default and ends as part of the state
• ephemeral - the data is not placed in the state. Vault auth tokens should have this one
• sensitive - the data is represented in the state with a one way hash.

This last term would only work if the state is just a state and not a cache. For this I am thinking of resource attributes like vault_generic_secret.data_json and aws_secretsmanager_secret_version.secret_string or anything else that is implicitly a secret and non-ephemeral

[jmccann]

I wonder if creating a child token could be a configurable option, defaulting to current behavior.

Right now a child token is required to be created. I would love to see an option for the provider to allow disabling that requirement and just use the token that was provided.

I get that with the correct policy setup in place this should work fine. However, sometimes we are not in control of the policies. In that case it's a bit frustrating when trying to provide a token that has perms necessary to do reads/writes but fails on trying to create a child token.

If people are open to this idea I don't mind PR'ing it.

[mjtice]

A configurable option is exactly what I'm looking for.

I'm trying to use a token that's created via the approle auth method. The problem I've found is that because that particular token doesn't have a parent (due to the auth method), it can't create a child - regardless of the policy I try to give it.

[m0ps]

this issue was raised a long time ago, but it still lacks attention from hashicorp team.
Patch for implementing sub-token as an optional (not a mandatory) feature was suggested a long time ago. I'm wondering why it still open, without any comments from dev team.

[krzkowalczyk]

I hit the same issue with not being able to use approle auth method with terraform vault provider. I hope the disable child token creation will be merged soon.

[ksummersill2]

I solved this by granting the update capability on the auth/token/create path to users that need to run terraform commands. It seems safer then digging out the root token in terms of security, accidents and audibility.

But I'm left wondering: Does create new security hole?

It also occurs to me that that ephemeral data like auth tokens are not really part of the terraform state. It's different each time. The vault address is part of the state, changing it changes the outcome, and, unfortunately, the secrets fetched from vault are normally part of the state, but the auth token is not logically. (Even if it is treated as such in practice)

I think that terraform needs a way of dealing with such data and lot of the problems go away.

That fixed the issue. The default policy that comes with terraform does not allow for the creation of a temp token. Therefore i create a copy of the default and added the policy of:

# Allow tokens to look up their own properties
path "auth/token/create" {
    capabilities = ["update"]
}