Terraform-provider-vault: Bug with vault_approle_auth_backend_role token_policies

[CarpathianUA]

Terraform Version

Terraform v0.11.14

• provider.vault v2.3.0

Affected Resource(s)

Please list the resources as a list, for example:

• vault_approle_auth_backend_role

Terraform Configuration Files

My module resources:

resource "vault_approle_auth_backend_role" "this" {
  count              = "${length(keys(var.vault_approles))}"
  backend            = "${var.approle_auth_backend_path}"
  role_name          = "${element(keys(var.vault_approles), count.index)}"
  token_policies     = ["${element(keys(var.vault_approles), count.index)}"]
  secret_id_num_uses = 0
  secret_id_ttl      = "36000"
  token_ttl          = "36000"
}

resource "vault_approle_auth_backend_role_secret_id" "this" {
  depends_on = ["vault_approle_auth_backend_role.this"]
  count      = "${length(keys(var.vault_approles))}"
  backend    = "${var.approle_auth_backend_path}"
  role_name  = "${element(keys(var.vault_approles), count.index)}"
}

resource "vault_policy" "this" {
  count  = "${length(keys(var.vault_approles))}"
  name   = "${element(keys(var.vault_approles), count.index)}"
  policy = "${element(values(var.vault_approles), count.index)}"
}

Configuration:

module "k8s-sre-approle" {
  source = "<path_to_module>"

  approle_auth_backend_path = "hub_approle"

  vault_approles {
    k8s-sre-kv-approle-jenkins     = "${data.vault_policy_document.vault_prod_k8s_sre_kv_approle_jenkins.hcl}"
    k8s-sre-kv-approle-artifactory = "${data.vault_policy_document.vault_prod_k8s_sre_kv_approle_artifactory.hcl}"
  }
}

Expected Behavior

token_policies argument is assigned with policies after apply and reflected in state.

Actual Behavior

terraform apply

Terraform will perform the following actions:

  + module.k8s-sre-approle.vault_approle_auth_backend_role.this[0]
      id:                        <computed>
      backend:                   "hub_approle"
      bind_secret_id:            "true"
      role_id:                   <computed>
      role_name:                 "k8s-sre-kv-approle-artifactory"
      secret_id_num_uses:        "0"
      secret_id_ttl:             "36000"
      token_policies.#:          "1"
      token_policies.2374882508: "k8s-sre-kv-approle-artifactory"
      token_ttl:                 "36000"
      token_type:                "default"

  + module.k8s-sre-approle.vault_approle_auth_backend_role.this[1]
      id:                        <computed>
      backend:                   "hub_approle"
      bind_secret_id:            "true"
      role_id:                   <computed>
      role_name:                 "k8s-sre-kv-approle-jenkins"
      secret_id_num_uses:        "0"
      secret_id_ttl:             "36000"
      token_policies.#:          "1"
      token_policies.2057305492: "k8s-sre-kv-approle-jenkins"
      token_ttl:                 "36000"
      token_type:                "default"

  + module.k8s-sre-approle.vault_approle_auth_backend_role_secret_id.this[0]
      id:                        <computed>
      accessor:                  <computed>
      backend:                   "hub_approle"
      role_name:                 "k8s-sre-kv-approle-artifactory"
      secret_id:                 <computed>
      wrapping_accessor:         <computed>
      wrapping_token:            <computed>

  + module.k8s-sre-approle.vault_approle_auth_backend_role_secret_id.this[1]
      id:                        <computed>
      accessor:                  <computed>
      backend:                   "hub_approle"
      role_name:                 "k8s-sre-kv-approle-jenkins"
      secret_id:                 <computed>
      wrapping_accessor:         <computed>
      wrapping_token:            <computed>

  + module.k8s-sre-approle.vault_policy.this[0]
      id:                        <computed>
      name:                      "k8s-sre-kv-approle-artifactory"
      policy:                    "# Policy for k8s-sre Artifactory Approle\npath \"k8s-sre-kv/artifactory/*\" {\n  capabilities = [\"read\"]\n}\n"

  + module.k8s-sre-approle.vault_policy.this[1]
      id:                        <computed>
      name:                      "k8s-sre-kv-approle-jenkins"
      policy:                    "# Policy for k8s-sre Jenkins Approle\npath \"k8s-sre-kv/*\" {\n  capabilities = [\"read\"]\n}\n"

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Next subsequents apply shows difference in state:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.k8s-sre-approle.vault_approle_auth_backend_role.this[0]
      policies.#:                "1" => "0"
      policies.1971754988:       "default" => ""
      token_policies.#:          "0" => "1"
      token_policies.2374882508: "" => "k8s-sre-kv-approle-artifactory"

  ~ module.k8s-sre-approle.vault_approle_auth_backend_role.this[1]
      policies.#:                "1" => "0"
      policies.1971754988:       "default" => ""
      token_policies.#:          "0" => "1"
      token_policies.2057305492: "" => "k8s-sre-kv-approle-jenkins"


Plan: 0 to add, 2 to change, 0 to destroy.

Beggining from provider version 2.0.0 policies argument was changed to token_policies and I wonder why it's present here. Actually, no values are being assigned to token_policies and each subsequent apply show a difference and vault_approle_auth_backend_role token_policies argument is now working at all.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply vault_approle_auth_backend_role resource with token_policies argument defined.
2. terraform apply again
3. See the differnce as above.

Important Factoids

Everything works perfect with provider version 2.0.0 and policies argument instead of token_policies

Thanks in advance!

[AndriiOmelianenko]

+1

[lawliet89]

token_policies is for Vault >= 1.2.0.

[CarpathianUA]

@lawliet89
token_policies works with Vault 1.1.2 and prover version 2.0.0, but with provider version 2.3.0 the bug is reproduced.

[universam1]

seeing the same bug - any progress?

[dhohengassner]

+1

[acornies]

+1

[StephenWithPH]

Adding commentary in case it helps anyone... I set the deprecated policies field rather than the recommended token_policies field, and that resolved a gnarly issue I was having (by way of Vault Agent auto-auth) with the https://github.com/hashicorp/vault-plugin-auth-gcp.

[earchibald-lv]

+1

Also, as @StephenWithPH noted, using the deprecated policies field throws a warning, but works, where the recommended token_policies field does not.

[pcman312]

This looks like an issue when using Vault < 1.2.0 with token_policies. I recommend upgrading to Vault >= 1.2.0

I was unable to reproduce this with Vault 1.2.0 and provider 2.3.0. However, I decided to try this with Vault 1.1.2 and was able to reproduce the error you were seeing. I also tried using Vault 1.1.2 with provider 2.0.0 but got an error:

Error: module.k8s-sre-approle.vault_approle_auth_backend_role.this[0]: : invalid or unknown key: token_policies

The token_policies fields aren't available prior to Vault 1.2.0. For Vault < 1.2.0, use the policies parameter instead. There is additional information at https://www.terraform.io/docs/providers/vault/r/approle_auth_backend_role.html#deprecated-arguments

[yermulnik]

Facing this bug too:

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.3.1
[…]

Terraform v0.11.14

- Downloading plugin for provider "vault" (2.7.1)...

resource "vault_approle_auth_backend_role" "tcs" {
  role_name      = "tcs"
  token_policies = ["default", "tcs_app"]
}



md5-cc902d02318e992c705b4ba6a7d9c0c9



vault_approle_auth_backend_role.tcs: Modifying... (ID: auth/approle/role/tcs)
  policies.#:                "2" => "0"
  policies.1971754988:       "default" => ""
  policies.2311870954:       "tcs_app" => ""
  token_policies.#:          "0" => "2"
  token_policies.1971754988: "" => "default"
  token_policies.2311870954: "" => "tcs_app"

[pcman312]

@yermulnik I can't seem to reproduce this locally. Here's what I've done:

Run vault 1.3.1 in dev mode (pulled from the v1.3.1 tag):

vault server -dev

```
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.3.1
Cluster Name vault-cluster-3f93e9b2
Cluster ID e806ec42-9ee9-3bc6-897a-3fd2572d5a3c
HA Enabled false

I've also run this locally in non-dev mode.

Manually enabled approle auth method:

$ vault auth enable approle
Success! Enabled approle auth method at: approle/

Here's my `main.tf` file:
```hcl
provider "vault" {
  version = "2.7.1"
}

resource "vault_approle_auth_backend_role" "tcs" {
  role_name = "tcs"
  token_policies = ["default", "tcs_app"]
}

(no other files in the folder)

Terraform:

$ terraform init
<successful>

$ terraform version
Terraform v0.11.14
+ provider.vault v2.7.1

terraform apply is successful. Subsequent calls show no changes.

I'm struggling to find what is different from your example.

[yermulnik]

@pcman312 Hmm, interesting… Thanks for giving it a try.
What if you enable approle backend and create tcs_app policy using Vault as well?

resource "vault_auth_backend" "approle" {
  type = "approle"
}

resource "vault_policy" "tcs_app" {
  name = "tcs_app"

  policy = <<EOT
path "tcs/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOT
}

[pcman312]

@yermulnik I keep getting issues with the vault_policy you gave me. It's complaining that the endpoint doesn't exist. I'm new to this provider, so I'm unsure exactly the problem there.

If I remove the vault_policy I get the same result as before: terraform apply works and doesn't show any changes on the second run.

[yermulnik]

@pcman312

It's complaining that the endpoint doesn't exist.

what was the actual error? (so that I can paste code snippet for you to fix)
also would be cool if provider devs could take a look into it — it's quite annoying issue as our CI plans loads of changes every run =(

[pcman312]

@yermulnik

Error: Error applying plan:

1 error occurred:
    * vault_approle_auth_backend_role.tcs: 1 error occurred:
    * vault_approle_auth_backend_role.tcs: error writing AppRole auth backend role "auth/approle/role/tcs": Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/approle/role/tcs
Code: 404. Errors:

* no handler for route 'auth/approle/role/tcs'

I tried ordering the vault_policy in different points in the config file with no change.

Here's my main.tf:

provider "vault" {
  version = "2.7.1"
}

resource "vault_auth_backend" "approle" {
  type = "approle"
}

resource "vault_policy" "tcs_app" {
  name = "tcs_app"

  policy = <<EOT
path "auth/approle/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOT
}

resource "vault_approle_auth_backend_role" "tcs" {
    role_name = "tcs"
    token_policies = ["default", "tcs_app"]
}

[yermulnik]

@pcman312 Hmm, looks strange.
What if you define backend for the role explicitly like this (so that it depends on vault_auth_backend and is not attempted to be created before the backend itself):

resource "vault_approle_auth_backend_role" "tcs" {
        backend = "${vault_auth_backend.approle.path}"
    role_name = "tcs"
    token_policies = ["default", "tcs_app"]
}

[pcman312]

@yermulnik That seems to have fixed it. Unfortunately I'm still not able to replicate the issue:
Initial apply

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "vault" (2.7.1)...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
vault_policy.tcs_app: Creating...
  name:   "" => "tcs_app"
  policy: "" => "path \"auth/approle/*\" {\n  capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n"
vault_auth_backend.approle: Creating...
  accessor:                  "" => "<computed>"
  default_lease_ttl_seconds: "" => "<computed>"
  max_lease_ttl_seconds:     "" => "<computed>"
  path:                      "" => "<computed>"
  type:                      "" => "approle"
vault_policy.tcs_app: Creation complete after 0s (ID: tcs_app)
vault_auth_backend.approle: Creation complete after 0s (ID: approle)
vault_approle_auth_backend_role.tcs: Creating...
  backend:                   "" => "approle"
  bind_secret_id:            "" => "true"
  role_id:                   "" => "<computed>"
  role_name:                 "" => "tcs"
  token_policies.#:          "" => "2"
  token_policies.1971754988: "" => "default"
  token_policies.2311870954: "" => "tcs_app"
  token_type:                "" => "default"
vault_approle_auth_backend_role.tcs: Creation complete after 0s (ID: auth/approle/role/tcs)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Second run:

vault_auth_backend.approle: Refreshing state... (ID: approle)
vault_policy.tcs_app: Refreshing state... (ID: tcs_app)
vault_approle_auth_backend_role.tcs: Refreshing state... (ID: auth/approle/role/tcs)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[yermulnik]

@pcman312 Interesting. Thanks for giving it a go.
Which version of Terraform do you use?
I'm leaning to think this might be related to existing configuration that had been using previous version of TF vault provider and vault and which had been upgraded to a newer version of vault and vault provider hence the issue with state.

[yermulnik]

Weird that I cannot re-deploy my vault clusters from the scratch to see if it would fix the issue since these are under heavy use at the moment =(

[pcman312]

@yermulnik that seems like a likely explanation for the problem that you and others have been seeing. I'm using Terraform 0.11.14, provider 2.7.1, and Vault 1.3.1, but I am using a completely fresh setup. What version did you upgrade from? I can try using the old version and then upgrading to see if it can be replicated.

[yermulnik]

@pcman312 We upgraded from vault provider 1.9.* to 2.7.1 and vault 1.1.3 to 1.3.1

[pcman312]

@yermulnik I've managed to partially reproduce this issue.

Running Vault 1.1.3 (not in dev mode), TF 0.11.14, Provider 2.7.1, I performed a terraform apply, followed by another terraform apply and got an error similar to the one you mentioned previously:

Terraform will perform the following actions:

  ~ vault_approle_auth_backend_role.tcs
      token_policies.#:          "0" => "2"
      token_policies.1971754988: "" => "default"
      token_policies.2311870954: "" => "tcs_app"


Plan: 0 to add, 1 to change, 0 to destroy.

If I then shut down vault and start up v1.3.1 and run terraform apply it shows the same diff as above, however subsequent calls to terraform apply don't show a diff any longer.

I'm going to try the same procedure but comparing Vault 1.1.3 & Provider 1.9.0 to Vault 1.3.1 & Provider 2.7.1 to see how the provider plays a role in this.

[yermulnik]

@pcman312

I've managed to partially reproduce this issue.

Oh, I saw this behaviour as well. So basically once token_policies were planned and applied subsequent plan/apply would want to make changes to policies like kind of reverting previously applied changes — basically what you _partially_ reproduced was what we started this journey with, but at some point it started to produce changes I mentioned in the first message I posted to this thread (replace policies with token_policies — I didn't apply these so no idea what was _supposed_ to go next).
Thanks again for looking into this.

[pcman312]

@yermulnik Ah, okay. I thought you had applied those changes and were still seeing the issue. Based on the diff, I'm reasonably confident it's going to remove policies and add token_policies which I believe is be expected behavior. Do you have a way of applying those changes into a non-prod environment to make sure they work as expected before going to prod?

[yermulnik]

@pcman312 I might had put myself not quite clear. Despite tf configuration having token_policies and having it applied TF wants to remove policies and put token_policies. Basically what the initial issue was about (the very 1st message from CarpathianUA in this issue). So once we've got our vault and tf vault provider upgraded to 2.7.1 and 1.3.1 respectively with a switch from policies to token_policies we have started to hit this issue — constant changes to plan nevertheless it was applied: removing policies + adding token_policies (while tf configuration has token_policies in there).

[pcman312]

@yermulnik I think I've reproduced the perpetual diff problem but want to confirm that this is the same steps you took:

1. Using Vault 1.1.3/Provider 1.9.0: main.tf:

resource "vault_approle_auth_backend_role" "tcs" {
  backend = "${vault_auth_backend.approle.path}"
  role_name = "tcs"
  policies = ["default", "tcs_app"]
}

1. $ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "vault" (1.9.0)...

Terraform has been successfully initialized!

1. $ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + vault_approle_auth_backend_role.tcs
      id:                        <computed>
      backend:                   "${vault_auth_backend.approle.path}"
      bind_secret_id:            "true"
      policies.#:                "2"
      policies.1971754988:       "default"
      policies.2311870954:       "tcs_app"
      role_id:                   <computed>
      role_name:                 "tcs"

  + vault_auth_backend.approle
      id:                        <computed>
      accessor:                  <computed>
      default_lease_ttl_seconds: <computed>
      max_lease_ttl_seconds:     <computed>
      path:                      <computed>
      type:                      "approle"

  + vault_policy.tcs_app
      id:                        <computed>
      name:                      "tcs_app"
      policy:                    "path \"auth/approle/*\" {\n  capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n"


Plan: 3 to add, 0 to change, 0 to destroy.
<completes successfully>

1. Upgrade Vault -> 1.3.1, Provider -> 2.7.1
2. Update main.tf to change policies -> token_policies (no other changes aside from the provider version)

resource "vault_approle_auth_backend_role" "tcs" {
  backend = "${vault_auth_backend.approle.path}"
  role_name = "tcs"
  token_policies = ["default", "tcs_app"]
}

1. Run terraform apply, get the diff:

Terraform will perform the following actions:

  ~ vault_approle_auth_backend_role.tcs
      policies.#:                "2" => "0"
      policies.1971754988:       "default" => ""
      policies.2311870954:       "tcs_app" => ""
      token_policies.#:          "0" => "2"
      token_policies.1971754988: "" => "default"
      token_policies.2311870954: "" => "tcs_app"

1. Repeat terraform apply and get the same diff every time it's run.

Is this an accurate description of the steps you took?

[yermulnik]

@pcman312 Looks very similar to what I can recall. Apart from using backend attribute.

[pcman312]

@yermulnik Sorry for the delayed update. I looked into the code for this but haven't come up with a solution to it yet. In the meantime you can look into using terraform state mv to move the policies values to token_policies. There is an rm command as well, but I'm a bit more hesitant to recommend it. It's the next thing I'm going to look into when I can loop back to this issue.

[yermulnik]

@pcman312 no problem. thanks for looking into this anyways. it's not very vital for us apart from having either back and forth with policies and token_policies or deprecation notices.
btw do you mean that terraform state mv should help to get rid of back and forth of policies and token_policies? having that terraform does the initial change from policies to token_policies itself but then it starts to move it back and forth. also I cannot see how terraform state mv would be used for a attribute of the resource.

[pcman312]

@yermulnik That's my current thought. policies is transformed to token_policies, but until we figure out a code solution to this, terraform state mv or terraform state rm might work. I don't know for sure if it will work, and I'm not sure if it will cause any other problems to surface. I want to do more testing with it.

[lucymhdavies]

As some additional information which may help identify what's going on here...

We have some AppRoles which were created with an old version of Vault (I think around v0.9.6, but it's from before the change from policies to token_policies)

We're now on Vault v1.4.0, and provider v2.10.0.

What I have found is that AppRoles created in the older version of Vault actually do have both the old policies and period fields, and the new token_policies and token_period fields when you read them from Vault:

$ vault read auth/approle/role/test
Key                        Value
---                        -----
bind_secret_id             true
local_secret_ids           false
period                     24h
policies                   [default test]
secret_id_bound_cidrs      <nil>
secret_id_num_uses         1
secret_id_ttl              5m
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              8h
token_no_default_policy    false
token_num_uses             0
token_period               24h
token_policies             [default test]
token_ttl                  20m
token_type                 default

Removing the AppRole from the Terraform state with terraform state rm, then re-importing it, we still see the issue:

  ~ vault_approle_auth_backend_role.test
      period:                    "86400" => "0"
      policies.#:                "2" => "0"
      policies.1971754988:       "default" => ""
      policies.309839343:        "test" => ""
      token_period:              "0" => "86400"
      token_policies.#:          "0" => "2"
      token_policies.1971754988: "" => "default"
      token_policies.309839343:  "" => "test"

i.e. Terraform detects that the polices field exists still, and tries to update token_policies

Newly created AppRoles do not have this issue. When I read those from Vault, they do not have the policies or period fields, and Terraform is able to successfully detect that it does not need to make any changes.

As another interesting data point, we do NOT see this issue with the vault_aws_auth_backend_role or vault_kubernetes_auth_backend_role resources, even though Vault behaves the same for those:

$ vault read auth/aws/role/test
Key                               Value
---                               -----
allow_instance_migration          false
auth_type                         iam
bound_account_id                  []
bound_ami_id                      []
bound_ec2_instance_id             <nil>
bound_iam_instance_profile_arn    []
bound_iam_principal_arn           [arn:aws:iam::REDACTED:role/va-demo-vault-client-role arn:aws:iam::REDACTED:role/vault-client-role]
bound_iam_principal_id            [REDACTED REDACTED]
bound_iam_role_arn                []
bound_region                      []
bound_subnet_id                   []
bound_vpc_id                      []
disallow_reauthentication         false
inferred_aws_region               n/a
inferred_entity_type              n/a
policies                          [default test]
resolve_aws_unique_ids            true
role_id                           a4062777-810a-eb1b-b8f0-375e5b9071d3
role_tag                          n/a
token_bound_cidrs                 []
token_explicit_max_ttl            0s
token_max_ttl                     0s
token_no_default_policy           false
token_num_uses                    0
token_period                      0s
token_policies                    [default test]
token_ttl                         0s
token_type                        default

Note from above that the policies field exists when I read this from Vault, and yet Terraform is able to handle this gracefully.

As I look into this further, it appears that the vault_aws_auth_backend_role has some code to explicitly handle this type of issue:
https://github.com/terraform-providers/terraform-provider-vault/blob/master/vault/resource_aws_auth_backend_role.go#L437-L448

Whereas vault_approle_auth_backend_role does not:
https://github.com/terraform-providers/terraform-provider-vault/blob/master/vault/resource_approle_auth_backend_role.go#L260-L267

[catsby]

Hello - I opened https://github.com/terraform-providers/terraform-provider-vault/pull/744 to address this. Thanks to @lucymhdavies for pointing me in the right direction for the fix.

If you have the ability to checkout the repo, compile it, and give it a try, please let us know if it works out.