Terraform-provider-vault: Feature request | Support vault list

[anshprat]

Feature request

Terraform Version

v0.11.1

Affected Resource(s)

Please list the resources as a list, for example:
data vault_generic_secret

Terraform Configuration Files

data "vault_generic_secret" "get_roles" {
  path = "database/roles/?list=true"
}

The code at https://github.com/terraform-providers/terraform-provider-vault/blob/master/vault/data_source_generic_secret.go#L70 calls Logical.read(). However if we could add a check before it to see if the path contains list=true and call Logical.List() accordingly. This can be used to check if a particular role exists or not since a read on the non-existent role will fail and cannot be used to determine further steps.

[apparentlymart]

Hi @anshprat! Thanks for this feature request.

The vault_generic_secret role can often be used as a way to read data from Vault backends that this provider doesn't yet support, but it's _intended_ purpose is to read from the key/value secret backend (which was, prior to hashicorp/vault#3292, called the "Generic" backend). So with that in mind, I think it would be preferable to introduce a new data source to meet your use-case rather than to overload this existing one.

Since Vault's LIST method support seems to have a standard interface across all backends, perhaps we could introduce a new vault_key_list data source that makes a Logical.List() call to the given path and exposes a keys attribute based on the response:

# (not yet implemented)
data "vault_key_list" "db_roles" {
  path = "database/roles"
}

output "db_roles" {
  value = "${data.vault_key_list.db_roles.keys}"
}

Would this meet your use-case?

Before we go forward with this I'd want to consult with the Vault team to confirm that LIST responses do indeed have a standard structure for all cases, since the client library just returns it as a map[string]interface{} and thus _in principle_ could return other structures here. Unfortunately in general Vault's very dynamic structure is hard to represent within Terraform's static schema mode, but hopefully in this case the protocol is standard enough that we _can_ use a single data source for all LIST operations.

[anshprat]

Hi @apparentlymart ,

Yes, the suggested solution will be good for my use-case.

[Gabology]

Just want to add that we would also be very happy to see this added. Our use case right now is for SSL certificate management, we are storing all our SSL certificates in Vault and and we add new ones we need to upload them to GCP and have them added to our HTTPS load balancer.

Currently to work around this we have expressed our Terraform configuration as a consul-template file because consul-template supports iterating over secrets so this in turn templates a local variable with the secret path for every certificate however if you would be to implement this directly in the provider we could skip this extra step of having to run consul-template before running Terraform. It would make sense to be consistent across the HashiCorp stack as well.

[Gabology]

@apparentlymart Any news on this? If you don't have time to implement it would you accept a PR adding the vault_key_list data source?

[draggeta]

This would make iterating over secrets indeed quite nice in Terraform, especially when the new HCL language comes out. I wouldn't mind trying my hand at this if list always returns a list.

[kirecek]

Indeed, this would be a very handy resource. Currently, we are using HTTP data resource which is not very elegant.

data "http" "vault_list" {
  url = "<vault_addr>/<path>?list=true"
  request_headers = {
    "X-Vault-Request" = "true"
    "X-Vault-Token"   = <token>
  }
}

toset(jsondecode(data.http.vault_list.body)["data"]["keys"])

data "external" also works but it would be nice to get support for listing directly from the provider itself.

Has anyone started working on this / has some fork? Or is there any better workaround?