Terraform-provider-vault: FR: Add option to avoid storing some values in resource vault_generic_secret

[jgiles]

I'm trying to use Terraform to configure and manage our Vault instances. As I understand it, I should be able to do most of this using vault_generic_secret resources.

I want to avoid persisting, tracking, or logging the values that are actually "secret". I want to provide these parameters at 'apply' time, release Terraform of any responsibility for state/drift tracking, and rest easy knowing there aren't secret copies lurking in Terraform state files.

The long-lived https://github.com/hashicorp/terraform/issues/516 is the general case of this problem in Terraform.

https://github.com/hashicorp/terraform/pull/15797 attempts a general Terraform solution to this problem, but is stalled on reasonable questions about the deep Terraform-wide implications of the change.

I propose something more modest: vault_generic_secret is specifically targeted at uploading configuration that often includes secrets. Can we solve the secret-persistence problem (even just temporarily) for just this resource?

Would it work to add a secret_data_json field or similar alongside data_json? secret_data_json would be marked Sensitive (data_json currently is not), and would not be persisted to state. It would be merged with data_json when sent to Vault.

I don't know how to prevent persisting the value - would setting StateFunc to a function returning "" work? Or is something more complicated required?

I'm happy to help with implementation.

References

https://github.com/hashicorp/terraform/issues/516
https://github.com/hashicorp/terraform/pull/15797

[paddycarver]

Hi @jgiles! Thanks for opening an issue. I must admit I'm a little confused. What it sounds like is you're looking for Terraform to make an API call, and then not keep any of the results around in state. Is that right? If that's the case, what does Terraform provide that using a provisioner that calls out to a vault CLI doesn't provide? If Terraform isn't keeping any state around, I don't really see what benefit it's providing. Can you elaborate a bit on what you're hoping to achieve?

[jgiles]

Sure!

(Note: Having written the below and then reading your comment again, it occurs to me that you might have meant I could use a trick with Terraform provisioners to accomplish this. If so, what would that look like?)

The vast majority of our Vault configuration data is non-sensitive: We are comfortable making visible to everyone with source access information like what people are in which groups, what policies are applied to which groups, how we have our internal CA configured, which IAM roles can get access to which secrets. We want to check in all this configuration and do change management through a "commit code, deploy" pattern.

However, there are a few pieces of information we do NOT want to check in: Okta/Duo API keys, AWS keys with the ability to generate temporary IAM roles, database credentials with root or near-root authority.

A provisioner calling out the the Vault CLI is our current solution: We do something similar to https://www.hashicorp.com/blog/codifying-vault-policies-and-configuration.html, checking in a mix of JSON files and HCL policy docs and running a Bash script calling vault write or vault policy write as appropriate. The aforementioned "sensitive" fields are subbed in from environment variables with an envsubst preprocessing step.

Terraform offers a number of improvements over this arrangement:

• Proper handling of removed/deleted resources
• No needless overwrites of unchanged resources (some Vault resources do not tolerate overwrites)
• Confirmation of the configuration delta at deployment time
• Significantly improved configuration authoring environment: Variables, commenting, more concise and forgiving syntax, sanely re-usable modules
• Less brittle/hacky/error-prone than rolling our own provisioner code (in Bash or otherwise)

For all of these reasons, we want to Terraform the entirety of our Vault configuration. Except, there are a few fields we can't tolerate ending up in Terraform's saved state. For things like the AWS and DB creds, we want to create them, put them in Vault, and destroy all other copies.

Hence this proposal: Add the option to exclude selected Vault resource fields from persistence in Terraform state. It's hashicorp/terraform#15797 scoped down to just Vault rules.

For now, I can use our current process for the resources with sensitive pieces and Terraform for the rest. But that's not a great arrangement long-term; this seems like a problem worth solving in the Terraform tooling.

[jgiles]

Here is my current Terraform-based workaround using a provisioner: I declare a module sensitive-value containing the following:

variable "path" {
  description = "The path to store the value at."
}

variable "value" {
  description = "The value to json-encode and store. Sensitive fields are like $${ENV_VAR}."
  type = "map"
}

variable "triggers" {
  description = "Triggers for refreshing the sensitive value."
  type = "map"
  default = {}
}

resource "null_resource" "sensitive_value" {
  triggers = "${var.triggers}"
  provisioner "local-exec" {
    command = <<EOT
envsubst <<EOTFJSON | vault write ${var.path} -
${jsonencode(var.value)}
EOTFJSON
EOT
  }
}

It is used like this:

module "okta_duo_access" {
  source = "../sensitive-value"
  path = "auth/${vault_auth_backend.okta.id}/duo/access"
  value = {
    host = "redacted_subdomain.duosecurity.com"
    ikey = "redacted_ikey"
    skey = "$${DUO_SKEY}"
  }
}

Where DUO_SKEY is the environment variable holding the secret.

This works pretty well. The key advantage of doing this in Terraform itself is that you get the ordering guarantees (e.g. ${vault_auth_backend.okta.id} makes sure this runs after the Okta backend is mounted).

It would still be great to have this directly supported by the provider, though.

EDIT: Updated example code to better handle arbitrary JSON values (in particular, the single quotes that can crop up in DB connection configs).

[daveadams]

Yes, it would be amazing to be able to mark certain fields to either not be stored in state or to be encrypted. Seems like a perfect application for Vault Transit. That level of integration would probably have to happen in Terraform itself, I suppose. But in any case, yes, there are often situations (like the Vault LDAP auth provider or the various secret backends) where I would like to configure the backends with Terraform and benefit from keeping the state of all of the fields... except the one field that's secret. I don't want a management token for Consul stored in my TF state, although I would like to be able to configure Consul secret providers with Terraform. etc etc.

I'm not sure if this is the right ticket to have made this rant but i figure any noise helps the team notice.

[paddycarver]

For reference, the suggestion I was making was to use local-exec to run a Vault command:

resource "null_resource" "vault_write" {
  provisioner "local-exec" {
    inline = [
      "vault write ......",
    ]
  }
}

Not keeping the results around in state means Terraform can't really do anything about diffing, ensuring the value is set, or anything like that, which basically breaks most of Terraform's functionality. At that point, you're essentially just running a CLI command, so why not just do that? If you want to make it happen when you call terraform apply, the above is a way to make that happen.

Most of the benefits of Terraform you mention, namely:

• Proper handling of removed/deleted resources
• No needless overwrites of unchanged resources (some Vault resources do not tolerate overwrites)
• Confirmation of the configuration delta at deployment time

All rely on state. They can't function without the value being in state.

I know the Terraform core team is interested in encrypting an entire state file or just some fields of it, but as far as I know it's not on the immediate roadmap.

[daveadams]

@paddycarver I think you're missing the point. We do want TF to handle the state for these secrets and keep track of things. But currently, doing so means we have to let the secrets be written to the tfstate file unencrypted. There is no option to encrypt certain fields at rest and let Terraform decrypt them in memory at runtime to do the comparisons.

[paddycarver]

I think you're missing the point.

I'm almost certain I am, because I remain very confused! :) I think all the talk of avoiding storing a value, keeping things out of state, and not persisting things had me confused.

My stance on encrypting the values is still that waiting for Terraform to encrypt fields marked as sensitive is a better approach than trying to reimplement encryption on each and every sensitive field ourselves, but I'm willing to be persuaded otherwise.

[betabandido]

@paddycarver How do you safely capture the output of the command executed in the local-exec provisioner? One option would be to redirect the output to disk, but then the secret gets leaked into the file.

My use case is something similar to:

provider "vault" {}

data "vault_generic_secret" "newrelic" {
  path = "secret/newrelic"

  # proposed flag
  persist_to_state = false
}

provider "newrelic" {
  api_key = data.vault_generic_secret.newrelic.data["newrelic_api_key"]
}

# Other New Relic resources
# ...

If the code looks something like this when using a provisioner, that would be great. And I would agree there is not so much value in a new flag to avoid persisting the secret to the state file. But, so far it seems there is no way to achieve anything similar to this. Do you know any?