Terraform-provider-azurerm: azurerm_key_vault_secret as a data source?

[tevert]

We'd like to use secrets managed in an Azure Key Vault to provision the rest of our infrastructure. For example, database connection info. My instinct is that the "most correct" way to do that is to retrieve the secrets via a data source so they can be passed around within the HCL.

As this data source doesn't exist yet, we currently work around it by using az commands to retrieve the secrets and pass them in as command line args, all in a wrapper script. This means that we have extra code to maintain and keep secure, and as our hierarchy of modules grows this will get in the way.

Does this idea break any terraform conventions or philosophies? Or would this secrets data source be valuable to others?

[tombuildsstuff]

hey @tevert

Thanks for opening this issue :)

I've been thinking about this over the last few days - and whilst I could see it definitely being useful this needs some investigation to tell how to proceed. The recommendation within Azure (AFAIK at least) is to pass in the Key Vault Secret ID (which generally isn't exposed in the Azure SDK/Terraform at the moment) - directly a resource and let Azure pull it out as needed.

By making Key Vault Secret either a Data Source or a Resource - the value of the Secret will be stored in the State (since this is how it's referenced) - which isn't necessarily ideal. Whilst I have no objection to adding a Data Source for this - I feel we need to do some investigation on how is best to move forward, either by exposing the Key Vault Secret ID fields on the relevant Resources (and not storing the Secret Values in the state - however this requires SDK changes) - or by just exposing the values in a Data Source (which as mentioned above isn't necessarily ideal)

I've opened an issue about this on the Azure SDK for Go (which we use to interact with Azure) about this to see about the Key Vault Secret ID fields being added. Once we've got an answer to that we can proceed further :)

Thanks!

[tevert]

I hadn't considered the state file... while we keep our shared state file in a fairly well protected location anyway, it might be an unexpected security gotcha for others that aren't thinking about it. With that in mind it would probably be better to follow Azure's pattern of letting the ARM API pull secrets, though it will likely end up being more work to expose the vault parameter set in every relevant place.

Thanks for following up on this!

[glenjamin]

I would like to :+1: on this - we currently are using the external data source provider to do this by shelling out.

There are a few azure resources where we could do with feeding them a "secret" which we don't want to check in to source control, but are happy enough for it to exist in the resource config. One example is a 12 factor app deployed to azure_app_service where we don't want to have to build-in custom logic to load values out of key-vault (and even if we did, we'd need to give it key-vault credentials somehow anyway!)

This is effectively the same workflow as the AWS parameter store API: https://www.terraform.io/docs/providers/aws/d/ssm_parameter.html

[glenjamin]

While it would be nice to automatically have the ARM API pick values from the keyvault without having them go through the client, it would still be useful to have a data source for when it is desirable to use the secret in other targets.

Is there anything blocking creation of a data source for the key vault secrets themselves?

[pixelicous]

@glenjamin how do you go about to an external provider and keep the credentials you are running terraform with? using an env variable? isn't that locked per process?

[glenjamin]

We’ve done this in 2 ways

1) use env vars to provide auth to terraform, these are inherited by subprocesses

2) we switched to using CLI auth via az to authorize terraform, so now we shell out to az to read from the keyvault as well

[pixelicous]

@tombuildsstuff - it would be great to have this, sometimes you dont need the value of the secret but want to pass its URI, we can't do it currently..

@glenjamin does that mean that you go out of terraform and then execute terraform again, or if you use it as external provider and coming from AZ already then it means you can go back to its session? thanks for your prompt reply

[glenjamin]

The latter. We know that az has to be logged in for terraform to be running, so we can expect that when we shell out the session will be re-used.

Here's the code which does it:

https://github.com/ministryofjustice/digital-studio-infra/blob/master/tools/keyvault-data-cli-auth.py

And an example of it in-use:
https://github.com/ministryofjustice/digital-studio-infra/blob/7ac5b6979cb050734c3e9b2679db65ee56a19621/licences/stage/ui.tf#L43-L52

[pixelicous]

@tombuildsstuff is there any chance to add this functionality? i have another use case, when creating a VM i need to specify the url of the certificate that was created in a keyvault.. there is the option of creating secrets/certificates in terraform, but referncing them later on is a problem, unless there is a way to do it that i do not know about

Referencing an issue i have - #1067

[tombuildsstuff]

👋 hey all

I spent some time thinking about how we could achieve this on Friday and have opened #1202 which includes a data source that exposes the value of a given Key Vault Secret. Whilst this isn't perfect (since the secret value is still being stored in the state) - I think this is preferable compared to threading a key_vault_secret_id property onto every resource which could need this. This also needs some extensive testing before we proceed any further - but we'll take a look later this/next week.

Thanks!

[glenjamin]

This looks sensible to me, i’ll see if I can get a prerelease build and try converting some of our existing setups to use this

[Supermathie]

Another use case for this is that you cannot ask an Application Gateway to ask the Key Vault for a certificate - AG certificates MUST be in your state.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!