Terragrunt: Add an http backend provider that can do encryption

[brikis98]

One of the biggest flaws with Terraform state is that it can contain secrets in plain text. Certain backends, such as s3, support encryption, which makes this slightly less risky, but anyone with access to that S3 bucket will still be able to see all those secrets in plain text. It would be nice if there was a better way to encrypt/decrypt secrets.

Possible solution

It turns out that Terraform supports using an arbitrary HTTP endpoint as a backend. We could update Terragrunt to do the following:

1. When you run Terragrunt, it starts a small HTTP server on localhost (on a configurable port) that implements the backend API.
2. Terragrunt would then run Terraform and configure itself as an HTTP backend.
3. Each time Terragrunt's HTTP servers gets a POST request, it encrypts the state file contents (see encryption options below) and stores the ciphertext (see storage options below).
4. Each time Terragrunt's HTTP server gets a GET request, it fetches the contents (see storage options below) and decrypts them (see encryption options below).

This way, the secrets will only be in plaintext when in memory and being passed to/from Terraform, and encrypted at all other times.

Encryption options

The easiest encryption option would be to use KMS. In the future, support for Vault and other encryption providers could be added.

Since the state files label which information is sensitive, it should be possible to encrypt solely the secrets, and leave the rest of the file unencrypted, making it easier to troubleshoot issues.

Storage options

The easiest storage option is S3. DynamoDB would be used for locking. In the future, support for Consul and other storage backends could be added.

Open questions

The key question here is whether it's worth the time to implement this functionality, or if Terraform is likely to add something like this natively in the near future?

[josh-padnick]

This is a neat idea! It opens up some interesting possibilities. That being said, if we use KMS, I'm not sure how much security benefit we actually gain since the same KMS key that's used to encrypt the Terraform state file via Terragrunt can be used transparently with the S3 Bucket directly. At the end of the day, if you have access to the KMS Key, either you can view the S3 Bucket or manually decrypt the data.

That being said, this limitation really only applies to KMS. Any other "out of band" encryption benefits from this kind of setup.

One other concern is that it could be a pain to maintain compatibility with all the different supported backends. I wish there were some kind of transparent way we could encrypt state and _then_ send it on using the existing state maintenance methods so there's less to maintain.

Finally, this does seem like a natural candidate for inclusion directly in Terraform since encryption/decryption could be treated as just one additional step in the "state management lifecycle". Given that we're no better off with KMS and the concerns above, my vote is not to invest time in this today, but I'm certainly open to other perspectives.

[brikis98]

That being said, if we use KMS, I'm not sure how much security benefit we actually gain since the same KMS key that's used to encrypt the Terraform state file via Terragrunt can be used transparently with the S3 Bucket directly. At the end of the day, if you have access to the KMS Key, either you can view the S3 Bucket or manually decrypt the data.

It's not exactly the same. When you enable the default S3 encryption built in to Terraform, AWS uses the default S3/KMS key. This is accessible to anyone who has access to that S3 bucket and there are multiple ways to get access to that bucket. Moreover, the encryption is transparent, so any time you open the file in that bucket, you see all the secrets in plaintext. With a custom KMS key, you have better control over who can access the key and encrypting/decrypting is an explicit step separate from just opening the file.

That said, I'm not sure it's a big enough difference to warrant the effort now.

Finally, this does seem like a natural candidate for inclusion directly in Terraform since encryption/decryption could be treated as just one additional step in the "state management lifecycle".

Most of the stuff in Terragrunt should be part of Terraform... But the process for getting it there is quite a bit slower.

[jrodonnell]

Given that this original question was posed almost two years ago and this problem still has not been solved by Hashicorp, please excuse me for digging up this old thread and +1-ing it. The corresponding issue to address this problem in the Terraform source repo has been open since 2014(!) so it seems pretty clear (at least to me) that Hashicorp is content with the "just encrypt the remote state bucket" approach and isn't planning on adding this capability anytime soon. The original solution proposed here seems like a great first step and could be another strong differentiator and selling point for people to use Terragrunt - and there are a _lot_ of comments and reactions on that thread in the Terraform repo begging for this functionality...

[binlab]

Great idea! Any news on it?