Terraform: Request: Enable simple/native way to read .env files

[MagnusBrzenk]

Current Terraform Version

Terraform v0.12.19

Use-cases

It's standard for code bases to have .env files, but I am not aware of any way to read their contents into terraform scripts.

Attempted Solutions

There is the option for terraform.tfvars, but this requires my code base to maintain two secret-variable files (and their templates).

Proposal

At minimum, instead of requiring that the -var-file flag take only files named exactly terraform.tfvars, terraform could be tweaked to allow for .env files to be specified. I'd also prefer to not have to also declare the variable name in the terraform script.

[apparentlymart]

Hi @MagnusBrzenk!

Could you provide a little more information about exactly what file format you mean when you say ".env files"? If you know a link to the documentation for the file format you're talking about, that would be helpful to understand what it would take to include a parser for it in Terraform.

[MagnusBrzenk]

I am not aware of any official standard for .env files. The idea of a .env file is that it stores variables that will be made available to a (linux?) shell when sourced. So, in theory, the syntax of a .env file will be such as to not error when sourcing from e.g. a bash shell. E.g.:

# Comment
aaa=AAA
export bbb=BBB

However, my impression is that, in practice, the acceptable syntax for a .env file just depends on the conventions chosen by the most popular/robust library used by the community for a given language. For example, in the world of JS, it is the dotenv library, which spells out its understanding of .env-file conventions with the following rules (which are more permissive/forgiving than what sourcing with bash would allow for):

• BASIC=basic becomes {BASIC: 'basic'}
• empty lines are skipped
• lines beginning with # are treated as comments
• empty values become empty strings (EMPTY= becomes {EMPTY: ''})
• inner quotes are maintained (think JSON) (JSON={"foo": "bar"} becomes {JSON:"{\"foo\": \"bar\"}"})
• whitespace is removed from both ends of unquoted values (see more on trim) (FOO= some value becomes {FOO: 'some value'})
• single and double quoted values are escaped (SINGLE_QUOTE='quoted' becomes {SINGLE_QUOTE: "quoted"})
• single and double quoted values maintain whitespace from both ends (FOO=" some value " becomes {FOO: ' some value '})

• double quoted values expand new lines (MULTILINE="new\nline" becomes
  {MULTILINE: 'new line'}

• Note: It's not mentioned in dotenv's documentation, but it also allows for the export keyword in .env files; e.g.: export xxx = 'XXX')

So if you were to enable direct terraform sourcing of .env files, then I think you would simply have to decide for yourselves what rules make sense given the constraints of terraform data structures, etc.

(BTW. I see now that you can import variables from .env files by using the format export TR_VAR_foo=bar, sourcing that .env file, then running terraform on a terraform script that declares the variable foo. This will do for me since I run Terraform from a bash script anyway.)

[apparentlymart]

Thanks for the extra context, @MagnusBrzenk!

In the interests of gathering as much context as possible: you mentioned that you're currently having to maintain two different definitions of environment variables, so I assume that means you already have something else in your system that understands this file format you're describing and so compatibility with that other system would be the most important thing to meet your use-case. Given that, are you able to identify that system so we can use it as an example?

I don't mean to say that we'd intentionally support only that system, but given the fuzziness of this set of requirements right now I think it's helpful to gather as much data as possible to inform a possible future implementation.

Thanks!

[MagnusBrzenk]

Sure. For example, when I make front-end applications involving static assets (html, css, js), I always have a .env file in order to, at minimum, build and deploy those assets to github pages. Rather than hardcoding my username and repo in the code, I pull them out into a .env file. Here is an example repo and its .env file. And if my web app needed 3rd-party keys, backend credentials, etc., I'd likewise put all of them into a .env file. With front-end assets, I typically read these variables at build-time (via sourcing in bash script), and with the backend, on a conventional server, it would read a remote edited copy of the .env file at run-time (using the dotenv library in the case of a nodejs server). In general, anything that is account-specific or location-specific (passwords, file paths, API endpoints, etc.), I'd want to pull out into a .env file. My impression is that this is standard practice certainly in the world of JS.

My motivation for terraform came from wanting to start supplementing my web apps with AWS products (viz. connecting a front-end html form to a AWS simple-email-service triggered via a lambda function), which is why I wanted to be able to simply extend my existing web app with a terraform script while continuing to add the extra required credentials (e.g. destination email address) to my trusty old .env file.

Hope that helps.

[apparentlymart]

Thanks for all that extra context, @MagnusBrzenk! It's definitely helpful.

With that said, it seems like the main requirement here is to identify a subset of standard POSIX shell syntax that is sufficient to get standard shells to export a set of variables when it's executed as a script, and that this hypothetical Terraform-side parser can reliably interpret and get identical names and values to what a standard shell would place in the environment if executing that input as a shell script. To keep it clear what is and is not supported, I expect it would return an error if it encountered anything outside of its expected subset, so the user can adjust the input for correct interpretation.

In particular, it seems like a _non-requirement_ to support arbitrary shell scripts and fully evaluate them the same way a shell would, which is definitely a relief. :relieved:

---

The npm package dotenv does look like it offers a good starting point for a set of processing rules, which is great. Thanks for sharing that! Some of the rules in that list seem like they deviate slightly from shell processing, and as you mentioned there are some other things to consider like the export keyword. For that reason, I think before adding a function like this and being bound to compatibility with its behavior it would be responsible for us to do a little more research just to make sure we're producing something folks can depend on and not be surprised by. In particular:

• Verify each of the rules the dotenv package follows and compare them to standard POSIX shell behavior. In case of any differences, we'll have to make a tradeoff about whether to prefer to be compatible with POSIX shell or with the dotenv package.
• Find some equivalent libraries to dotenv from other language ecosystems and see how they compare to one another. Ideally we'd want to be able to find a subset that is mutually-parsable by all commonly-used libraries _and_ POSIX shells _and_ Terraform, but I suspect we'll find some more compatibility tradeoffs to make here too, due to this not being a strictly-defined format.

Once we ship something for this it will be hard to make changes it its behavior without breaking compatibility; having it suddenly start returning a new result would likely cause aws_lambda_function resources to get unwanted updates, for example. Therefore I want to be cautious about not jumping into something without making a plan with explicit compatibility tradeoffs first. Then we can document the tradeoffs we've made in the function docs so that it's clear to users what they can expect it to support and what rough edges they might encounter with particular combinations of parsers.

Unfortunately our attention is elsewhere at the moment, so I don't expect someone on the Terraform Team at HashiCorp will be able to undertake that research and make a detailed proposal in the near future. I do like the idea of it though, and if we can find a practical way to specify it to avoid creating maintenance/compatibility problems in the future I can definitely see this being useful! If anyone else has a similar use-case, I'd encourage adding a :+1: reaction to the initial proposal comment above; issue reaction votes are one of the inputs we use to prioritize work.

---

In the meantime, for those who have this need and want to solve it with Terraform as it exists today, I think probably the easiest answer is to use the external data source with a NodeJS program that uses dotenv to parse the file and JSON.stringify to produce a JSON representation that the external provider can consume. Something like this, perhaps:

const env = require('dotenv').config();
if (result.error) {
  throw result.error;
}
console.log(JSON.stringify(env));

data "external" "env" {
  program = ["node", "${path.module}/read-env-json.js"]
}

output "environment" {
  value = external.env.result
}

[techdragon]

Having lost a day down this rabbit hole before finding this issue (".env" is very hard to search for), I'd like to share some extra information.

The most popular libraries for this appear to be the already mentioned javascript library, as well as these:

• https://github.com/theskumar/python-dotenv for Python
• https://github.com/bkeepers/dotenv for Ruby
• https://github.com/joho/godotenv for Go (which is MIT licensed and usable as a library by other Go programs)

@apparentlymart If you're interested in identifying the "minimal subset", I would also _highly_ recommend this implementation in shell https://github.com/madcoda/dotenv-shell which is extremely well broken down and is honestly a great example of "parsing simple things with shell scripts"

Also. its worth mentioning that if this is treated as a "naive" loading mechanism and terraform were to treat "environment variables from .env files" the same as "environment variables from the parent shell", it would somewhat defeat the purpose of using a .env file because as documented https://www.terraform.io/docs/configuration/variables.html#environment-variables terraform expects environment variables to be prefixed with TF_VAR_ so people using a .env file would still have to duplicate their configuration information, just inside the .env file this time, once with the TF_VAR_ prefix and once without.

[beanaroo]

Would this capability not first depend on ${env.VAR_NAME} to be implemented so that the loaded .env values can be referenced?

As @techdragon pointed out, terraform currently expects variables to be prefixed with TF_VAR_.

From https://github.com/hashicorp/terraform/pull/1621:

In the future, we'd still like to add ${env.NAME} syntax to the configuration, but the way we'd like to do that represents significantly more engineering effort. This is a nice stop-gap solution to support env vars for configuration that is simple to implement. And it isn't mutually exclusive to env.NAME interpolations: we'll keep both when we implement them.

Is this still being tracked somewhere? I can't find any relevant issue apart from the, now closed, https://github.com/hashicorp/terraform/issues/62.

I feel that both these would be beneficial as .env is great of local development and in CI pipelines we rely heavily on both predefined and project/group level variables.

[MagnusBrzenk]

FYI -- here's the approach I've been taking with terraform "as is" that satisfies me.

https://stackoverflow.com/a/60870365/8620332

[jrhouston]

A very similar request was made in https://github.com/hashicorp/terraform-provider-kubernetes/issues/889.

We have a couple of Kubernetes resources (config_map and secret) where it makes sense to read the data field directly from a dotenv. I proposed a hacky workaround where you can parse a dotenv file (with some gotchas) using the existing functions built into Terraform, but I'd also like to propose that we add a function to help with this.

I've opened https://github.com/hashicorp/terraform/pull/25433 to illustrate, using the direnv/go-dotenv package.

[apparentlymart]

To answer some of the questions since my last comment, I want to clarify that I am not expecting that Terraform will do any automatic processing of environment files, but rather that we might consider offering a new built in function -- similar to the existing jsondecode, yamldecode, csvdecode functions -- so that you can write a configuration that will explicitly read settings from a file and take some action based on them, such as populating the set of environment variables associated with an AWS Lambda function.

Terraform automatically setting variable values from any other location than it already reads is not in scope for this issue, and so if you'd like to talk about that I'd suggest to open a new issue. (though since there are already arguably "too many" ways to provide variables to a root Terraform module, the bar for adding another is likely to be very high.)

Aside from those things, we've not done any additional research on this in the meantime and so I don't have anything else to add. I do appreciate you all sharing additional examples of parsers which could help inform the behavior of a hypothetical new function. As I noted before, if you have a use-case that could benefit from a function that could read a file of this sort I'd encourage you to add a :+1: reaction to the original comment of this issue, which can influence its prioritization in relation to other work.

[rayjlinden]

@apparentlymart asked about a system that uses .env files they might currently have to maintain. I am surprised no one mentioned docker. (Well k8 is close.). But right now we have our dev environment using docker-compose with .env files. We are moving to terraform everywhere else. I recently added terraform to our "dev" environment to configure Localstack which launch as another container docker-compose. All of our other services there use .env files. I found this thread looking for a way to get those files into our lambda definitions to load in Localstack.

So +1 on needing a dotenvdecode utility.