Dotenv: Document best practices

[ivan-kleshnin]

I struggle a long time to understand this 12-factors point about env-powered configurations.

I've read through all issues of this and the mirror Ruby repos. Through a number of stack overflow threads.

The final picture is still obscure to me. Maybe it's really a wrong direction (if it causes so much confusion and debates between people) or maybe everyone needs a _better explanation_ of the entire workflow.

I surely see a number of benefits in the env approach. Cross-language / cross-platform way of app configuration being the most vivid. But I also see drawbacks and I'm yet to find some comparison table which would ensure me that benefits are bigger in the common case.

Here are the questions from the top of my head:

1) Where to put default .env to copy at installation step? Should we exactly symlink or copy this file?

2) If answer to 1) is copying: do you pass variable name to replace NODE_ENV value in destination .env at installation step?

3) How exactly do you set different .env for testing? At what place it occurs?
There are multiple ways to implement this. Which one is recommended?

4) Does the lack of global fallback values cause practical issues (your personal experience)?
process.env.xxx || <something> is more or less a DRY violation...
Most config systems provide an option to set global fallback value for every config option.
Did anyone ended implementing his own custom solution to cover this case?

5) Does string-only config cause practical issues (your personal experience)?
Most config systems use js files with native types (and highlighting!) available.
Did anyone ended implementing his own custom solution to cover this case?

6) Does the lack of a syntax for _required config_ cause practical issues (your personal experience)?
Most config systems provide this feature throwing exceptions in a "broken environment".
Did anyone ended implementing his own custom solution to cover this case?

7) Configuration in a frontend (client, browser)...

Personally, I find such blurry conventions undermine the whole _"it's more obvious"_ promo declaration. It's hard to call something "superior" when it's just about tradeoffs and personal preferences.
I'd prefer to be wrong though.

[Added]: I already start to discover issues with 4) and 5). Most libraries expect boolean config values to be boolean. So we're going to repeat not only default values but also type conversions (or string comparisons) in every place it will be used...

port = parseInt(process.env.HTTP_PORT) || 80 // in every place
...
port = parseInt(process.env.HTTP_PORT) || 80 // in every place

useEtag = process.env.HTTP_USE_ETAG == "true" ? true || false // in every place
...
useEtag = process.env.HTTP_USE_ETAG == "true" ? true || false // in every place

:disappointed:

[maxbeatty]

Thanks for the questions, Ivan. I'll do my best to answer and maybe a wiki page can come of this thread.

Where to put default .env to copy at installation step?

You should put it in the root of your project. If you run your app as node index.js, your .env file should be in the same directory as index.js. This is customizable with the path option, but by _default_, we look for it in the directory you're executing node from.

Should we exactly symlink or copy this file?

Either works. This is very dependent on your deploy process. Personally, we have used symlinks with a lot of success. We keep multiple releases so we can rollback bad deploys. We symlink the .env file for the project to each release directory. Here's an example directory structure (truncated for clarity):

$ pwd
/home/my-app
$ ls -al
drwxr-xr-x .
drwxr-xr-x ..
-rw-r--r--    .env
drwxr-xr-x release-1/
drwxr-xr-x release-2/
$ ls -al release-2/
drwxr-xr-x .
drwxr-xr-x ..
-rw-r--r--    .env -> ../.env
-rw-r--r--    index.js

do you pass variable name to replace NODE_ENV value in destination .env at installation step?

We set this by hand in the .env file just like any other variable. Preparing the file is, again, very dependent on your deploy process. The sky is the limit for automation.

How exactly do you set different .env for testing? At what place it occurs?
There are multiple ways to implement this. Which one is recommended?

I see an environment as being a combination of software and hardware (and that's a very loose definition). If you're running your app on your laptop, that is one environment. If you're running tests on Travis-CI, that's another environment. If you're running tests on your laptop and want to use a different database for integration tests, I would suggest using a VM or Docker containers to separate. Trying to use the same environment (i.e. your laptop) as different environments (development and testing) is not a best practice.

Does the lack of global fallback values cause practical issues (your personal experience)?

No, having a misconfigured environment (incomplete .env), to me, is at the same level as not having your database running. Your app's environment needs to be prepared to run and that includes a complete .env file.

Does string-only config cause practical issues (your personal experience)?

Nope, I've never had a type issue. Passing string ports to express, hapi, sequelize, and mysql have worked just fine in my experience. I avoid boolean flags and instead opt for "only do this in production"

if (process.env.NODE_ENV == 'production') {
  // turn on something critical
}

If you need to use boolean environmental variables, I'd recommend not setting it in your .env for environments where it would be false. Then, it will evaluate to false because it will not be set on process.env

Does the lack of a syntax for required config cause practical issues (your personal experience)?

I don't understand this question. Will you elaborate?

Configuration in a frontend (client, browser)...

First, be careful you're are not sending sensitive information to the client. Stripe's publishable key is a good example of a environmental-dependent value that you would load on the server and pass to the client.

1. load environment variables on the server
2. render an environmental variable in your server-side HTML in a script tag
3. reference that variable as needed in client code

It's hard to call something "superior"

Please let me know if dotenv has ever referred to itself as "superior", as that is not our intent, so I can correct it. TMTOWTDI when it comes to configuration. I've personally had a lot of success and enjoyment using dotenv. We're always looking to improve but will also stick to our guiding principles.

[ivan-kleshnin]

I'll do my best to answer and maybe a wiki page can come of this thread.

Thank you very much.

We symlink the .env file for the project to each release directory.

I never saw multiple release folders before. It's just build or dist everywhere.
A very brief explanation would help.

Trying to use the same environment (i.e. your laptop) as different environments (development and
testing) is not a best practice.

I agree but with limited time / money budgets we often need to approximate best practices.
If I do a pet project with unknown language / framework I don't want too much "enterprise" buzz in there. So the question remains. Do you think it's a bad practice to have env variables hardcoded in your npm scripts?

// package.json
"scripts": {
  "test": "NODE_ENV=testing mocha ..."
  "build": "NODE_ENV=production gulp bundle ..."
}

Please let me know if dotenv has ever referred to itself as "superior", as that is not our intent, so I can correct it. TMTOWTDI when it comes to configuration. I've personally had a lot of success and enjoyment using dotenv. We're always looking to improve but will also stick to our guiding principles.

I was talking about 12-Factor "manifesto". It's very cited and authoritative source often mentioned as "superior".

Does the lack of a syntax for required config cause practical issues (your personal experience)?

I don't understand this question. Will you elaborate?

In some packages you can declare your expectations about configurations, i.e. list keys which must be present in config file(s). In this case, when you load configuration, library cares to detect "broken" or "partial" configuration. Providing a centralized place for all exceptions caused by misconfigurations.
"SQL_HOST must be defined" is much more readable than "undefined is not a function" :wink:

As I understand you don't need this because of an established approach to configuration management.
"Never use bools", "replace config objects with flat data" and other practical rules, etc.

I mean

sql: {
  host:
  port:
}

can be always represented as

SQL_HOST =
SQL_PORT =

What _looks better_ is a subjective question. But it's obvious that second option is much easier to work with Env approach. So when people imagine how they apply JSON.parse every time they need SQL port they start to blame the approach. When in reality it just requires different way of thinking.

If you need to use boolean environmental variables, I'd recommend not setting it in your .env for environments where it would be false.

Yeah, that's exactly how Bash "behaves". Bash is limited to strings and all it's statements are made for string comparisons. Again, requires a mind shift for some people (including me).

I've personally had a lot of success and enjoyment using dotenv. We're always looking to improve but will also stick to our guiding principles.

It's great to know it works for you, at least. So I will investigate this further. Classic config approach where you keep settings in language files is much more familiar to me but I hope to find Env alternative better by the summary.

[ivan-kleshnin]

Let's start with two assumptions.

1) We don't need additional configuration system over environment variables. This statement must be proved, but let's skip this step for now.
2) package.json replaces makefile. No need to have both.

So package.json:scripts should be a right place to put environment variables.

package.json

{
  "test": "NODE_ENV=testing NODE_CONFIG_DIR=shared/config mocha --opts specs/mocha.opts specs"
}

As it starting to look clunky it may be replaced with

package.json

{
  "test": "source .testing; mocha --opts specs/mocha.opts specs"
}

.testing

export NODE_ENV=testing
export NODE_CONFIG_DIR=shared/config

We've got everything dotenv provides without a single additional line of JS and without an extra dependency... Am I wrong somewhere?

[maxbeatty]

Putting your environmental variables in package.json would mean committing them to your version control system which is what the twelve-factor app is trying to _avoid_. This is the first FAQ in our README.

Having a plaintext .env file is a very lightweight "configuration system." This module bridges the gap between the file system where your variables are stored and the node process where those values are needed. As I've said, there's more than one way to do it. Best of luck on your projects!

[wilmoore]

@ivan-kleshnin Yup, you can approximate what dotenv is doing with source .env or even source .env.test if you like. Those files aren't tied to dotenv or any node package for that matter.

The only caveat is that you "might" run into problems on windows. With windows, you'd need to replace export with set.

If you have no developers working via windows, then you should be all set.

[ivan-kleshnin]

@wilmoore thanks for clarification!

[justindeguzman]

Just to clarify the following question for future readers, which also confused me for a bit (and I think the README only lightly touched on):

"How do you deal with multiple .env files for dev, prod, etc.?"

This article clarified it. It basically says the reason you should only have one .env file is because Heroku, AWS, etc. should be managing your production variables for you. Your own .env file should only be for your local environment.

With that being said, lots of products are encouraging environment variables to be checked in to your repo in some form. Whether or not to do so is really a question about practicality and how much you want to adhere to the 12-factors. See Serverless or CircleCI for example.

[maxbeatty]

TMTOWTDI and your hosting provider having environment variables as a feature is the main reason dotenv fails silently when .env is missing. I would _not_ go so far as to say your .env file is only for local development. The last company I worked for used .env files in many environments for many years without any problems.

I still strongly advise against putting environment variables in version control for security's sake. Your repo is most likely being distributed to developers' laptops, a hosted VCS like GitHub, CI services, and hosting providers. If your source code gets exposed with sensitive data, you could be in for a very bad time.

AWS key? Someone will most likely start a fleet of expensive EC2 instances and you'll be responsible for paying.
Stripe key? Someone can initiate a transfer and empty your account.

If you're changing your table name between environments like the Serverless example, by all means commit that to git. (That's not a great example of an environment variable.) I believe CircleCI's environment variables are more intended to configure the environment and not provide values to your code. See their docs on setting GOPATH for Go.

Maybe I should write an article on all of this 😸

[justindeguzman]

That'd certainly be helpful! There's a lot of conflicting advice about this subject. An article that compares the pros and cons of the different approaches to managing .env files across multiple teams and environments would help a ton of developers.

[dashmug]

With that being said, lots of products are encouraging environment variables to be checked in to your repo in some form. Whether or not to do so is really a question about practicality and how much you want to adhere to the 12-factors. See Serverless or CircleCI for example.

I have yet to see a product that _encourages_ environment variables to be checked in. It's just plain stupid! See this: http://www.theregister.co.uk/2015/01/06/dev_blunder_shows_github_crawling_with_keyslurping_bots/

Putting sensitive environment variables in your repo is never practical and is discouraged. If there's one out of the twelve recommendations in 12-Factor that is the most important, this is it!

Non-sensitive environment variables are fine in version control, but API keys and access keys should be git-ignored.

The page that you reference from _Serverless_ was written like that for brevity. It is assumed that the reader knows how to handle environment variables properly. See Serverless Variables for reference on how to do it.