Polis: Improve and align configuration of apps (proposal: node-convict)

[crkrenn]

All (including @metasoarous, @colinmegill & @urakagi):

I have two questions:

1. Would there be interest and support in migrating toward consolidating the several pol.is configuration files into a more 12 factor compliant form? It would make things easier for both new and old users and make deployments easier and more secure.

Everything in polis.config.js can be moved into a .env file. AWS keys and webpack.config.js variations can be treated with configuration variables as well.

1. Would it be reasonable to have the current system of configuration override the new system for backward compatibility? E.g. if there is a polis.config.js file present, use those settings. Otherwise, use environment variables?

-Chris

PS. I'm happily doing all development in docker and docker-compose now. No more /bin/rm -rf node_modules for me.

PPS. As part of this process, I can modify Dockerfiles to be backward compatible as well.

[urakagi]

I can't answer this well since I don't know about the .env deployment. But
if you're talking about making another config file and read both, there's
no reason you can't in my opinion.

crkrenn [e-mail] 於 2020年3月16日 週一 上午12:52寫道：

All (including @metasoarous https://github.com/metasoarous, @colinmegill
https://github.com/colinmegill & @urakagi https://github.com/urakagi):

I have two questions:

1. Would there be interest and support in migrating toward
   consolidating the several pol.is configuration files into a more 12
   factor compliant form? It would make things easier for both new and old
   users and make deployments easier and more secure.

Everything in polis.config.js can be moved into a .env file. AWS keys and
webpack.config.js variations can be treated with configuration variables
as well.

1. Would it be reasonable to have the current system of configuration
   override the new system for backward compatibility? E.g. if there is a
   polis.config.js file present, use those settings. Otherwise, use
   environment variables?

-Chris

PS. I'm happily doing all development in docker and docker-compose now.
No more /bin/rm -rf node_modules for me.

PPS. As part of this process, I can modify Dockerfiles to be backward
compatible as well.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pol-is/polis-issues/issues/126, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AASCEPBGCA3IDQNEOHF77N3RHUBT7ANCNFSM4LLFLKXQ
.

--
Romulus Urakagi Ts'ai

[crkrenn]

Can you me how you do your configuration? What files do you use? -Chris

From: "蔡仲明 (Romulus Urakagi Tsai)" [e-mail]
Reply-To: pol-is/polis-issues [e-mail]
Date: Sunday, March 15, 2020 at 10:40 PM
To: pol-is/polis-issues [e-mail]
Cc: Ckl [e-mail], Author [e-mail]
Subject: Re: [pol-is/polis-issues] 12 factor configuration and pol-is (#126)

I can't answer this well since I don't know about the .env deployment. But
if you're talking about making another config file and read both, there's
no reason you can't in my opinion.

crkrenn [e-mail] 於 2020年3月16日 週一 上午12:52寫道：

All (including @metasoarous https://github.com/metasoarous, @colinmegill
https://github.com/colinmegill & @urakagi https://github.com/urakagi):

I have two questions:

1. Would there be interest and support in migrating toward
   consolidating the several pol.is configuration files into a more 12
   factor compliant form? It would make things easier for both new and old
   users and make deployments easier and more secure.

Everything in polis.config.js can be moved into a .env file. AWS keys and
webpack.config.js variations can be treated with configuration variables
as well.

1. Would it be reasonable to have the current system of configuration
   override the new system for backward compatibility? E.g. if there is a
   polis.config.js file present, use those settings. Otherwise, use
   environment variables?

-Chris

PS. I'm happily doing all development in docker and docker-compose now.
No more /bin/rm -rf node_modules for me.

PPS. As part of this process, I can modify Dockerfiles to be backward
compatible as well.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pol-is/polis-issues/issues/126, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AASCEPBGCA3IDQNEOHF77N3RHUBT7ANCNFSM4LLFLKXQ
.

--
Romulus Urakagi Ts'ai

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/126#issuecomment-599354770, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5VZB5E776Q32PLK6HUDRHW3UZANCNFSM4LLFLKXQ.

[urakagi]

gulpfile.js will require the config js file.

crkrenn [e-mail] 於 2020年3月16日 週一 下午10:29寫道：

Can you me how you do your configuration? What files do you use? -Chris

From: "蔡仲明 (Romulus Urakagi Tsai)" [e-mail]
Reply-To: pol-is/polis-issues [e-mail]
Date: Sunday, March 15, 2020 at 10:40 PM
To: pol-is/polis-issues [e-mail]
Cc: Ckl [e-mail], Author [e-mail]
Subject: Re: [pol-is/polis-issues] 12 factor configuration and pol-is
(#126)

I can't answer this well since I don't know about the .env deployment. But
if you're talking about making another config file and read both, there's
no reason you can't in my opinion.

crkrenn [e-mail] 於 2020年3月16日 週一 上午12:52寫道：

All (including @metasoarous https://github.com/metasoarous,
@colinmegill
https://github.com/colinmegill & @urakagi https://github.com/urakagi):

>

I have two questions:

1. Would there be interest and support in migrating toward
   consolidating the several pol.is configuration files into a more 12
   factor compliant form? It would make things easier for both new and old
   users and make deployments easier and more secure.

Everything in polis.config.js can be moved into a .env file. AWS keys
and
webpack.config.js variations can be treated with configuration variables
as well.

1. Would it be reasonable to have the current system of configuration
   override the new system for backward compatibility? E.g. if there is a
   polis.config.js file present, use those settings. Otherwise, use
   environment variables?

-Chris

PS. I'm happily doing all development in docker and docker-compose now.
No more /bin/rm -rf node_modules for me.

PPS. As part of this process, I can modify Dockerfiles to be backward
compatible as well.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pol-is/polis-issues/issues/126, or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AASCEPBGCA3IDQNEOHF77N3RHUBT7ANCNFSM4LLFLKXQ>

.

--
Romulus Urakagi Ts'ai

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<
https://github.com/pol-is/polis-issues/issues/126#issuecomment-599354770>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/ABOJ5VZB5E776Q32PLK6HUDRHW3UZANCNFSM4LLFLKXQ>.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pol-is/polis-issues/issues/126#issuecomment-599567665,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AASCEPCVF2LSKDTVCOJWIWTRHYZT3ANCNFSM4LLFLKXQ
.

--
Romulus Urakagi Ts'ai

[metasoarous]

Thanks for bringing this up @crkrenn

Environment variables are not actually a particularly secure method for conveying sensitive information to the application. Other processes running on the system can potentially get access to these environment variables, thus making them less safe than a config file with the appropriate Unix permissions.

Some parts of our system do still use environment variables, but we'd like to move these to centralized configuration files.

I'm not completely shutting down support for also supporting environment variable configuration, but I'd rather we focus on supporting consistent config file support as a priority.

Please let me know what you think about this.

Thanks!

[crkrenn]

I’m open to other solutions. I actually was going to implement a private polis-secrets git repo.

How about a yaml file? Config should be accessible to node, clojure, python, etc.

The current mix between .env and .polis.config is confusing.

https://gist.github.com/telent/9742059

https://github.com/hashicorp/vault/blob/master/README.md

Redis was also mentioned.

What other options are there? What do other open source platforms with multiple languages use?

Chris

---

Sent from Workspace ONE Boxerhttps://whatisworkspaceone.com/boxer

On March 31, 2020 at 4:33:10 PM PDT, Christopher Small [e-mail] wrote:

Thanks for bringing this up @crkrennhttps://github.com/crkrenn

Environment variables are not actually a particularly secure method for conveying sensitive information to the application. Other processes running on the system can potentially get access to these environment variables, thus making them less safe than a config file with the appropriate Unix permissions.

Some parts of our system do still use environment variables, but we'd like to move these to centralized configuration files.

I'm not completely shutting down support for also supporting environment variable configuration, but I'd rather we focus on supporting consistent config file support as a priority.

Please let me know what you think about this.

Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/126#issuecomment-606940402, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V5Z6YFBFNJ5IY5YMPDRKJ4SHANCNFSM4LLFLKXQ.

[crkrenn]

There also is the question of what is needed for build (aws tokens, etc.) and run (whitelists, ports, facebook creds, database URL and creds).

From: Christopher Small [e-mail]
Reply-To: pol-is/polis-issues [e-mail]
Date: Tuesday, March 31, 2020 at 4:33 PM
To: pol-is/polis-issues [e-mail]
Cc: Ckl [e-mail], Mention [e-mail]
Subject: Re: [pol-is/polis-issues] 12 factor configuration and pol-is (#126)

Thanks for bringing this up @crkrennhttps://github.com/crkrenn

Environment variables are not actually a particularly secure method for conveying sensitive information to the application. Other processes running on the system can potentially get access to these environment variables, thus making them less safe than a config file with the appropriate Unix permissions.

Some parts of our system do still use environment variables, but we'd like to move these to centralized configuration files.

I'm not completely shutting down support for also supporting environment variable configuration, but I'd rather we focus on supporting consistent config file support as a priority.

Please let me know what you think about this.

Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/126#issuecomment-606940402, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V5Z6YFBFNJ5IY5YMPDRKJ4SHANCNFSM4LLFLKXQ.

[metasoarous]

The current mix between .env and .polis.config is confusing.

:100: % AGGREE!

We should definitely move towards static files to unify things.

For some added context on this decision: If you look at Postgres docs, they explicitly suggest _not_ putting your password in environment variables: https://www.postgresql.org/docs/9.3/libpq-envars.html. So part of the point here is that if it ends up being better for _some_ of the credentials to be in a file, you may as well have _all_ of them there.

Right now we support json, but I'd be fine with supporting yaml or edn where appropriate. Easy enough to introduce flexibility in that regard (e.g. https://github.com/shaunxcode/jsedn).

[patcon]

Would this be as good a place as any to be the "revamp config" issue?

@crkrenn if so, you down to rename to something like "Improve and align configuration of apps"? (@metasoarous being able to retitle and consolidate tickets is helpful to issue queue gardening fwiw)

Did some research.

Top candidate for future dual support of envvar + file config conversation: https://github.com/mozilla/node-convict

It's the successor of nconf: https://github.com/indexzero/nconf

Alternative, but not my pref: https://github.com/dominictarr/rc

[crkrenn]

Node-convict looks great.

Precendence order

When merging configuration values from different sources, Convict follows precedence rules. The order, from lowest to highest, is:

1. Default value
2. File (config.loadFile())
3. Environment variables (only used when env property is set in schema; can be overridden using the env option of the convict function)
4. Command line arguments (only used when arg property is set in schema; can be overridden using the args option of the convict function)
5. Set and load calls (config.set() and config.load())

https://medium.com/@sherryhsu/managing-configurations-in-node-js-apps-with-dotenv-and-convict-d74070d37373

[patcon]

LGTM! 1-3 are my wishlist, but CLI args sounds neat too

thanks for rename! 🙏

[crkrenn]

@metasoarous, (cc: @patcon)

I made some progress with cloud deployment and am now shifting back to configuration.

I searched for clojure configuration management packages, but couldn't find any that 1) did not use environment variables, and 2) were compatible with node-convict.

Could you help writing clojure functions that would read a set of json files at math startup and stash the data in a way that would be accessible to all of the other math clojure functions that need them?

For simplicity, you could overwrite the current env variables.

For bonus points, you could make the functions backward compatible with env variables, so that env variables would override settings in the json files.

Or, could you recommend someone more fluent that I in clojure who might be willing to take this on?

-Chris