Salt: Managing secrets / secure data with Saltstack in a bigger team

[andygrunwald]

In your best practice guide there is a chapter Storing secure data.
In this chapter you recommend to store passwords / secrets / secure data in pillars.
The basic concept for this is fine. Only one server got access to the needed passwords. That is great.

But i see several disadvantages or challenges (we got no problems here ;)) with this:

Disadvantages / Challenges to tackle

Versioning of pillars

You are able to put pillar data into files.
That is great.
First we thought "Great! Lets put them under version control as well like the states, custom modules and so on. Move all the things into Git and lets use the git filesystem backend".
This sounds still like a great idea. But then you will store the passwords, that are plain text in a pillar file, in Git as well.
And you know VCS + Secrets ... there are better solutions.

We store several other server related data in pillars as well. So passwords are only one part of it.
Okay, one idea would be to create a structure like pillar/password/mysql.sls, pillar/password/redis.sls, pillar/other-data/server.sls, and so on
And set the pillar/password folder to .gitignore.

This is a solution, but this depends on human discipline and doesn`t avoid faults.
A person can "check in" passwords by not storing them in the pillar/password/* folder.

Still plaintext

Even if the password / secret is stored in a pillar, it is still plain text.
This means the diffs contain the passwords as well.

Imagine you got a config file that access a password stored in a pillar with the jinja template engine.
During a deploy of a config change (through a state file) the file will be compiled and at the end of the operation, a diff with removed + added lines will be shown.
In this operation you see all passwords / secrets in plain text.

And think further: If you use a job cache / returner that moves all result in a 3rd party system (e.g. kafka / redis) to process them further, you transfer you passwords through your infrastructure.
I think this is not the best / a good way.

Big / Interdisciplinary teams

Both points mentioned above will get a real challenge if you work with a bigger / interdisciplinary team.
In those teams your try to be as transparent as possible, but keep a minimum amount of security alive.
If you want to enable all people to modify your sls scripts, pillars, etc. by e.g. Pull Requests, the plain text passwords / secrets will be a problem.

Ideas

Encrypted data bags (like Chef)

Chef introduced encrypted data bags for this kind of problem. Maybe this or a similar solution would be possible as well?

Support for 3rd party secret management

Some 3rd party tools try to tackle this problem.
One of them is Vault by HashiCorp.
Maybe a 3rd party integration / layer for those tools is a idea.

Know / flag your secrets

Another idea would be to "mark" / "flag" secrets in pillars.
With this the salt engine / jinja is able to know if the current data is a secret and can treat them differently. Like mask them with "**" instead of outputting them in plaintext.

Various

Don't get me wrong. I don't want to criticize the system, but i think this area can be improved.
I am open for feedback and want to get your opinion about this.
Thank you

[andygrunwald]

Settings like "pillar_safe_render_error" (see Master-Configuration and Master Provided Pillar Error) are looking good and are parts of the right direction.

[ryanwalder]

Salt supports GPG as a renderer which may be what you're after.

http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html

[jfindlay]

@andygrunwald, thanks for bringing up these issues. I am sure we could document pillar strategies for scaling secret data. @whiteinge and @UtahDave will have better responses here than I.

[whiteinge]

All good points and questions. You're very right that this area can be improved. Pillar is purposefully open-ended and that often means you must architect the dev workflow for your team. An opinionated doc specifically geared for large team environments would be particularly helpful in providing direction that is likely to fit most teams.

Storing sensitive data in flat files has caveats, as you've pointed out. It is very common to move that data out of flat files in large environments where many people across many teams need access to either see or update that data. An external and centralized data store that has it's own access controls is not an uncommon choice. There's quite a few options for this in Salt. See below:

This is a solution, but this depends on human discipline and doesn`t avoid faults.

Salt's sdb system is not overly well-known and can be a good way of separating particularly sensitive Pillar data from other Pillar data that does still come from flat files. Depending on the backend, it can even enforce that certain sensitive data go into a specific location. People viewing the processed Pillar data see the sdb:// URI instead of the actual data.

http://docs.saltstack.com/en/latest/topics/sdb/
http://docs.saltstack.com/en/latest/ref/sdb/all/index.html

This means the diffs contain the passwords as well.

Turn this off for sensitive data with the show_diff flag to file.managed.

Support for 3rd party secret management

The external Pillar system provides first-class access to third-party tools. It would be great to have a Vault module as well.

http://docs.saltstack.com/en/latest/ref/pillar/all/

I hope something there gives you an idea of where you'd like to go with your team.

[jfindlay]

I'm labeling this with Documentation so we can be sure to remember to add some docs on the issues and ideas @andygrunwald and @whiteinge have discussed.

[DanyC97]

@andygrunwald good comments, i'm in the same place like you and considering to use Vault hence in few weeks time we'll start a PoC and see how it goes.

[foundatron]

This issue reflects a lot of the issues i've been struggling to solve these past couple days. Pillars in a s3 bucket that only the master has access to, and everything else in gitfs seems to be the road i'm going down...though thats a lot of code not in version control... which feels wrong.

[DanyC97]

@foundatron but the Pillars data contains config & secrets data all in same bucket correct? You don't differentiate i guess..

[andygrunwald]

@foundatron Where do you put the s3 secrets / credentials?

[vitalyisaev2]

We really have an urgent need in this feature

[mastrolinux]

This is indeed a solved problem using https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html and external pillars (you can have a db storing pillars and username/pass encripted via GPG). I am suprised it is not mentioned in the best practice document here: https://docs.saltstack.com/en/latest/topics/best_practices.html

[foundatron]

@andygrunwald we use AWS IAM Roles / Instance profiles to grant instances permissions to various AWS resources.

This of course only works for instances launched in AWS.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.