Salt: Canonical solution for securely transferring data between minions

[Nick2253]

Description of Question

I've been trying to determine the canonical solution to the following problem:

• Generate secure data on a first minion.
• Pass the secured data from the first minion to a second minion.

An example of this process is generating a Let's Encrypt certificate from a web-accessible client (first minion) for a web-inaccessible client (second minion), and passing the certificate and key to the second minion. This particular example will become even more important once Let's Encrypt allows wildcard certs at the end of February.

Another example of this process is generating an Icinga2 client ticket. When a new minion (the second minion) is added to the network, the Icinga2 master (the first minion) generates a ticket, and the second minion needs that ticket to complete client setup.

A further example of this process is a username/password. Many legacy services are only accessible via usernames/passwords, and they have requirements for regular password changes (usually for regulatory reasons). These services can be configured to automatically generate a new password (on the first minion), and these passwords need to be send to clients that need them (the second minion).

Possible Solutions, and Problems

I have seen a few possible solutions discussed on the internet, but no solution seems to provide all these features:

• Handled entirely by Salt
• Secure the data so that only the master and the two minions know about it

Generally speaking, the first part of the problem is easily done in Salt. Whether with a custom method, a command, etc, it's fairly easy to either create a file, or even a pillar file, containing the secure data on the first minion. The challenge is the second part: moving the file securely using Salt.

I'm assuming that the best way to handle this is transfer the file from the first minion to the master, and then from the master to the second minion. This assumption is also shared with some of the solutions I've seen below.

cp.push

As suggested in #31863

The general workflow is to use cp.push to send the file from the first minion to the master. However, cp.push uses salt.transport.Channel.send(), which is not guaranteed to be confidential.

If we could add a "secure" flag to the cp.push method to use the crypted_transfer_decode_dictentry, that would enable a means to securely pass the files to the master.

event.send

As suggested in #27156

The general workflow is to generate the secure data on the first minion, and then trigger an event, passing the secure data from the first minion to the master. A reactor is triggered on the event, and applies the data to the second minion.

However, this method has the same problems as cp.push. All data passed in events, including pillar data, uses the .send() transport.

Salt Mine

As suggested in #27156, #44564

The general workflow is to create a mine function that takes the secure data from the first minion. Then, the data can be used by the second minion.

However (other then the general inapplicability of a salt mine for rarely changing data), the Salt Mine's permission schemes do not support the ability for one minion's data to be selectively parsed out to other minions.

Concluding Thoughts

I've been trying to research these options, diving into the Salt code for the last couple weeks. However, I've come up empty handed. Looking at how this is handled in other configuration management systems, to me, is straightforward. I can't imagine that this task is impossible in Salt, since everything else I've seen tells me that Salt is very powerful. I can only assume that I'm making some deep-seated assumption that is creating an implied solution, and forcing me into a narrow box.

My goal here is to either (1) learn what I'm missing, and help produce documentation for other new users in my shoes, or (2) bring this issue to the attention of the Salt team, and hopefully contribute to a solution to this problem.

[gtmanfred]

Have you looked at vault + sdb?

I would generate the certificate, and store it in sdb, and then put it under a role so that only certain minions can directly access it.

You can also use something like https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.sdb.html#salt.modules.sdb.get_or_set_hash to generate a shared password between a bunch of minions, so they can all access it.

Another option would be to store the certificate in an external data base pillar, and then use regular salt states to set the pillar data by controlling the database directly.

I think both of those options fulfill your need to move it securely, and that is how I would do it so that it does not get put somewhere where other minions can't see it when it is being set.

Vault + SDB is probably my perferred, because there is already an sdb.set () function where you can just put the certificate there, instead of having to try and write custom states to query a database and insert data.

I also know that there are people out there right now using vault for this purpose with sdb.

[Nick2253]

Have you looked at vault + sdb?

I've looked at Vault, and it definitely is the direction I'd go if you needed to manage lots of credentials, but that's another piece of third-party software to manage. The goal was to do this all with Salt. If you have to manage separate credentials and ACLs, that defeats the purpose, and you might as well just run a secure file host (like git) on the first minion.

Another option would be to store the certificate in an external data base pillar, and then use regular salt states to set the pillar data by controlling the database directly.

Again, third-party software, and another ACL.

I think both of those options fulfill your need to move it securely, and that is how I would do it so that it does not get put somewhere where other minions can't see it when it is being set.

But the whole point is that we _want_ to put it somewhere where other minions can't see it.

[gtmanfred]

That is the point of sdb though, to be able to put secure data where other minions can see and access it.

If you have to use a salt system, you could use publish.publish and allow only specific minions to run commands on the first minion to get the data back, but that puts a lot more load on the salt master server.

This is one of the usecases that sdb was created for.

[Nick2253]

Perhaps I'm missing the point then.

I dug into the SDB documentation, and I see that I can use (for example) a YAML file stored on the master, which I feel would meet my "do this all with Salt" criteria. However, I'm not sure how to manage the permissions/ACLs for this data. The documentation for SDB does not indicate any means of setting any kind of limits on access.

publish.publish

Publish also uses the .send() transport method, and so is not a secure way of passing data from the first minion to the master.

[gtmanfred]

The vault sdb uses the vault execution module can have acls to allow only certain minions access to the data.

https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.vault.html

saltstack/minion/{minion}

And then in vault you would only allow each minions role access to whatever data, and then only allow them to read it, and then your minion that generates the variable would have access to pull the data.

There are several other modules that could be used but vault is by far the most used, specifically because it is meant as a secret store.

https://docs.saltstack.com/en/latest/ref/sdb/all/index.html

[Nick2253]

Is it therefore correct to say that there is no built-in method to securely transfer data from a minion to the master (such that no other minion can snoop on this data)? Obviously, Salt has built-in methods to securely transfer files via third-party tools/services.

With this in mind, I'm trying to understand the Vault solution, since it sounds like that's the easiest solution that will work for me. However, I'm struggling with the "policies" part. If I wanted to create a policy that allows all minions access to read a path of the form salt/<minion id>/secure-data, how would that work? I see how I could use the {minion} template to match a minion to a policy, but then it seems like I would need to create a different policy for each minion. From the Vault documentation, the only matching seems to be the * as a glob at the end of the path.

[gtmanfred]

@Nick2253 there is a zmq_filtering option, which should make it so that only the targeted minion can actually see the messages.

Also, can you show me how it other minions would be able to see the information that is sent to the salt master to would be able to be seen by other minions?

Pillars especially, because no other minion should be able to see that data, except for the minion it is sent to.

[Nick2253]

Also, can you show me how it other minions would be able to see the information that is sent to the salt master to would be able to be seen by other minions?

The grammar is kinda broken in this sentence, so I'm not exactly sure what you're asking. I'm assuming you're asking "how can other minions see the data that a first salt minion sends to the master?".

To answer that, I'd refer you to the Salt documentation on the request channel:

The req channels have two basic interfaces when talking to the master. send is the basic method that guarantees the message is encrypted at least so that only minions attached to the same master can read it-- but no guarantee of minion-master confidentiality, whereas the crypted_transfer_decode_dictentry method does guarantee minion-master confidentiality.

By diving into the code for the different methods, you can see that they ultimately use the relatively insecure send method to move data to the master.

I'm not suggesting that the data from the master to the minions (including any pillar data) is insecure.

[github-abcde]

I have local patches to the salt mine to allow for minion-pushed ACL, where the mine.send- and mine.update-functions get two additional arguments: allow_tgt and allow_tgt_type, similar to the tgt and tgt_type arguments in https://docs.saltstack.com/en/latest/ref/states/all/salt.states.saltmod.html.
The master saves these with the mine value (which does cause the structure of mine-items to change from a simple key-value to key-dict-with-acl-and-value) and, on request, only returns the value when the requesting minion matches the allow_tgt.
However, I haven't verified that this modification is totally secure, so I haven't created a PR for this yet.

[mirceaulinic]

Hey @github-abcde - that sounds interesting. Would you mind sharing your code (and eventually submit a PR?). Cheers!

[dawidmalina]

@github-abcde @mirceaulinic I am also interested in this change.

If I am not mistaken this could be this branch:
https://github.com/github-abcde/salt/tree/mine_client-side-acl

Looking forward to this change 👍

[MTecknology]

If the goal is to get LE certs from one minion to many others, that seems relatively easy to do within salt.

1. Set up gpg encryption on the salt master
2. Distribute the master's public gpg key to minions
3. Minion creates private data to distribute
4. Minion encrypts data with master's pubkey
5. Minion sends encrypted data up to master via event bus
6. Master reads the event and uses a reactor to stick the data into sdb (sqlite backend would be fine)
7. SDB is now populated with gpg-encrypted data
8. Other minions request data from pillar
9. Pillar sls validates request and pulls data from sdb
10. Master runs the pillar data through the gpg renderer
11. Master sends data down to the minion

For added fun, the reactor could, after putting data into sdb, run an orchestration that tells specific minions they need to refresh pillar and run a set of states.

[MTecknology]

Another option, partially suggested by someone on IRC...

1. Set up minionfs
2. Put a key for symmetric gpg encryption into pillar for specific nodes
3. Minion creates private data to distribute
4. Minion encrypts data with symmetric gpg key
5. Minion pushes file to minionfs
6. Other minions retrieve data from minionfs
7. Other minions decrypt data using symmetric gpg key

This is a lot less steps, but also less flexible. I suspect it would still require use of the event, reactor, and orchestrate systems.

Their suggestion also mentioned that the minions could leverage existing key pairs that the minions already use for communication with the master.

[Thambirajap]

Hi,

Apologize for posting my clarification here.
Can we transfer files without using salt file server from master to minion?
Say files stored under /tmp in salt master to be transferred to my specific targets.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.