Currently defining defining pillar key the second time will overwrite the contents of the first definition, i.e.:
first.sls:
data:
love
second.sls:
data:
hate
top.sls:
base:
"*":
- first
- second
The result pillar:
data:
hate
As the pillar definitions are pure data, there is no need to have unique IDs, or unique keys at all.
first.sls:
data:
love
second.sls:
data:
hate
resulting pillar:
data:
love
hate
Pillar is data, data is a set. Implement basic set operations: union, intersection, difference for overlapping data definitions.
This can get very complicated because pillar data is not structured like state data, but it should be a viable thing to add, I have set this to 0.15.0, but it might get bumped if it proves too complicated
+1*10^3 :-)
Default should be simple, and the current behavior -- overwrite -- is fine for the default. At least you can control it with the order of include's. Union will change arity of the data, which becomes problematic for nested structures.
Let's define a uniform convention and have tests which will demonstrate each case. Then the tests can serve as the docs on include/extend behaviors.
ran into this today, wanted to extend a network setting pillar with another one derived from the first and was puzzled why the data was missing.
+1 from me!
+1 Please make some way of mixing data structures. Otherwise it is impossible to manage complicated structures such as access lists without lots of CUT & PASTE.
Another +1 here. I tend to use the top level pillar object name for scoping the pillar data and would like to be able to extend the object with more data from another pillar file.
pillar of pillars :-)
+1 from me, avoids tedious copying/maintenance of long lists (e.g. users) with only some different entries.
+1. Forcing us to have unique IDs is making our state configuration messy and difficult to maintain.
Yes, I am sorry we keep pushing this off, we will get it in!
This seems to be a duplicate of an older issue I posted #2466. But I am still really looking forward to this being added in 17.0!
I posted a proposal about include/extend issues on #2466. Eventually we can use the same logic for top files ?
+1 forced to use unique id's is giving me a long complicated pillar dict for a server. and the state configuration now needs to look for different keys, where it could just loop through one if pillar dicts were extensible.
+1 If pillars could have "defaults" then my life would be easier.
+1
why don't use Jinja {%- include 'common.sls' -%} ?
redbaron you can include without jinja. It will still not merge namespaces.
+1 for Pillar files with some form of shared/extended namespaces.
+1
To reiterate what I mentioned in the referenced issue above, what I'm looking to do by extending and replacing Pillar data is this: https://gist.github.com/garethgreenaway/7237050
I want common data available to all hosts but have the ability to overwrite those values on a host by host basis.
Yes exactly, right now I have to do a bunch of copy-and-paste just to "override" one value.
@garethgreenaway @axiom I'm doing something similar by using include
with defaults. I think you can do something like this:
common.sls
:
openssh:
keys:
gareth:
allow: {{ allow_gareth }}
root:
allow: {{ allow_root }}
dns1.sls
:
include:
- common:
defaults:
allow_gareth: True
allow_root: True
This was the workaround I was able to create until we get a proper extend
implementation :)
Oh, that's an interesting work-around, @ruimarinho. That'll definitely help me out until we get extend
implemented.
@ruimarinho
+1 Good work around
+1
+1 for deep merging of pillar data, it would be so useful!
I've also heard of reclass...
Do you know if reclass or equivalent external pillar could solve this problem?
Hey, I've just created a bounty for this issue:
https://www.bountysource.com/issues/1378459-feature-request-extend-functionality-in-pillar-sls
if someone wants to help reward this feature or actually contribute the pull request ;)
A branch was mentioned https://github.com/saltstack/salt/issues/2466#issuecomment-20582430 that might solve the issue but not in the way suggested by Proposition A. I kind of like the approach more, but the code has not been touched in 7 months. I'm thinking about poking at this.
Hi folks. I am currently working on a serialization cleanup. With this branch, I am able to propose another approche of this problem, which seems the most elegant to my mind.
If you want to see my work in progress (https://github.com/johnnoone/salt/tree/serializers).
Here is the idea.
YAML allows to defines custom tags, so my proposition is to decorate nodes with an !aggregate
yaml tag in order to instruct salt if aggregation is permitted or not:
this yaml document have duplicate keys, which values are tagged !aggregate
:
foo: !aggregate first
foo: !aggregate second
bar: !aggregate {first: foo}
bar: !aggregate {second: bar}
baz: !aggregate 42
but tagged values instruct to salt that overlaping values they can be merged together:
foo: [first, second]
bar: {first: foo, second: bar}
baz: [42]
this is retrocompatible with the current .sls files.
for example, this yaml document have still duplicate keys:
foo: first
foo: second
bar: {first: foo}
bar: {second: bar}
baz: 42
so the late found values must prevails:
foo: second
bar: {second: bar}
baz: 42
foo: !aggretate {key: value}
bar: !aggretate [42]
baz: !aggretate ~
qux: !aggretate 42
is equivalent to
foo: {key: value}
bar: [42]
baz: []
qux: [42]
aggregation is permit between tagged object, and these objects must same the same type.
if not, the default merge strategy prevails.
for example, these examples fails:
foo: {first: value}
foo: !aggretate {second: value}
bar: !aggretate {first: value}
bar: 42
baz: !aggretate [42]
baz: [fail]
qux: 42
qux: !aggretate fail
should be interpreted as:
foo: {second: value}
bar: 42
baz: [fail]
qux: [fail]
Any comments and suggestions are welcome.
@johnnoone that is very interesting. One question that comes to my mind is how this works with nested values. Would I be able to do something like:
foo: !aggregate {bar: !aggregate first}
foo: !aggregate {bar: !aggregate second}
foo: !aggregate {another: value}
yes @jesusaurus, this spec also works with nested values.
your previous example should be interpreted as:
foo:
bar:
- first
- second
another: value
Excellent
i've also added a !reset
tag that allow to flush the computing value.
placeholder: {!aggregate foo: {foo: 42}}
placeholder: {!aggregate foo: {bar: null}}
!reset placeholder: {!aggregate foo: {baz: inga}}
is roughly equivalent to
placeholder: {!aggregate foo: {baz: inga}}
Hi, OP here.
I'll cut the story short. After using salt for more than a year, with
different setups, I'v come to realisation that I'd like to share and
that is relevant to this issue.
And the realisation is such: pillar structure is great, because it is
simple. When you start to extend
, reduce
or in any other way
transform data structures by overlaying one structure on the other - the
management of such structure becomes less clear, less simple and thus
harder to maintain.
Thus, if one wants to have data structures that could be merged, use
reclass, use database as external pillar, or any other external pillar
that is already available. Default pillar is great because of it's
simple and consistent rules and application.
I'd say leave it as it is, and use, or even better - expand already
existing external pillars for higher acrobatics of the data structure. :)
~ Zogg
On 02/21/2014 10:41 AM, johnnoone wrote:
i've also added a
!reset
tag that allow to flush the computing value.placeholder: {!aggregate foo: {foo: 42}} placeholder: {!aggregate foo: {bar: null}} !reset placeholder: {!aggregate foo: {baz: inga}}
is roughly equivalent to
placeholder: {!aggregate foo: {baz: inga}}
Reply to this email directly or view it on GitHub:
https://github.com/saltstack/salt/issues/3991#issuecomment-35708593
this might be good for some of the use cases
https://github.com/ranl/salt-pillar-linker
@Zogg thanks for replying with an update to your ticket. I agree with your conclusions.
I'll mention one other easy way to share data between Pillar files. Add a defaults.yaml
or defaults.json
file that contains the shared data and make it available in multiple Pillar files with {% import_yaml "defaults.yaml" as defaults %}
.
http://docs.saltstack.com/ref/renderers/all/salt.renderers.jinja.html#template-serializers
It would be wonderful if the functionality mentioned in the original bug report could be implemented.
+1
+1
+1
:+1:
+1 Being able to merge pillar keys would be really helpful
+1 even multiple inheritance, such as in python classes, would be great. something an extends/_extends key could take a list of other paths it extends. that would be awesome.
A little late to the party, but another solution is to move common items into a document with YAML anchors, and then include that document (via Jinja) somewhere in order to reference the anchors:
# common/users.sls
common_users: &common_users
foo: bar
# users1.sls
{% include 'common/users.sls' %}
users:
<<: *common_users
baz: bat
# users2.sls
{% include 'common/users.sls' %}
users:
<<: *common_users
qux: quux
Granted, this comes with the caveat that it's on a per-key level, so any true nested merges would still be an issue; however, if it's simply an issue of single-level merging (e.g. adding new keys to a higher-level dict), this would be one way to solve that. Additionally, #14181 proposes potentially deferring YAML parsing until includes have been resolved so this would be supported with pillar's include
keyword vs relying on the template engine.
Jury's still out on whether it is maintainable in the long run, but for simple cases it seems to work fine for now.
+1
+1 :+1: if it doesn't break the security :)
@mway I like this approach, but it still causes some trouble :( See #14550
+1
At least implement merging of list structures:
/pillar/top.sls:
base:
'*':
- roles.common
'front*":
- roles.frontend
/pillar/roles/common.sls:
roles:
- ldap_client
/pillar/roles/frontend.sls:
roles:
- frontend
So for a host matching front* hostname this would result in a concatenated list:
roles:
- ldap_client
- frontend
+1111 lists!
lol this feature request brings back memories. Alexy and I wrote this
request back when we were both working at Versal.
My expected behavior would be to have saltstack implement a recursive
merge, or 'deep merge' :-)
Recently implemented for json processing using shell and jq:
https://github.com/airstack/json-utils/blob/master/command/json-deep-merge
Should be fairly simple to add a deep-merge functionality to saltstack yaml
processing. Maybe have a default global setting to specify how to handle
conflicts. (Current default is to clobber) And/or allow individual state
commands to specify how to handle conflicts?
On Tue, Sep 30, 2014 at 4:47 AM, Arnold Bechtoldt [email protected]
wrote:
+1111 lists!
—
Reply to this email directly or view it on GitHub
https://github.com/saltstack/salt/issues/3991#issuecomment-57301430.
Being able to use grains to adjust pillars.
The big issue with puppet is data in modules is missing. People do not want to place big case statements in their code for say package names. They just want to say load the RedHat data, load the Solaris Data. Load the development environment data, load the data for the network the system is on.
This sort of coding below should be prevented
pkgs:
{% if grains['os_family'] == 'RedHat' %}
apache: httpd
vim: vim-enhanced
{% elif grains['os_family'] == 'Debian' %}
apache: apache2
vim: vim
{% elif grains['os'] == 'Arch' %}
apache: apache
vim: vim
{% endif %}
init.sls should be able to do
include grains['os'].sls|salt://pillar/grains['nodename'].sls|salt://pillar/grains['network].sls
Or
hiera: (go hunting global list for a match which might contain salt://pillar/grains['nodename'].sls, then the local version as define below)
- grains['osfinger'].sls
- grains['os'].sls
- grains['os_family'].sls
pkgs:
apache: {% heira( 'pkgs::apache') %}
vim: {% heira( 'pkgs::vim') %}
Another example is different OS have sshd_config in different locations.
The idea is re-use. You should be able to take something someone else has developed and not change it. Just drop it in place, and then use an external file be able to change values in it, e.g. it might not support solaris 11 and you an add the package name, or you have a private version of apache called my-company-apache. The original source changes, you can just pull it down again, with out having to patch it again. Keys need to be unique so they need to be prefix in some way, i.e. an expected standard.
By allowing look-up, people can add support for OS or company specific values to be included with out changing the original code. Sorry if this is out side the scope of the original issue.
@damon-atkins Have a look at how Salt formulas are implemented (esp. their map.jinja
file). I think this is pretty much what you want.
+1 - all we need is a clean deep merge
+1 I see this in the approved milestone.
+1 This is a feature that ir highly required for more complex scenarios. I want to have a default.sls with a nested dict that can be aggregated to the dict I set in my pillar.
While you are waiting for this to be resolved, you can achieve the same effect (default.yaml, merged with lower down yaml) by setting pillar_source_merging_strategy: aggregate
in master config, and then in each yaml, lead with #!yamlex
on the top, then on each leaf node you want to support aggregation, add a !aggregate
stanza.
This was detailed a bit earlier in the thread, but some extra details were missing (adding !aggregate to each leaf for example).
Works like a charm. It solved the problem that many in this thread are looking for. Obviously there is no way to remove data, just change or append. Having to use !aggregate on each leaf is a problem for some yaml parsers. It would be nice for this to just be a native solution at some point similar to hiera with deep_merge.
ie.. default.yaml
#!yamlex
users: !aggregate
bob:
user_data: !aggregate
fullname: 'Bob Flarn'
uid: 5001
gid: 5001
gid_from_name: True
optional_groups: !aggregate
- sudo
shell: /bin/zsh
Then later in someother.yaml
#!yamlex
users: !aggregate
bob:
user_data: !aggregate
optional_groups: !aggregate
- someothergroup
fran:
user_data: !aggregate
fullname: 'Fran Foobar'
uid: 5002
gid: 5002
gid_from_name: True
optional_groups: !aggregate
- sudo
The yamlex parser is not a good solution for this. I've tried to use it and it has its problems along with the fact that maintaining leaf level tags in the pillars sucks badly. I know the recurse_list merging strategy has been added to the dictupdate util. This will basically provide the features for deep_merging. Don't know if it will be in 2015.8.2 or in the next release though.
https://github.com/saltstack/salt/blob/develop/salt/utils/dictupdate.py
I agree having to annotate every leaf is a pain and hard to track down problems if you forget. Again, for me this is a working solution until the proper one is deployed - I couldn't wait, and this request goes back to 2014 already.
Functionality for "deep merge" of dictionaries is already implemented.
Correct me if I'm wrong.
At least, (1) the example in the initial post for this issue and (2) example of this SO question are essentially the same:
http://stackoverflow.com/questions/24631231/how-to-join-two-salt-pillar-files-and-merge-data
So, in case of key-value dictionaries, the solution is in place by default at least since version 2014.7.0
using no keywords like extend/include
at all.
Now, merging lists is _completely different_ and _less straightforward_ problem which:
Shouldn't this issue be closed?
Similar problem exists when configured pillars are merged with are overridden by pillar
argument supported by some functions on the command line - see #29516.
Merging has to be consistent regardless of pillar data source (configuration or command line).
+1 for a deep merge please!
I am also missing list mergeing feature. My use case:
# logcheck.sls:
logcheck:
logfiles:
- '/some/defaut/log.log'
Now, if I want to deploy apache to _some_ machines, I add apache.sls
state and apache/init.sls
and apache/logcheck.sls
pillars for that machines top files.
apache/logcheck.sls
could be like this:
#apache/logcheck.sls
logcheck:
logfiles:
- '/var/log/apache/error.log`
And I somehow expected that these two lists would merge. I guess I have to add dummy dicts:
# logcheck.sls:
logcheck:
logfiles:
some_log:
- '/some/defaut/log.log'
#apache/logcheck.sls
logcheck:
logfiles:
apache:
- '/var/log/apache/error.log`
Now logcheck:logfiles
dict is merged, just need change template to iterate through these internal lists...
@Talkless, I think you can rely solely on merging dict
s.
You already mentioned "dummy dict
s".
However, I want to make a slight touch (to simplify it syntactically).
logcheck.sls
:logcheck:
logfiles:
some_log: '/some/defaut/log.log'
apache/logcheck.sls
:logcheck:
logfiles:
apache: '/var/log/apache/error.log'
The result will be:
logcheck:
logfiles:
some_log: '/some/defaut/log.log'
apache: '/var/log/apache/error.log'
It is easily loopable list
(!) of values()
:
{% for log_path in pillar['logcheck']['logfiles'].values() %}
...
{% endfor %}
Is it suitable in your case?
The majority of "merge list
" use cases are easily implemented through dict
s.
That's why I even proposed to close this issue above.
The only exception is when _order_ of items listed is important - see example here. I bet in such cases nobody wants to split lists across files and then merge them somehow in a less deterministic way (instead, it is much clearer to have a single file with one large single list - the problem of merging it does not exist in this case).
Thank you @uvsmtid , it's nicer to use with values() . Good to know.
+1
I'm using Pillar data to set a grain which defines what states are run.
having the ability to inherit this by it combining the pillar entries would be really handy.
I have the following pillar definitions
File1 (server group)
grains:
appliedstates:
- state1
- state2
File2: (server specific)
grains:
appliedstates:
- state3
This way just ends up with the final pillar containing state3. This may be an odd way of using it but using the Pillars to set states gives me a lot more control for automating the configuration of this.
and if i can set things higher and have it filter down would save a lot of duplication..
Guys, the external pillar which provides this functionality is already included in 2016.3 release https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.stack.html !
Thanks @genuss! Will take a look at pillarstack. This could save tons of jinja effort :)
Thanks @genuss. reading through the page it looks like it would do what i want.. although one thing I'm unsure about is the following line in the PillarStack Configuration Files section
"The path of these yaml files must be relative to the directory of the PillarStack config file."
This would imply that it only works if the pillar yaml files are locally on the salt master with the pillarstack config file?
I'm using the gitfs option to share my state/pillar files accross multiple masters. Any one have PillarStack working with gitfs? or know that it definitely is possible?
PillarStack (and reclass and suchlike) is absolutely overengineered when it comes to simple and intuitive application of pillar data aggregation in relation to merging lists. Using another complex module -- that also requires extra configuration -- just to butter over the lack of a basic future in the default pillar system is just a messy workaround. It's not a good solution.
@uvsmtid 's suggestion to just use dicts is also just a hack because there are actually reasons to lay out data as a list. I.e. templating a list of values in some sort of config file
services:
- ssh
- nscd
- pam
is just $thatlist | join(", ")
in Jinja; exchanging that and laying out the same thing as a dictionary with essentially unused keys disturbs both the structured configuration format Salt is all about and adds extra loops to the templating code.
Just merging the list structures in the default pillar system appears to be the most simplest and only reasonable solution.
@paul-mulvihill, I'm wondering if you could use salt:// paths that would allow PillarStack to read from git clone? I'd be interested to see the two working together.
@mlalpho not sure how well that would work for the setup i am going for.. I'm intending to have the top.sls file become an automatically generated file as well down the line so given the pillarstack lives on the master that wouldn't work. I have a system of records that i'm aiming to create most of the pillar files and then potentially this one as the hierarchy expands.
The salt/top.sls or pillar/top.sls ?
From what I've read the PillarStack would have access to the pillar variable, so if you have some things defined in your git pillar, you can operate with them using jinja within PillarStack, which, is something you can't do with normal salt pillars at the moment without using jinja templating and loading yaml, json etc in to a jinja var. Aside from the merging of pillar data as a whole, that's already a big win in my opinion.
I'm refering to the pillar top.sls. my setup is gitfs for everything with seperate repos for states and pillars. Eventually i'm goign to have our system of records generate the top.sls file for pillar information,
In my setup the state\top.sls reads the "appliedstate" grain I define and uses it's values to apply states.
I want to avoid any key configuration being installed on the masters except for telling them to look at gitfs. I'd rather changes to the pillar hierarchy or values are purely within the repo.
By using PillarStack you do get nice merging strategies, but you lose out on being able to use them with other ext_pillar sources, like git_pillar or pillar_ldap.
It would be nice if the merging strategies that PillarStack offers were part of the 'core' pillar code.
:( guessed it would be something like that.. for the scale i'm building a salt estate I need git as the backend, so i'll have to find a work around until we can get this sort of features in the core code.
You can have this very same feature using the slsutil.merge module. Here is a snippet to place in your map.jinja:
# vim: sts=2 ts=2 sw=2 et ai
{# Start with defaults from defaults.yml #}
{%- import_yaml 'nginx/proxy/web/defaults.yml' as default_settings %}
{# Only add key:values here that differ from whats in defaults.yml #}
{%- set os_map = salt['grains.filter_by']({
'Debian': {},
'RedHat': {
'pkg': {
'name': [
'nginx',
],
},
'service': {
'name': 'nginx',
'state': 'running',
'enable': True,
},
'user': 'nginx',
'group': 'nginx',
'conf': '/etc/nginx/nginx.conf',
'confd_dir': '/etc/nginx/conf.d',
'certs_dir': '/etc/nginx/conf.d/certs',
'vhosts_dir': '/etc/nginx/conf.d/vhosts',
'docroot': '/usr/share/nginx/html',
'pool': salt['grains.get']('pool', 'default'),
},
},
grain="os_family"
)
%}
{# Merge the os_map and the pillar data to the default_settings #}
{%- set nginx = salt['slsutil.merge'](
salt['slsutil.merge'](
os_map,
default_settings.nginx.proxy.web,
merge_lists=True
),
salt['pillar.get'](
'nginx:proxy:web', {}
),
merge_lists=True
)
%}
With that you'll have a full merged dict including lists.
PillarStack author here: personally I'm also storing my pillar and states data in git (on bitbucket private repositories), and I'm using a docker container to automatically update my pillar and states tree when a new commit is pushed to bitbucket (new commits trigger POST web hook requests to my docker container which update the relevant pillar or states tree).
Here is the docker container I'm using:
https://github.com/bbinet/docker-hooked-git-workdir
Ran into this today. Would love to see this implemented at some point!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
Most helpful comment
PillarStack (and reclass and suchlike) is absolutely overengineered when it comes to simple and intuitive application of pillar data aggregation in relation to merging lists. Using another complex module -- that also requires extra configuration -- just to butter over the lack of a basic future in the default pillar system is just a messy workaround. It's not a good solution.
@uvsmtid 's suggestion to just use dicts is also just a hack because there are actually reasons to lay out data as a list. I.e. templating a list of values in some sort of config file
is just
$thatlist | join(", ")
in Jinja; exchanging that and laying out the same thing as a dictionary with essentially unused keys disturbs both the structured configuration format Salt is all about and adds extra loops to the templating code.Just merging the list structures in the default pillar system appears to be the most simplest and only reasonable solution.