Salt: Order in top.sls file is not respected for pillar data in local mode

[astehlik]

My pillar setup looks like this:

In the minion.conf the pillar_roots config looks like this:

pillar_roots:
  lib:
    - /srv/salt/salt-lib/pillar
  base:
    - /srv/salt/pillar

My pillar top.sls file in /srv/salt/pillar/top.sls looks like this:

lib:
  '*':
    - main

base:
  '*':
    - basic

The main.sls file in /srv/salt/salt-lib/pillar/ contains:

test:
  setting: myvalue-main

The basic.sls file in /srv/salt/pillar/ contains:

test:
  setting: myvalue-override

When I now run

sudo salt-call --local pillar.get test:setting

the output is:

local:
    myvalue-main

The expected output would be:

local:
    myvalue-override

The salt version is 2014.7.1

Is this related to #1432 ?

[jfindlay]

@astehlik, does this also happen on 2015.5.2?

[astehlik]

Just tested it with 2015.5.2, the value is still not overwritten :(

[jfindlay]

Have you changed the value of pillar_source_merging_strategy in the master config? I think the default might not do what you want. You could try pillar_source_merging_strategy: overwrite. http://docs.saltstack.com/en/latest/ref/configuration/master.html#pillar-source-merging-strategy

[bbinet]

I have the same issue, but I think this is a known issue since the documentation says "Remember: conflicting keys will be overwritten in a non-deterministic manner!": http://docs.saltstack.com/en/develop/topics/pillar/index.html#pillar-namespace-merges

It would be great if the order could be preserved when pillar files are read and merged so that conflicting keys will be overwritten in a deterministic manner (last wins).

[ahus1]

@jfindlay I tried the settings pillar_source_merging_strategy with the default smart setting and the overwrite setting. The result is still myvalue-main.

I looked at 2015.5.5 and 2015.8.0 and the result is the same.

The source of the problem seems to be preserving the order of environments in pillar/__init__.py: It starts out with self._get_envs() returning a standard set of environments defined as the pillars in pillar_root, and continues throughout the file with collections.defaultdict, for example in merge_tops(). While the pillars within each environment are ordered, the environments themselves are not ordered.

The behaviour I would expect: A deterministic order of the top files to be read (as listed in pillar_roots?), and then processing the environments as listed in each top file.

My workaround is now to have an external pillar to override values, as all external pillars override pillar_roots as a default (unless ext_pillar_first is specified). Any external pillar will do:

ext_pillar:
  - cmd_yaml:
      command: cat /srv/salt/pillar/basic.sls

[jfindlay]

@ahus1, thanks for the info.

[npetzall]

Just to clarify not only a problem in local mode.

As @ahus1 points out it has to do with dict usages instead of ordereddict.

So #19332 with the comment that order is as declared is only true within an environment.

Use-case:
Single pillar top.sls
last environment is called vagrant and has a grain matching on virtual:VirtualBox
when pillar is rendered the prod environment is rendered last and overwrites the vagrant pillar data.

For my use-case changing 3 dict to ordereddict got my desired behaviour.

[whytewolf]

ZD-2198

[wwalker]

Any idea when this will get fixed? It was marked bug, high severity 2.5 years ago.
It would immensely simplify many of my pillars if this worked.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[wwalker]

This shouldn't be closed. The problem still exists. It would make pillar data management much easier.

[notpeter]

@whytewolf Can you reopen please.

[whytewolf]

@notpeter @wwalker I can reopen. I must have missed the stalebot email last month.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[wwalker]

https://docs.saltstack.com/en/latest/topics/pillar/#pillar-namespace-flattening

Implies that merging would work. The example doesn't say that it is non-deterministic if both dictionaries contain the same key.

Can we add a note in the mean time to the docs?

[sagetherage]

@barbaricyawps let's look at a doc change here.

[whytewolf]

@sagetherage a doc fix is a good first step and a stopgap. Is there any way to actually get the code to be deterministic?

[mparson]

Coming up on 5 years, seems like we should have something better than a doc fix. :(

[sagetherage]

the doc fix is only a portion yes, but something that may be done by a separate person so was asking, merely

[sagetherage]

There are several things going on here and the team needs to research what exactly is happening now, we can only commit to the research for now.

[mkirkland4874]

This patch seems to fix the pillar order problem for me on 2019.2.7. Based on https://github.com/saltstack/salt/pull/37190/commits/59657b12eb0f2a9d03891260b531036eb066c175

--- salt/pillar/__init__.py.orig    2020-12-30 13:31:20.840940457 -0800
+++ salt/pillar/__init__.py 2020-12-30 13:34:18.585825693 -0800
@@ -498,9 +498,10 @@ class Pillar(object):
         '''
         Pull the file server environments out of the master options
         '''
-        envs = set(['base'])
+        envs = ['base']
         if 'pillar_roots' in self.opts:
-            envs.update(list(self.opts['pillar_roots']))
+            envs.extend([x for x in list(self.opts['pillar_roots'])
+                          if x not in envs])
         return envs

     def get_tops(self):

[s0undt3ch]

@mkirkland4874 Thanks! Looks like a good fix. Want to take a stab at a PR?

[mkirkland4874]

I can do that. The above patch is for the 2019.2 branch--would you actually pull that into 2019.2 or do you want a master branch patch. __get_envs_ in master seems to be slightly different (i.e., _envs = set(['base'])_ in 2019.2 is _envs = {"base"}_ in master)

[s0undt3ch]

@mkirkland4874 against the master branch please.