Salt: Pillar targeting starts to work only after calling `saltutil.refresh_pillar`

[vutny]

Description of Issue/Question

Salt Master is unable to get a response from the Minion targeted using Pillar from a command line until I call saltutil.refresh_pillar function against that Minion (or saltutil.sync_all or state.highstate).

Setup

I did a fresh setup of Master and Minion (named test) without any additional configuration (except of pillar_roots option).
My files:

# pillar/top.sls
base:
  'test':
    - test

# pillar/test.sls
test: 'string'

Steps to Reproduce Issue

Pillar is visible on Minion test:

# salt test pillar.items
test:
    ----------
    test:
        string

Pillar targeting seems to work, but there is no return from the Minion:

# salt -l debug -I 'test:string' pillar.items
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/b3/a65df54bfaed4f07b17f4da7e2d205/.minions.p
[DEBUG   ] get_iter_returns for jid 20160325130624345444 sent to set(['test']) will timeout at 13:06:29.350370
[DEBUG   ] Checking whether jid 20160325130624345444 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] LazyLoaded no_return.output
test:
    Minion did not return. [No response]

If there are no Minions to match the target, Salt behaves as expected:

# salt -l debug -I 'test:other_string' pillar.items
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master', 'tcp://127.0.0.1:4506', 'clear')
No minions matched the target. No command was sent, no jid was assigned.
[DEBUG   ] LazyLoaded nested.output
ERROR: No return received

And after I ran salt '*' saltutil.refresh_pillar Pillar targeting started to behave correctly.

Concerns

1. It's hard to figure out that manual Pillar refreshing is a requirement for this targeting method.
   Such thing should be reflected in the documentation.

2. Also it's easy to forget to do refresh all the time after Pillar data was changed.
   I think we need to ship a scheduled job configuration with the minion like it's already done for mine.update.
   For example:

   # /etc/salt/minion.d/_schedule.conf 
   schedule:
    __mine_interval: {function: mine.update, jid_include: true, maxrunning: 2, minutes: 60}
    __pillar_interval: {function: saltutil.refresh_pillar, jid_include: true, maxrunning: 2, minutes: 60}
   

   By the way, mine.get with compound matches using Pillar would definitely need this.

3. Looks strange that Salt actually performs Minion lookup by Pillar data correctly.
   It just unable to provide return from requested function (test.ping gives the same result).
   Does Salt really needs the Pillar data cache for targeting and providing return if it already does a half of the job without any cache?
   

   Versions Report

Salt Version:
           Salt: 2015.8.8

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: 0.21.0
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.10.1.el7.x86_64
         system: CentOS Linux 7.2.1511 Core

[jfindlay]

@vutny, thanks for reporting. Pillar data is stored on/officially retrieved by the master by design. Minion target resolution is done by the master, so its pillar data will be updated when the pillar updates.

[vutny]

Yeah, I've checked the code and I think I got some basic understanding of Pillar data flow and storage.
As for me it was little bit confusing, that making or updating Pillar cache on the Master side is only possible by executing saltutil.refresh_pillar function on each Minion.

[vutny]

It is still under question, should we add persistent scheduled job for Pillar refresh? I guess there could be some performance penalties in case of deployment with high number of Minions. But this is just a guess.

[sjmh]

Please don't add a _default_ scheduled job for pillar refresh. With a high number of minions, and somewhat expensive external pillars, this would cause a tremendous amount of unwanted load. You can make it optional, but I don't want to have this by default.

[cachedout]

@sjmh Don't worry. Somebody tried to sneak that in a while back and it was firmly rejected. We've got your back. ;]

[sjmh]

@cachedout thanks ;)

I do have a question about this ticket though - @jfindlay mentioned that minion-target resolution is done via the master and I'm a little confused by that. From what little I know, I was assuming that saltutil.refresh_pillar was allowing the minion to cache it's pillar data and that it played into the same client-side filtering that I always assumed Salt did. Does a minion actually cache it's rendered pillar data at all?

In other words, until you ran a saltutil.pillar_refresh, the minion didn't have a cache of it's own pillar data, so couldn't respond to a request that targeted minions via that pillar data. I'd assume the master would also need a cache of what minions had what pillar data, so that it could compile it's list of what minions _should_ return.

Maybe I'm misunderstanding what was meant by 'target resolution'.

[vutny]

@cachedout Now I understand the possible implications of having such scheduled job enabled by default. So, this issue is resolved for me.

@sjmh Your suspicions are correct. saltutil.refresh_pillar does not put rendered Pillar cache on Minion. It actually populates /var/cache/salt/master/pillar_cache/ directory on Master with files named by Minion ID where the Pillar data saved.

Do you have any concerns about that?

[sjmh]

@vutny - No real concerns - although it seems like the master has to do more work in that case, since it would seem that it would have to compile a list of minion names from the pillar data to target, where as if the minion's kept a local cache of pillar data, they'd be able to do target resolution on the client side ( which is how I thought most targeting mechanisms worked ).

Also, it seems strange that if a minion isn't connected - shouldn't the master still be able to retrieve the pillar data for that minion, especially if that data is master-side?

[cachedout]

There are concerns I have about minion-side pillar caching, mostly revolving around security. I don't like the notion, for example, that an individual might might use a GPG-encrypted pillar on the master-side and then end up with unencrypted data sitting in a minion cache. I'm not ruling it out of course, just pointing out that it makes me uneasy.

[sjmh]

@cachedout - No, that makes complete sense and something I hadn't thought of.

@vutny - is /var/cache/salt/master/pillar_cache only in 2016.3? Where is pillar data cached in 2015.8?

[root@qcysaltmom01 tmp]# ls -l /var/cache/salt/master/
total 348
drwxr-xr-x    4 root root     30 May 12 23:41 file_lists
drwx------   10 root root   4096 May 17 18:31 gitfs
drwxr-xr-x    3 root root     66 May 17 00:11 git_pillar
drwxr-xr-x  258 root root   8192 May  7 06:52 jobs
drwxr-xr-x 6310 root root 253952 May 17 06:48 minions
drwxr-xr-x    2 root root      6 May  2 17:17 proc
drwxr-xr-x    2 root root      6 May  2 17:17 queues
drwx------    2 root root     22 May  2 17:17 roots
drwxr-xr-x    2 root root      6 May  2 17:17 syndics
drwxr-xr-x    2 root root     45 May 17 20:40 tokens

[vutny]

@sjmh I have such directory on Salt Master 2015.8.8.2. I thought it was created during saltutil.refresh_pillar job run.

[sjmh]

@vutny - Do you have pillar cache settings enabled on your master? Because I don't have that directory.

[root@qcysaltmom01 minion]# salt --versions
Salt Version:
           Salt: 2015.8.8.2
...

But, I also don't have normal pillars - just ext_pillar's defined.

[vutny]

@sjmh Sorry for the late answer...

You're totally right, I did experiments with Pillar settings and enabled caching temporarily, that's why that /var/cache/salt/master/pillar_cache was cleated. It's not related here.

I've checked what saltutil.refresh_pillar is doing, and it appears that it instructs Minion to update its Pillar data in runtime (by putting it in the opts dict). So, targeting by Pillar will work only till the next restart of Salt Minion.

Now this makes a sense to me. Master is correctly targeting Minion by Pillar data, but the Minion is unable to respond because it didn't retrieved its Pillar dictionary before...

Should really the Master force Minion to obtain fresh Pillar data in prior to execute requested module?
That would eliminate the need of calling refresh_pillar job each time when Pillar data changes or the Minion got restarted.
But it appears to be too deep architecture topic for me.

@cachedout Corrections? Thoughts?

[sjmh]

I think refreshing the pillar prior to each execute of a requested module would be too expensive - especially if you have multiple external pillar values.

I think it makes more sense that the minion refreshes it pillar when:

• It starts up
• It's explicitly asked to via saltutil.pillar_refresh
• An optional schedule ( once every 12 hours or so )

[vutny]

Yeah, I like your points. But still I'm not sure that Salt Minion really does not obtain fresh pillar during startup... Need to dive in the code.

[vutny]

Using Salt version 2016.11.2 I don't experience weird Pillar issues anymore. And it appears that Pillar data fetching logic is properly described in the docs: https://docs.saltstack.com/en/latest/topics/pillar/#in-memory-pillar-data-vs-on-demand-pillar-data

With that said, I'm closing this out.

[timwsuqld]

https://github.com/saltstack/salt/issues/34735 is related.
Even though the salt master knows the the pillar targeting is for that minion, the minion doesn't respond until a saltutil.refresh_pillar is run against the minion. I think that a minion pillar refresh should be run automatically, when targeting minions by pillar, to ensure that the minion will respond.

I'm having this issue currently with salt 2018.3.2

salt -V
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.5.2
       cherrypy: 3.5.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-131-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[llamaqcom]

The same situation with 2018.3.2 as @timwsuqld describes, even worse. Even salt '*' saltutil.refresh_pillar didn't help to refresh cached pillar data on a minion. Only rm -rf /var/cache/salt/minion/ and then systemctl restart salt-minion before salt '*' state.highstate test=True did the trick.

[morsik]

I have the same problem with

[root@masterofmasters ~]# salt --version
salt 2018.3.3 (Oxygen)

Works fine after doing saltutil.refresh_pillar.