Salt: [2017.7] Pillar empty when state is applied from orchestrate

Created on 19 Jul 2017 · 8Comments · Source: saltstack/salt

Description of Issue/Question

When states are applied via orchestration, the pillar data is empty.

Setup

(Copied from https://gist.github.com/astronouth7303/3d17cf6679a9a395bb93f1648fd8ceb0 )

test_orch.sls:

test:
  salt.state:
    - tgt: minion
    - sls:
      - test_state

test_state.sls:

{% do salt.log.error("Start pillar") %}
{% do salt.log.error(pillar) %}
{% do salt.log.error("End pillar") %}

Steps to Reproduce Issue

Add above sls files
Run test_orch with state.orchestrate

Logged output:

2017-07-19 17:12:49,055 [salt.minion                                           ][INFO    ][29846] Starting a new job with PID 29846
2017-07-19 17:12:49,713 [salt.state                                            ][INFO    ][29846] Loading fresh modules for state activity
2017-07-19 17:12:49,974 [salt.fileclient                                       ][INFO    ][29846] Fetching file from saltenv 'base', ** done ** 'test_state.sls'
2017-07-19 17:12:49,980 [salt.loaded.int.module.logmod                         ][ERROR   ][29846] Start pillar
2017-07-19 17:12:49,980 [salt.loaded.int.module.logmod                         ][ERROR   ][29846] {}
2017-07-19 17:12:49,981 [salt.loaded.int.module.logmod                         ][ERROR   ][29846] End pillar

Versions Report

Salt Version:
           Salt: 2017.7.0

Dependency Versions:
           cffi: 1.9.1
       cherrypy: unknown
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.3.7
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jan 19 2017, 14:48:08)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.0 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-3-amd64
         system: Linux
        version: debian 9.0 
 ```

salt-minion 2017.7.0 (Nitrogen)
```

Bug Core P1 ZRELEASED - 2017.7.1 fixed-pending-your-verification severity-critical severity-medium

Source

AstraLuma

👍1

All 8 comments

I am unable to replicate this.

[root@salt salt]# cat /var/log/salt/minion

[root@salt salt]# salt-run state.orch test
salt.c.gtmanfred-1263.internal_master:
----------
          ID: test
    Function: salt.state
      Result: True
     Comment: States ran successfully. Updating salt.c.gtmanfred-1263.internal.
     Started: 14:07:22.258824
    Duration: 355.534 ms
     Changes:
              salt.c.gtmanfred-1263.internal:
              ----------
                        ID: /touched
                  Function: file.touch
                    Result: True
                   Comment: Created empty file /touched
                   Started: 14:07:22.465406
                  Duration: 36.475 ms
                   Changes:
                            ----------
                            new:
                                /touched

              Summary for salt.c.gtmanfred-1263.internal
              ------------
              Succeeded: 1 (changed=1)
              Failed:    0
              ------------
              Total states run:     1
              Total run time:  36.475 ms

Summary for salt.c.gtmanfred-1263.internal_master
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 355.534 ms
[root@salt salt]# cat /var/log/salt/minion

2017-07-20 14:07:22,428 [salt.loaded.int.module.logmod][ERROR   ][4844] Start pillar
2017-07-20 14:07:22,428 [salt.loaded.int.module.logmod][ERROR   ][4844] {'hello': True}
2017-07-20 14:07:22,428 [salt.loaded.int.module.logmod][ERROR   ][4844] End pillar

Did you make sure to do a pillar_refresh on the minions?

Here are my files

[root@salt salt]# tail -c +0 /srv/*/*.sls
==> /srv/pillar/test.sls <==
hello: True

==> /srv/pillar/top.sls <==
base:
  '*':
    - test

==> /srv/salt/test.sls <==
test:
  salt.state:
    - tgt: '*'
    - sls:
      - test_state

==> /srv/salt/test_state.sls <==
{% do salt.log.error("Start pillar") %}
{% do salt.log.error(pillar) %}
{% do salt.log.error("End pillar") %}

/touched:
  file.touch

Even without the file.touch, the orchestrate fails, but the pillars are still passed.

[root@salt salt]# echo > /var/log/salt/minion
[root@salt salt]# vi /srv/salt/test_state.sls
[root@salt salt]# salt-run state.orch test
[ERROR   ] Run failed on minions: salt.c.gtmanfred-1263.internal
Failures:
    salt.c.gtmanfred-1263.internal:
        Minion did not respond

salt.c.gtmanfred-1263.internal_master:
----------
          ID: test
    Function: salt.state
      Result: False
     Comment: Run failed on minions: salt.c.gtmanfred-1263.internal
              Failures:
                  salt.c.gtmanfred-1263.internal:
                      Minion did not respond
     Started: 14:08:41.822141
    Duration: 396.308 ms
     Changes:

Summary for salt.c.gtmanfred-1263.internal_master
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time: 396.308 ms
[root@salt salt]# cat /var/log/salt/minion

2017-07-20 14:08:42,103 [salt.loaded.int.module.logmod][ERROR   ][5037] Start pillar
2017-07-20 14:08:42,103 [salt.loaded.int.module.logmod][ERROR   ][5037] {'hello': True}
2017-07-20 14:08:42,103 [salt.loaded.int.module.logmod][ERROR   ][5037] End pillar

gtmanfred on 20 Jul 2017

I'm able to reproduce this in a fresh salt-vagrant-demo environment. pillar.items run against minion1 and minion2 report items ({foo: bar}), but {} is still logged.

AstraLuma on 20 Jul 2017

Ok, I am able to replicate this now.

https://gist.github.com/gtmanfred/b52f8a9b6a7edda45853f2da6d43fe55

Thanks,
Daniel

gtmanfred on 20 Jul 2017

This will be fixed in the above PR #42433

gtmanfred on 20 Jul 2017

I have done some extensive testing on 2017.07 and it seems that regardless if saltmod state is called from orchestrate or salt CLI, same issue applies; pillar data are empty on the minion which causes a great deal of trouble.

Does PR #42433 fix all saltmod state use cases or just those related to state.orchestrate?

sokratisg on 21 Jul 2017

My understanding is that yes it would because the change is in the state module. But can @gtmanfred or @terminalmage confirm this?

Ch3LL on 21 Jul 2017

Can you provide an ETA for the rpm packages including this fix?

sokratisg on 23 Jul 2017

We do not have an ETA on this yet, but we are working on a 2017.7.1 release
Soon™

Thanks,
Daniel

On Sun, Jul 23, 2017 at 3:59 PM, Sokratis Galiatsis <
[email protected]> wrote:

Can you provide an ETA for the rpm packages including this fix?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/42403#issuecomment-317285227,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAssoUJjdVnmBKG6j2cO_c0XUIE7jxu4ks5sQ8JKgaJpZM4OdT_P
.