Salt: Orchestrate order of states from different targets

[bebosudo]

Orchestrate states from different targets

I'd like to setup a webserver on the salt master/frontend (managed by salt itself) to serve linux packages to my fleet of minions; to achieve this, the webserver needs to be enabled before the minions start

Setup && Steps to Reproduce Issue

# top.sls
base:
  'frontend*':
    - wait_and_touch

  '*minion*':
    - should_run_after

# wait_and_touch.sls
sleep 2s && nc -l -p 1500 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date)"':
  cmd.run: []

# should_run_after.sls
curl -s http://salt:1500/:
  cmd.run: []

When I apply the previous config with salt \* state.apply:

salt_minion_1:
----------
          ID: curl -s http://salt:1500/
    Function: cmd.run
      Result: False
     Comment: Command "curl -s http://salt:1500/" run
     Started: 15:05:48.987954
    Duration: 34.366 ms
     Changes:   
              ----------
              pid:
                  12224
              retcode:
                  7
              stderr:
              stdout:

Summary for salt_minion_1
------------
Succeeded: 0 (changed=1)
Failed:    1

(the master node hangs and doesn't return because nc blocks the execution until it receives a request. enter in a minion and paste curl http://salt:1500 to unblock it)

Question

Is there a way to make the minions wait for the frontend to end its setup?

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.13.16-100.fc25.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

Both the frontend and minion(s) are fresh CentOS docker images, running the same salt versions.

[Ch3LL]

you could run the command in the background using the bg argument for the cmd state. from these docs: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.cmd.html

bg
: False
If True, run command in background and do not await or deliver its results.

[bebosudo]

Hi @Ch3LL, thanks for your reply!
Did you mean to add bg to the "blocking" state, the wait_and_touch.sls? because it doesn't help so much.. The state is executed, but the state running on the minion still fails because the server to reach is not running on the frontend yet.

What I was looking for is some kind of synchronization between states executing on different targets, and I'm fine with having the minions blocked by/waiting for the frontend node to end spawning the webserver.
I was looking for a requisite for a state running on a different target, in this example sth like:
# should_run_after.sls curl -s http://salt:1500/: cmd.run: - require: - wait_and_touch <-- this is a state of the frontend target, so cannot be seen by a minion

Meanwhile I've also thought a couple of "dirty hacks" to solve this specific problem, in case this behaviour is not natively supported by Salt.

1. Add a blocking long polling in the minion, in a state to be executed before the minion starts contacting the frontend, something like (pseudo-bash-code, just to illustrate):
   ```# should_run_after.sls
   while curl -s http://salt:1500 != OK; do sleep 10s; done
   cmd.run:
   
   
   
   • require_in:
     
     
     
     
     
     • curl -s http://salt:1500/
     
     
     
   
   

curl -s http://salt:1500/:
cmd.run
```

1. Prepare different top files, and the first to be executed also has a symlink to /srv/salt/top.sls, and after the first top file has been executed, change the symlink and automatically run salt \* state.apply again.

Does Salt naturally support this kind of setup?

[Ch3LL]

If you are attempting to target multiple targets you can use saltstack orchestrate runner documented here: https://docs.saltstack.com/en/2017.7/topics/orchestrate/orchestrate_runner.html

Saltstacks orchestration can use requisites as well so should have everything you need in there. Does that help with your use case?

[bebosudo]

Wow! That's exactly my use case! I completely missed the orchestrate runner, now I'm gonna head back to study! :)