Salt: Scheduler adds job, but job never executes!

[SaltMPA]

Description of Issue/Question

When I use
salt -G 'os:windows' schedule.add myjob function="cmd.run 'get-date >> c:\tmp\nowdate.txt' shell=powershell" seconds=5

to test the scheduler, the job gets successfully added to the schedule, but it never runs. Am I missing something?

Here's how my schedule looks after adding the job:

[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
    schedule:
      keepalivejob:
        enabled: true
        function: test.ping
        jid_include: true
        maxrunning: 1
        name: keepalivejob
        seconds: 60
      myjob:
        enabled: true
        function: cmd.run 'get-date >> c:\tmp\nowdate.txt' shell=powershell
        jid_include: true
        maxrunning: 1
        name: myjob
        seconds: 5

I am trying to schedule calling highstate in a scheduled way. So far I have only tested states by manually executing them like:

[root@cloud-salt02 mpa]# salt -G 'os:windows' state.sls mpa.services

I want to do that using the builtin scheduler in SaltStack, but I can't get it to work. Should I use cron instead?

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
Salt: 2017.7.1

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: 2.0.2
gitpython: 2.1.5
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.4.3
pygit2: Not Installed
Python: 2.7.5 (default, Nov 6 2016, 00:28:07)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.3.1611 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-514.21.2.el7.x86_64
system: Linux
version: CentOS Linux 7.3.1611 Core

[root@cloud-salt02 mpa]#
[root@cloud-salt02 salt-winrepo-ng]# salt-run manage.versions
Master:
2017.7.1
Up to date:
----------
test-mpa01.cloudstack.dk:
2017.7.1
test-salt01.cloudstack.dk:
2017.7.1

[gtmanfred]

first, since you are using redirects, you will probably want to use cmd.shell instead of cmd.run

Second, you need to pass get-data and shell=powershell to the correct variables, otherwise they get tacked onto the function.

salt -G 'os:windows' schedule.add myjob function=cmd.shell job_args='["get-date >> c:\tmp\nowdate.txt"]' job_kwargs='{"shell": "powershell"}' -l debug

This will result in a scheduled job that looks like this.

[root@salt ~]# salt-call schedule.list
local:
    schedule:
      myjob:
        args:
        - "get-date >> c:\tmp\nowdate.txt"
        enabled: true
        function: cmd.shell
        jid_include: true
        kwargs:
          shell: powershell
        maxrunning: 1
        name: myjob

And will be able to run.

[SaltMPA]

Thank you for your quick and precise answer.

However, when I run the command it says cmd.shell is not available. And I have 2017.7.1. :/

[root@cloud-salt02 mpa]# salt -G 'os:windows' function=cmd.shell job_args='["get-date >> c:\tmp\nowdate.txt"]' job_kwargs='{"shell": "powershell"}' -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: cloud-salt02.cloudstack.dk
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: cloud-salt02.cloudstack.dk
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Popen(['git', 'version'], cwd=/srv/salt/mpa, universal_newlines=False, shell=None)
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'cloud-salt02.cloudstack.dk_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/8a/1e3f485d6e2b322cd5dfabb4f8ad38400b76afb4bb64b917ad2f0a152b572c/.minions.p
[DEBUG ] get_iter_returns for jid 20171023165044326481 sent to set(['test-salt01.cloudstack.dk']) will timeout at 16:50:49.333671
[DEBUG ] Checking whether jid 20171023165044326481 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'cloud-salt02.cloudstack.dk_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] retcode missing from client return
[DEBUG ] jid 20171023165044326481 return from test-salt01.cloudstack.dk
[DEBUG ] return event: {'test-salt01.cloudstack.dk': {'jid': '20171023165044326481', 'retcode': 254, 'ret': "'function=cmd.shell' is not available.", 'out': 'nested'}}
[DEBUG ] LazyLoaded nested.output
test-salt01.cloudstack.dk:
'function=cmd.shell' is not available.
[DEBUG ] jid 20171023165044326481 found all minions set(['test-salt01.cloudstack.dk'])
ERROR: Minions returned with non-zero exit code
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
----------
schedule:
----------
[root@cloud-salt02 mpa]#

[gtmanfred]

Hrm, you might need to use cmd.powershell on windows.

[SaltMPA]

You are right. Also, I noticed that the backslashes needed to be escaped on a Windows minion.
Here's what worked for me:
salt -G 'os:windows' schedule.add myjob function=cmd.powershell seconds=5 job_args='["get-date >> c:\\tmp\\nowdate.txt"]' job_kwargs='{"shell": "powershell"}' -l debug

Thank you for your answer, you can close the case now. :)

[SaltMPA]

I have a related question: What is the recommended way to schedule periodical invocations of state (highstate or a substate)? I thought it must be builtin Saltstack scheduler, but now I see a page in your documentation talking about using cron to run highstate etc.
https://docs.saltstack.com/en/latest/topics/tutorials/cron.html

We use cron only in cases where the builtin scheduler cannot be used for very detailed custom date settings and we want to avoid using cron because it also runs non-Saltstack jobs and we want to keep the Saltstack jobs separate if possible.

[SaltMPA]

Also, now that I am using the Saltstack scheduler to run states, I see the 1 Windows minion very quickly getting too busy for the salt master to query it.
I get complete unresponsiveness:
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]
[root@cloud-salt02 mpa]# salt -G 'os:windows' schedule.list
test-salt01.cloudstack.dk:
Minion did not return. [No response]

The state it runs is very simple and just meant for testing:
services.sls:
ensure_primary_service:
service.running:
- name: wuauserv

ensure_secondary_service:
service.running:
- name: XblGameSave

stop_secondary_if primary_fails:
service.dead:
- name: XblGameSave
- onfail:
- service: ensure_primary_service

I use another state to ensure the scheduling of periodical state invocation:
myjob:
schedule.present:
- function: state.sls
- job_args:
- mpa.services
- seconds: 60
- splay: 10

It looks like an instance of this issue:
https://github.com/saltstack/salt/issues/35470

Should I use cron?

[gtmanfred]

You can use cron, but I bet you run into the same problem

The windows minion is significantly slower than the linux minion.

On linux, we have access to a kernel function called os.fork this forks the process, and we get the same memory namespace in both places, for windows, we don't have anything like this so when the minion process forks off to run the state, all of the modules need to be reloaded.

So once every 60 seconds is probably too often.

What I would recommend doing is using beacons. You could use the service beacon to check when a service stops or starts, and send an event back to the master, which would be able to trigger a state then that puts the service back into the correct state. This would remove the need to run that service state on a schedule.

I hope that this works using windows, since it should just use the service module...

https://docs.saltstack.com/en/latest/ref/beacons/all/salt.beacons.service.html

But it might require some changes to work with windows, at which point you could distribute it using dynamic modules

https://docs.saltstack.com/en/latest/ref/file_server/dynamic-modules.html

[SaltMPA]

Thank you for your answer, it was very helpful! :)

[gtmanfred]

If that beacon doesn't work out of the box with windows (I think it might), if you make changes we would greatly appreciate seeing them upstream!

Thanks!
Daniel