Salt: Some clarity on cmd.wait/watch vs. cmd.run/onchanges?

[jefferyharrell]

The documentation on cmd.run vs. cmd.wait is a little confusing to me. Here's what it says right now:

The important thing to remember about them is that cmd.run states are run each time the SLS file that contains them is applied. If it is more desirable to have a command that only runs after some other state changes, then cmd.wait does just that. cmd.wait is designed to watch other states, and is executed when the state it is watching changes.

This seems to me to be only _kind of_ true. In fact, I think I'm getting much more satisfactory results using cmd.run with onchanges than I get from using cmd.wait with watch. Am I badly misunderstanding the documentation?

Here's what I mean: Say on your minion you have a file, and if (and only if) that file changes, a local command should be run. (In my case, the file is a FlexLM license file and the command is lmreread but I think that's unimportant here.) The trivial example looks like this:

testfile1:
  file.managed:
    - name: /tmp/testfile1
    - contents: "Bring me the passengers. I want them alive."

testfile2:
  cmd.wait:
    - name: /bin/touch /tmp/testfile2
    - watch:
      - file: /tmp/testfile1

(I'm using touch here so I can easily tell whether the command has actually been run or not.) On first run, the results are exactly what you'd expect:

local:
----------
          ID: testfile1
    Function: file.managed
        Name: /tmp/testfile1
      Result: True
     Comment: File /tmp/testfile1 updated
     Started: 13:30:09.592725
    Duration: 10.936 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: testfile2
    Function: cmd.wait
        Name: /bin/touch /tmp/testfile2
      Result: True
     Comment: Command "/bin/touch /tmp/testfile2" run
     Started: 13:30:09.606103
    Duration: 9.692 ms
     Changes:   
              ----------
              pid:
                  28544
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  20.628 ms

Both testfile1 and testfile2 are created. If I remove testfile2 and run the state again, then no testfile2 appears … but I get this confusing output:

local:
----------
          ID: testfile1
    Function: file.managed
        Name: /tmp/testfile1
      Result: True
     Comment: File /tmp/testfile1 is in the correct state
     Started: 13:31:06.431885
    Duration: 10.742 ms
     Changes:   
----------
          ID: testfile2
    Function: cmd.wait
        Name: /bin/touch /tmp/testfile2
      Result: True
     Comment: 
     Started: 13:31:06.443919
    Duration: 1.047 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time:  11.789 ms

There's not much there to indicate that /bin/touch /tmp/testfile2 didn't actually execute on the minion. Comparing run #1 to run #2 I can see the difference, but you have to know what to look for, you know?

On the other hand, if I construct the state like this…

testfile1:
  file.managed:
    - name: /tmp/testfile1
    - contents: "Bring me the passengers. I want them alive."

testfile2:
  cmd.run:
    - name: /bin/touch /tmp/testfile2
    - onchanges:
      - file: /tmp/testfile1

…then on first run I get this…

local:
----------
          ID: testfile1
    Function: file.managed
        Name: /tmp/testfile1
      Result: True
     Comment: File /tmp/testfile1 updated
     Started: 13:33:45.287159
    Duration: 11.936 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: testfile2
    Function: cmd.run
        Name: /bin/touch /tmp/testfile2
      Result: True
     Comment: Command "/bin/touch /tmp/testfile2" run
     Started: 13:33:45.300506
    Duration: 9.3 ms
     Changes:   
              ----------
              pid:
                  29452
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  21.236 ms

…and on second run I get this…

local:
----------
          ID: testfile1
    Function: file.managed
        Name: /tmp/testfile1
      Result: True
     Comment: File /tmp/testfile1 is in the correct state
     Started: 13:34:19.418901
    Duration: 10.884 ms
     Changes:   
----------
          ID: testfile2
    Function: cmd.run
        Name: /bin/touch /tmp/testfile2
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 
    Duration: 
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time:  10.884 ms

…which is _much_ more useful output, at least for me. That tells me specifically that the command did not execute, and it tells me why.

But my question is, since the documentation seems so specifically to say that I should be using cmd.wait here instead of cmd.run … am I shooting myself in the foot?

Thanks.

[basepi]

onchanges with cmd.run is the better way to do it.

onchanges is actually a fairly recent addition. Before it was introduced, we had to hack together a similar solution using the watch requisite. Because of this hack, the watch requisite is largely misunderstood.

The idea behind watch is that by default, it acts exactly like a require -- the state runs normally if the watched state ran successfully. However, if the watched state has changes, then we can run additional behavior defined by the special mod_watch function. The classic example is the service state. That mod_watch function restart/reloads services when there are changes.

If you look at the code for cmd.wait, you'll see that its default behavior is to just return a basic True return structure. It doesn't execute anything. The real work is done in the mod_watch, where the command is run as normal. This approximates the behavior of only executing a command when there are changes.

Since we now have onchanges, I've been wanting to deprecate cmd.wait since it is obsolete. I just haven't gotten around to it.

I hope that answers your questions. In the meantime, we definitely need to update the docs. If that's something you have a few minutes to do, we'd love a pull request! Otherwise we'll use this issue to track that documentation update.

[jefferyharrell]

Ohhhhhhh. I see. When you explain it that way, that makes a lot more sense. It's still a little convoluted (no offense meant!) and so understandably tricky to explain in the documentation. It sounds to me as if what watch does is entirely implementation-dependent, as the saying goes. It sort of poses a "What does watch do here?" problem. As you say, in service watch triggers a restart, but in pkg it does something else entirely. (I don't know if pkg has a mod_watch; it's just an example.)

Yeah, that makes for a very tricky documentation problem indeed. I'm not sure how I would solve that one if it were entirely up to me. I'll noodle it.

Thanks very much for the explanation, though. It completely cleared up my misunderstanding.

[basepi]

I'm glad I could help! In practice, 99.9% of watch uses are with the service state. There are just not that many states which support it, it's a very specialized behavior.

[twellspring]

Looks like this has been documented in #29384

[basepi]

Thanks for pointing that out @twellspring. I'm going to close this.

[byteknacker]

Would it be possible to also add this comment on deprecating cmd.wait in favour of cmd.run with onchanges composite here: https://docs.saltstack.com/en/latest/ref/states/requisites.html

I used this one first to learn about requisite and there is a small section for onchanges but it did not contain any examples. Would be great to give an example there with this notice of deprecating cmd.wait.