Salt: Showing progress with 'salt' command

[kfdm]

One very minor advantage Ansible has for new users is that when you run a command, ansible-playbook it gives you progress about what steps are executing.

If you run the command on the minion side with salt-call, you can get some general output by adding -l info though it's a touch noisy if you don't know what you're looking for.

If you add state_events: True to your master configuration, then you can view the general progress by running salt-run state.events though this can also be a touch noisy.

If you run salt ... there is only the final report when everything is finished.

What would it take to be able to more easily watch the progress of a running state? Is this something that would make sense as an extension to the salt command or perhaps building it as a salt-run command would be better.

The local client provides a cmd_async command that returns a jobid which could be used to filter events

• salt/job/<JID>/new
• salt/job/<JID>/ret/<MID>
• salt/job/<JID>/prog/<MID>/<RUN NUM>
  though perhaps there is a slight delay between getting the jid from cmd_async and being able to subscribe to salt.modules.state.event

Are there perhaps any other race conditions that one would need to be aware of?

Copied from: https://groups.google.com/d/msgid/salt-users/afa89988-d358-40de-a451-9417bdb90206%40googlegroups.com.

[garethgreenaway]

@kfdm Thanks for the report. There is a progress out put module that can be used to show some progress during a Salt calls. It requires that the python progressbar library is installed where you're running Salt, eg. on the master.
https://docs.saltstack.com/en/latest/ref/output/all/salt.output.progress.html#module-salt.output.progress

Real time output from the minion is tricky since they don't return their output to the master until the run has completed.

[kfdm]

--progress is one way to show that the command is still running, but I'm curious about what it would take to show more details about what is running. Unlike ansible the commands are not run in sync (some minions will finish before others) but for a proof-of-concept I'm not worried about that.

While it would require state_events: True to be set, my initial brainstorm was something similar to this

salt '*' state.apply
salt/job/<JID>/prog/A/1 file.manage /path/to/foo
salt/job/<JID>/prog/B/1 file.manage /path/to/foo
salt/job/<JID>/prog/A/2 service.running bar
salt/job/<JID>/prog/A/3 service.running baz
salt/job/<JID>/prog/B/2 service.running bar
...
Normal output.highstate

Perhaps I should try a quick implementation as a salt runner to test this idea a bit more.

[kedare]

I confirm this is more than needed.
We have some huge highstate (and on Windows...) that can take between 30 minutes and 1h to apply, it's very frustrating to not have any progress report

[dbernadett]

I have the same need.

[Oloremo]

+1

[adam12b1]

+1! And I would say, even within the current architectural constraints on getting realtime data back from minions, it would still be better than nothing to at least have a "Now running [state] on [target]..." or similar update from the Salt master as it moves through its work... although, hm, I guess that's still currently only known on the minion, eh?

[Mapel88]

+1

[bmaeck]

+1

[pthirumalachar]

+1. + if this could report not only the state transitions, but could also capture the STD outs/warning of the underlying processes being run, is very much needed. And having the ability to list only the delta output would be very helpful

[digitalformula]

+1

[damon-atkins]

The minion is polled to see if the job is running. But it does not return where it is up to. state_events: True is most likely the best way. You can push event onto slack or similar as well.

[digitalformula]

The minion is polled to see if the job is running. But it does not return where it is up to. state_events: True is most likely the best way. You can push event onto slack or similar as well.

Out of interest - how difficult would it be to add some sort of progress bar? I understand that there are reasons why SaltStack doesn't do that now, and that's 100% fine, but if there was a command line switch that showed something as simple as "Running state 3/17", that would be huge.

salt-call state.apply blender.install --progress

My use case is the cloning of source from Github, which then needs to be built (Blender, specifically). I've got it all working without issue but there's zero output until the end, when the state application has either succeeded or failed.

[damon-atkins]

find_job response should lead to how many minions are working on it.

[tdonovic]

+1 States (when im testing) and cant see why the state times out until 20 min later is really slowing things up. Being able to tail the logs to the master that executed the state would make my life incredibly easy

[amunoz951]

+1 - we have salt-calls that can take hours and it's extremely helpful to see stdout in realtime

[BeehiveSystems]

+1 - we have salt-calls that can take hours and it's extremely helpful to see stdout in realtime

Same here, this would be a very useful feature.

[NickSu86]

+1 , get used to the ansible progress output, and find it not safe enough to run a salt script without any progress and eventually throw me an error, killing me

[gigi206]

+1

[rawkode]

I'd love it if we could get real-time output when using salt-call --local.

Could this be implemented?

[tantrum-junkie]

+1

[damon-atkins]

find_job response should lead to how many minions are working on it. Maybe even return the number of states completed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Oloremo]

not stale

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[amunoz951]

Not stale

On Thu, Feb 6, 2020 at 8:39 AM stale[bot] [e-mail] wrote:

This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you
for your contributions.

If this issue is closed prematurely, please leave a comment and we will
gladly reopen the issue.

—

You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/46665?email_source=notifications&email_token=AEXQNJP44H7WH7KLWKS2C63RBQ4LDA5CNFSM4EW4AVY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK74M3I#issuecomment-582993517,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEXQNJOZH3LVVWGO5A2RFVTRBQ4LDANCNFSM4EW4AVYQ
.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[gigi206]

Not stale

[damon-atkins]

A solution would be logs per job. Instead of one main log. Or watching the event bus with a filter?
Maybe a simple 1/123 tasks stored in the job file after each job is completed, with name of the last id executed. Maybe appead the state "id" to the job file as they are executed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Oloremo]

not stale. Event-based progress bar must be a thing

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[KevoSoftworks]

+1 We have a number of systems with high resource usage peaks Being able to monitor the progress of applying states would benefit our workflow during these periods.

[theunsa]

+1 fwiw

[hemantaltair]

+1 otherwise it's very difficult to go on a minion and check logs which is not practical when you have 1000 of infrastructure.

[SJay3]

+1 We need a progress for salt

[MalloZup]

+1

[network-shark]

+1

[alexkolomolo]

+999 This is really needed for anyone wanting to use salt seriously to manage their infra.

[amunoz951]

Our organization already abandoned salt in part for this reason.
Reliability being the other.

On Fri, Dec 11, 2020 at 11:50 AM Kolomolo123 [e-mail]
wrote:

+999 This is really needed for anyone wanting to use salt seriously to
manage their infra.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/46665#issuecomment-743393750,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEXQNJKDDFKFAHTAZX55HM3SUJZZPANCNFSM4EW4AVYQ
.