Salt: idea: display more info while doing job

[alekibango]

Sometimes, running salt takes forever (minutes or even many minutes) to finish...
For whatever reasons.. (too many files opened, minions under heavy load, ... salt master process crashed...)

It would be nice to display some info in 'current' line so i would know - ha, we are waiting for ABC.
The current line could be erased each time something new is printed, visible when we wait over 5 seconds, with some useful information...

Just staring at cursor is not helping much.
I am not totally sure what and how to display, but i think someone else might help with that.

For inspiration, run dstat 10 and watch it for a minute.

[ghost]

+1
I agree some kind of progress bar or dots would be a good idea. Otherwise you just have to wonder if it's time to go digging around to see if something bombed. I'm always a little more impressed when software goes to the trouble of printing some neatly formatted progress bumps.

[thatch45]

I thought we printed some of this in verbose mode, I am probably getting confused with something else.

Sorry I did not comment on this sooner! It slipped through, so much is going on now!

[alekibango]

while --verbose is really great, it displays info after all is done.
I wanted something to see during execution. just watching cursor blink is making me nervous. A progressbar makes people feel better. They even might feel like they are having the process under control :).

Sometimes waiting takes ages, especially when salt-master dies for whatever reasons, or waiting for dead or slow minion. Just seeing which (or how many if there is too much) minions are working on it and who(how many) didnt answer at all would be cool. When some minion is not answering fast enough, i would like to get it hinted as soon as possible, and without using ctrl-c and separate salt-call manage.down command.
And some new answer comes in, the status line should dissapear, only coming in again after some waiting (2-4secs).

example of what could be seen after running a salt command (only one of many possible solutions, i bet you can invent better):

salt -t 20 \* test.ping

server001.domain.net: True
server002.domain.net: True
server003.domain.net: True
server005.domain.net: True
INFOLINE: working: 34   |  not answering:  server04.domain.net, server09.domain.net 

which turns later to:

...
INFOLINE: working: server28.domain.net, server29.domain.net  |  not answering:  server04.domain.net

and finally:

server001.domain.net: True
server002.domain.net: True
server003.domain.net: True
server005.domain.net: True
server007.domain.net: True
...
server009.domain.net: True
...
server 030.domain.net: True
STATUS REPORT:
  servers which failed to answer: server04.domain.net
  servers which failed to deliver results before timeout: server29.domain.net

Possibly it could even distinguish servers not talking at all (offline) from servers which failed to finish the job before timeout (slow).

This progress/infoline might make problem to scripts which read salt stdout and parse it, so there should be a way to either disable or enable it.

[thatch45]

Yes, this is a potentially more complicated request, primarily because most minion functions do not give any feedback in the middle of execution.

[ghost]

Getting into the actual running jobs' status is a little more than I would expect. I was thinking just having the master print dots or "waiting.." every 10 sec during long commands issues with a long timeout. Perhaps a synchronous timeout option to say 'wait for this one', maybe this can imply the verbose option.

[ghost]

the -s (static) option could be shined up a little to do that

[thatch45]

Good ideas, I will most likely get to this early next week.

[alekibango]

do not hurry.. Maybe the issue (which prompted this idea) of salt processes stopping to communicate is more important, rather than this. when i run salt ... command in a loop, way too often stops displaying anything after few minutes. I am not really sure why and who to blame, but i need to restart my master and start my minions way too often to like it...

[thatch45]

What versions of salt and zeromq are you running?

[alekibango]

salt- tried many versions starting from cca 0.10-1
my last is my custom package from git Oct. 01 -- 50b291ff9c1e051425aab2583fb815c1dc35c093

zeromq from debian wheezy
ii libzmq1:amd64 2.2.0+dfsg-2 amd64 lightweight messaging kernel (shared library)
ii python-zmq 2.2.0-1 amd64 Python bindings for 0MQ library

i should probably write another issue for this, but i was not really sure how to describe whats wrong...

[thatch45]

Those are the right versions...
can you try setting the sub_timeout option to 0 on your minions to see if the issue goes away? This is a ZeroMQ issue we have been working on for a while, any info you can gather is invaluable, especially any network information

[alekibango]

please see #2229
i will try that sub_timeout today

[alekibango]

thatch45: i didnt try sub_timeout, as i first upgraded and tried to run salt using runit
And suddenly i am ok! No problems in few days! (on debian wheezy)

[thatch45]

Thats good to hear, I will look into how runit works!

[Omeryl]

Well, this never went anywhere I suppose?

[cachedout]

@Omeryl We did implement a progress bar somewhat recently, actually. See salt --progress. It requires that the progressbar python package be installed.

[Omeryl]

Oh neato, thanks for pointing that out @cachedout. Another defunct issue closed then.

[Omeryl]

@cachedout OK, so it's not exactly what I was expecting, I still want the overall output at the end. Is that possible?

[cachedout]

@Omeryl I'm sure it can be arranged. :] Could you file a separate issue outlining exactly what you'd like to see?

[ghostsquad]

how do you do this with highstate? --progress doesn't work