Powershell: Write-Progress and $ProgressPreference

[vexx32]

Write-Progress has long been perhaps the most neglected of the standard Write-* cmdlets. The problems with it are both in terms of its user-facing and in terms of its behind-the-scenes implementation, and I'll attempt to break them up as such.

Behind-the-Scenes Problems

• There is no Progress stream. This makes it the only default Write-* cmdlet without a matching stream (barring of course the special-cased Write-Host, which utilises the Information stream).
  
  
  
  • I believe there should be a stream, and that stream should be handled in a similar way to the Information stream in terms of the objects that pass through it (though naturally Progress displays by default, unlike Information).
  
  
  • Providing a stream for this permits suppression and redirection of the progress information from specific cmdlets instead of requiring that an implementation continually override $ProgressPreference and hope that it works.
  
  
  • The fact that it is stream-like is more confusing for users, given that it seems to respect $ProgressPreference variable values that mimic the standard stream preferences' available levels, but it cannot be redirected and there is no common parameter for handling it as with all the other stream-type Write-* cmdlets.
  
  
• The $ProgressPreference variable is not handled the same as, for example, $ErrorActionPreference, or indeed most other preference variables that work with streams; it seems to be simply a variable that the Write-Progress cmdlet looks to see is defined and acts accordingly.
  
  
  
  • Because Progress is not a stream, the only way to attempt to handle, suppress, or indeed do anything at all with the Progress data is via this variable. This makes it incredibly unwieldy, because in order to tell a command not to show a progress bar, the variable needs to be modified both before & after the command (in order to restore it to normal order).
  
  
  • Combined with this, there's no way to check if the preference is being altered by the caller of the cmdlet (i.e., as you can do with $PSBoundParameters['ErrorAction']) or if the value is simply stored in the $ProgressPreference variable from... really, no one knows where, for sure.
  
  

User-Facing (Usability) Problems

See this project: https://github.com/mazzy-ax/Write-ProgressEx

Write-Progress is probably the only (that I've seen...) default Write-* cmdlet that spawned an entire project to help deal with its idiosyncrasies. Write-Information has spawned a couple, to allow easier interaction with the necessary metadata that sets display colors on those objects (but that's a separate issue), but pretty much no other default stream / associated Write- cmdlet has received or needs this much coddling from the community.

We need only look at the README.MD file on the above linked repository for a fairly comprehensive summary of the present pain-points with the default Write-Progress cmdlet that the creator felt it necessary to fix.

Write-ProgressEx extends the functionality of the standard powershell cmdlet. Write-ProgressEx is a powershell native cmdlet that provides a simple way to show ProgressBars with PercentComplete and SecondsRemaining.

The cmdlet:

• works with pipe;
• works with empty activity string;
• completes all inner progresses if no parameters;
• automatically completes with pipe;
• automatically calculates percents, remaining seconds and elapsed time;
• automatically displays current iteration and totals on progress bar;
• automatically set parent id for a inner loop;
• stores totals, current values and actual parameters into the module hashtable;
• provides get/set cmdlets to access actual parameters;
• uses script blocks to show messages with date, time, iterations and elapsed time on events:
• first iteration;
• activity changed;
• status changed;
• completed.
• provides a counter functional. See Write-ProgressEx as a counter;
• uses the caller function name or the caller script file name as the Activity;
• accepts -ShowProgressBar Auto parameter to reduce the overhead for redrawing a screen. It recognizes None and Force values also.

The sheer amount of manual calculations required to use the default Write-Progress cmdlet is pretty absurd, in my opinion. We don't need to determine for ourselves the basic functional information or metadata for any default stream-writing cmdlet, really.

There is so much manual input the user is forced to almost completely create the progress bar, and all Write-Progress is doing is working out purely the display issues. This makes some sense, but it doesn't seem to make any sense to me to make it such a difficult command to work with. The current implementation of the Write-Progress cmdlet seems more apropos for an internal utility command to support a fully-fledged Write-Progress cmdlet, and I would argue appears to have been designed as such.

And if you think it's not that important, because it doesn't get that much use -- there's a reason it doesn't see much use: it's really hard to use decently well. A related sidenote here is that it probably needs some capability to measure things otherwise opaque to the user -- for example when Copy-Item is used, it's relatively difficult to figure out how to measure the progress of copying, say, a single large file. Something would likely need to be worked out for that to be more intuitive for users.

Environment data

Written as of PS 6.1.0

/cc @SteveL-MSFT @mklement0

[SteveL-MSFT]

@vexx32 thanks for taking the time to write this up. I agree with what you are saying. My only concern is this is not a small work item.

[vexx32]

Well, no, of course not. 😄

It would be unrealistic to expect it all fixed at once, but I think it's well past the time where the framework for this should be in place.

It may be worth reaching out to the author of the aforementioned module and asking if they are interested in potentially translating portions of their module to be incorporated into the native Write-Progress cmdlet.

I think the most sensible overall course of action is:

1. First have it be treated with parity to the other stream-writing cmdlets in terms of implementing appropriate stream handling and object classes for it.
   
   
   
   • I have... some ideas... in terms of what the framework for what you guys may want the ProgressRecord object to consist of.
   
   
   • Unlike the existing streams, the Progress stream would likely need some measure of concurrency and may be best handled inside a secondary runspace entirely -- there are some cases where multiple progress displays may need to be handled at once, for example, as well as the fun chaos that can happen with nested progress displays.
   
   
   • Additionally, the stream may need to be written such that the objects in the stream can be accessed and written to by the cmdlet in order to update them -- complicated. Potentially: retain ref items for things submitted to the stream for display?
   
   
2. Once the framework behind the scenes can be trusted, then the next step is to rework the display formatter for the progress stream and decouple it from the cmdlet itself. There are some edge cases there that we may want to deal with at that point.
   
   
   
   • e.g., should / can the display change with X amount of progress items currently being shown? Squash progress bar(s)? Should the stream be given some kind of reference that can indicate whether the stream is done with, or is that burden on the caller (i.e., Write-Progress)? Timeout values? How do we avoid leaving "dead" / "abandoned" items in the display? Etc., etc.
   
   
3. Once we have a relatively consistent handling of the display and of how it works with both cmdlets and functions that want to call out to the stream, then we would have the room and the backing implementation more or less ready to implement the more user-facing fixes.

But, of course, several of those are fairly large ??? in terms of how long they would take and the finer details of their implementation.

That said, however, I think that although PowerShell's focus is automation, it is relatively important to be capable of supporting a consistent and accessible progress display. Currently, the verbose and other streams are, I would think, somewhat overburdened, because there is no other effective and simple way to really report progress on a large job.

[mklement0]

NIcely structured write-up, @vexx32

Supporting -ProgressAction as a _common parameter_ readily makes sense to me, not least because advanced functions inside modules do not see the caller's _preference variables_ - see #4568.

Knowing little about the current Write-Progress and how it could be improved, what is the benefit of creating a progress _stream_, given the _ephemeral_ nature of progress display?
_Collecting_ the stream output wouldn't make much sense, for instance.
I'm sure I'm missing something, but perhaps you can flesh out the rationale for that, and for who would act on the stream when.

[vexx32]

My thought is essentially that having it mimic the current stream natures is a good parallel and provides more flexibility. For example, currently only one command can really display progress at a time. Providing the capability for more commands to display progress (potentially as sub-progress-bars?) at the same time permits a more verbose display if needed without excessively heavy amounts of Write-Verbose within a script which leaves you with screens of yellow text that sometimes manages to be harder to make sense of than just a blinking cursor with no indication of what's going on.

Granted, you may not always want all of these to display; potentially we could deviate a bit from the standard "action" parameters of ErrorActionPreference and the like for something more suitable, perhaps setting the number of 'nested command progress bars' that are displayed with the main command as a preference variable.

I am more than open to alternative ideas, and as you point out _completely_ copying the existing stream behaviour is not likely to yield a worthwhile result, as I have previously touched upon with a few points here and there. 😄

[Jaykul]

If we did not have Write-Progress -- would you consider doing any of this work to add it?

What's the purpose of the new stream? Obviously everything written to it currently is _inherently_ ephemeral and thus, it's _never_ used for information which needs to be logged -- in fact, logging the information would almost always result in huge logs with very little value.

I'm not inherently opposed to the idea of making progress a full-blown stream, but I feel like I should defend the current implementation a little:

1. The most important thing about progress is that it needs to be VERY FAST.
2. There's a lot of value in things that are _like_ other things, even if they're not perfect. People intuitively know how to use streams and deal with ActionPreference variables, even when not all of the features of streams are available.
3. There is _also_ value in having an ephemeral status display aside from output streams. Think of it as the one pure "user interface" component of PowerShell -- we throw things out there that are rapidly out of date and which are not useful once they're out of date. Does anyone really want thousands of rows of "{current} bytes of {total} bytes" in a log file?

Having said all that ... if we did want to log-enable progress output ...

What if we harness the information stream?

That is, rewrite Write-Progress the same way Write-Host was written:

• Write-Host: creates a HostInformationMessage to hold a few extra pieces of information (color, newlines, etc) and outputs it to the InformationStream with the tag PSHost.

• Write-Progress: create a HostProgressMessage to hold a few extra pieces of information (percent, parent/child, etc), and output it to the InformationStream with the tag PSProgress.

Then hook the host up to pull that out and display it the way we display it now...

[vexx32]

I think that idea is a good one. But we probably don't want to log progress records to transcript, so they'd have to be filtered out.

[iSazonov]

Previously, Write-Progress slowed down scripts and we made a lot of effort to fix this. There are a few more problems. See the tracking Issue #3366.
Because of allocation problem I'm sure it should not be a stream.
And we could consider async methods to resolve some problems with many runspaces and locks.
And take in account a new remoting batching #8038
It is not clear for me how we'll process background jobs.

[Omio]

Maybe have Write-Progress become a sub-task of the host task performing an operation, and have Write-Progress use local-scope variables from the host task to output window-only messages, that don't output to event logging, bar for the standard messages that the host task would normally output (if any)?

[felixfbecker]

Forgive me if it has been mentioned before, but Write-Progress is also _freaking slow_ if the script outputs anything. I have a script that runs through a long loop and on every iteration logs a warning and updates the progress, runs 20s if run with either of warning or progress enabled, but takes 1h 5min 43s if run with both warning and progress on!

[iSazonov]

@felixfbecker If you see the problem in latest PowerShell Core please new Issue and link to #3366.