Powershell: Feature Request: Async Cmdlets

[mikeTWC1984]

Summary of the new feature/enhancement

Start-ThreadJob is a great tool, although it's a bit cumbersome if you just need to invoke single cmdlet with parameters. Would be nice just to indicate this with some switch, e.g. "Async". Somewhat similar to AsJob switch, but returning task or thread job.

Example

Invoke-WebRequest -Uri $Url -Method $Method -Headers $h -Async

This should return task object, or be a shortcut to this:

Start-ThreadJob -ScriptBlock { param($Url, $Method, $h)

    Invoke-WebRequest -Uri $Url -Method $Method -Headers $h

} -ArgumentList ($Url, $Method, $h)

If not possible as general mechanism, this could be implemented for some selected cmdlets. I think this could be useful for iwr/irm, test-connection and GetWMI-Object

[vexx32]

You can do this with the & suffix operator:

$job = Invoke-WebRequest $Url -Method $Method -Headers $headers &
$job | Receive-Job -Wait

[mikeTWC1984]

@vexx I believe this is shortcut for start-job. I was thinking about runspace/task mechanism.

[SteveL-MSFT]

Isn't this a dupe of https://github.com/PowerShell/PowerShell/issues/9873?

[mklement0]

It is in _intent_, @SteveL-MSFT, but not in suggested implementation.

@mikeTWC1984, #9873 suggests a syntax that is part of the "plumbing" rather than relying on parameters supported by individual cmdlets (or even a new _common_ parameter), which to me is the better way to go.

[SteveL-MSFT]

I believe the ask is to have ubiquitous parameter to run as a threadjob, like -AsThreadJob. This would be more discoverable than a new background operator. We DO currently have both -AsJob and the background operator, so I guess we COULD have both.

[vexx32]

Making -AsThreadJob a common parameter would be... interesting. @PaulHigin, any thoughts on that?

[mklement0]

Yes, making it a common parameter would be an alternative way to make the functionality available to every cmdlet, but I don't think it belongs there _conceptually_:

It's awkward to specify an _invocation method_ - something that is extrinsic to the invoked cmdlet - as one of its _parameters_.

Note to how the existing -AsJob parameter is sensibly _not_ a common parameter, because it only applies to cmdlets that _themselves_ have invocation functionality, such as Invoke-Command.

Also, an operator such as the postpositional & is more comprehensive in that it also covers _expressions_ (to use an overly simplistic example: 1..10 &).

As for discoverability: yes, all symbol-based functionality is not easy to discover, but the reason for choosing a symbol to begin with is that the feature is so common (or is expected to become that) that it warrants maximum concision.

With frequent use and exposure comes familiarity.

Also, to help with discoverability, the feature should be documented alongside its verbose equivalent, i.e. Start-ThreadJob, just as postpositional use of & _should_ be documented in the Start-Job topic - but currently isn't: see https://github.com/MicrosoftDocs/PowerShell-Docs/issues/4965

[KirkMunro]

There's another problem with using -AsJob as a common parameter.

In PowerShell 6.x and earlier, we have -AsJob as a parameter on some commands (most notably: Invoke-Command), and it returns a PSRemotingJob for the job that is being run in the background.

In PowerShell 7, we have -AsJob as a parameter on the new ForEach-Object -parallel parameter set, and it returns a PSTaskJob.

We also have ThreadJob as a third type for jobs that are launched using Start-ThreadJob.
(Aside: I find it quite confusing that we have ThreadJob and PSTaskJob types -- do we need two types of jobs here or can we move forward with just one?)

If we were to try going the common parameter route, do we provide a common parameter for each type of job? Or have a default job type for -AsJob and another common parameter to define the job type to use?

Plus, as @mklement0 pointed out, using common parameters here feels awkward. I'd rather we control invocation of commands using post-positional operators.

For those who are interested, I opened an RFC for this a while back when I felt this was missing and would add value to the PowerShell async story. You can read that proposal and add comments in https://github.com/PowerShell/PowerShell-RFC/pull/205.

[mikeTWC1984]

I actually came to this from .net's async methods (like DoSmth()/DoSmthAsync()). As I understand from above there is already a mechanism to invoke cmdlet asynchronously without creating extra scopes (which was my main concern) and we just discuss implementation. I'm good with either operator or switch, latter sounds more intuitive to me, but I agree this could be awkward to see it on every cmdlet. Probably switch appearance could be indicated inside CmdletBinding attribute.

Btw, what is a PSTaskJob? Thread job seem to create a decent overhead, unlike task that came from async methods. Wondering if PSTaskJob can improve this. Below are some dummy examples.

This eats/cloggs 100MB+ of memory on my machine (on average)

1..100 | foreach {Start-ThreadJob -ScriptBlock {sleep 1} }

This takes 1-2 MB

1..100 | foreach {([System.Net.NetworkInformation.Ping]::new()).SendPingAsync("www.example.com") }

[PaulHigin]

This is all a great discussion.

Cmdlets aren't really designed to run asynchronously. There are threading, data streaming, and state sharing issues. Over the years PowerShell has added various ways to run script/commands asynchronously: PowerShell remoting, Start-Job, Workflows, asynchronous API.

More recently I have tried to make the async API easier to use through ThreadJobs and the new ForEach-Object -Parallel experimental feature. This has been mostly successful but has also accentuated PowerShell's inherent limitations, in particular the large overhead required to run PowerShell scripts concurrently (see my blog post for more information: (https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/).

I am a bit ambivalent about adding syntactical ways to automatically invoke cmdlets in parallel. The feedback we have received from the new ForEach -Parallel experimental feature makes clear that it does not meet many users expectations. The large overhead, often slower performance, and lack of awareness of basic threading safety makes ThreadJobs and ForEach -Parallel potentially dangerous to use, or at minimum a disappointment for some users (again see blog link above for more explanation).

I would love to build something that 'just works'. But unfortunately, due to inherent PowerShell design limitations, this is not possible and each existing concurrent model has its own benefits and side effects.

PowerShell

PowerShell ForEach-Object Parallel Feature | PowerShell

PowerShell ForEach-Object Parallel Feature PowerShell 7.0 Preview 3 is now available with a new ForEach-Object Parallel Experimental feature. This feature is a great new tool for parallelizing work, but like any tool, it has its uses and drawbacks. This article describes this new feature,

[BrucePay]

Adding -AsJob to _commands that can natively support it_ is the right way to go. It is the existing idiom for running things asynchronously in PowerShell. And regardless of the underlying implementation, -AsJob should always return a Job2 object. PowerShell jobs are the abstraction for asynchronous operations. As long as you return a Job object, the user can use the standard *-Job cmdlets to manipulate it. If you break this abstraction, then you'll end up rewriting these cmdlets for each type of asynchronous "handle". In the OP's example Invoke-WebRequest -Uri $Url -Method $Method -Headers $h -AsJob would probably return a subclass of Job2 that wrapped a Task. Not a background job object, not a threadjob but still a job object that can be manipulated with standard patterns.

[mklement0]

@BrucePay

Adding -AsJob to _commands that can natively support it_ is the right way to go.

It is, but only for "meta cmdlets" that themselves invoke _other_ commands - but that isn't the issue at hand, which is about a general-purpose, cmdlet-agnostic way to invoke _any_ cmdlet in a separate thread.

Thanks for the background information, @PaulHigin - I think it would make sense to create a conceptual help topic based on your blog post.

I am a bit ambivalent about adding syntactical ways to automatically invoke cmdlets in parallel.

There's no way around users needing to understand the constraints and appropriate use case, and burdening those that do with verbose syntax doesn't sound like the best way forward.

We already have postpositional & to make implicit Start-Job calls easy, so there's no reason not to provide the same convenience for Start-ThreadJob - see @KirkMunro's RFC at https://github.com/PowerShell/PowerShell-RFC/pull/205

[mklement0]

Regarding documentation: I've created https://github.com/MicrosoftDocs/PowerShell-Docs/issues/4977, asking that guidance be added to about_Jobs, along with extending the topic to cover Start-ThreadJob and ForEach-Object -Parallel -AsJob too.

[mikeTWC1984]

@PaulHigin Thanks for the info. So far I'm more than happy with thread jobs, it's trade-offs are quite reasonable. Question - is PSTaskJob an attempt to reduce thread overheads, or it's still something based on runspaces?

[PaulHigin]

PSTaskJob is similar to ThreadJob except that it is specific to ForEach-Object -Parallel when run with the -AsJob switch. Unlike a ThreadJob, which encompasses only one running job at a time, PSTaskJob contains child jobs for all scriptblocks invoked by ForEach -Parallel (one script block is invoked for each piped input object). So if you run 1..10 | foreach -parallel { $_ }, there will be 10 child jobs within the PSTaskJob. Note that throttling still applies so only 5 child jobs will run at a time if the throttle limit is set to 5.

But basically it allows you to use PowerShell job cmdlets on a foreach -parallel running asynchronously as a job.