Powershell: Feature Request: Allow writing to output streams from other threads

[replicaJunction]

Summary of the new feature/enhancement

As a C# developer, when writing a PowerShell frontend to my .NET Standard or Core libraries, I would like to expose my library's existing logging to PowerShell's output streams - Verbose, Debug, Progress, etc. This is already easy to do with synchronous methods, but any async method causes threading problems that PowerShell cannot handle.

Currently, when Cmdlet.Write* is called from a background thread, an error message is thrown:

The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services.

I would like the Cmdlet.Write* methods to allow writing on background threads, and handle marshalling data back to the primary thread as necessary.

Proposed technical implementation details (optional)

Here is a simple reproduction:

public interface ILogger
    {
        void Debug(string message);
        void Info(string message);
        // ...etc
    }

    public class PwshLogger : ILogger
    {
        public Cmdlet Cmdlet { get; }

        public PwshLogger(Cmdlet cmdlet) { this.Cmdlet = cmdlet; }

        // ILogger implementation
        public void Debug(string message) => this.Cmdlet.WriteDebug(message);
        public void Info(string message) => this.Cmdlet.WriteVerbose(message);
    }

    public class Widget
    {
        public ILogger Logger { get; set; }

        public int PerformTest()
        {
            Logger?.Info("Beginning test");
            Thread.Sleep(3000);
            Logger?.Info("Completing test");
            return 0;
        }

        public async Task<int> PerformTestAsync()
        {
            // Contrived example - normally, this would have some awaits in it
            var task = Task.Run(() => {
                Logger?.Info("Beginning async test");
                Thread.Sleep(3000);
                Logger?.Info("Completing async test");
            });
            task.GetAwaiter().GetResult();

            return 0;
        }
    }

    [Cmdlet(VerbsDiagnostic.Test, "Widget")]
    public class TestWidget : PSCmdlet
    {
        private PwshLogger _logger;
        private Widget _widget;

        protected override void BeginProcessing()
        {
            base.BeginProcessing();
            _logger = new PwshLogger(this);
            _widget = new Widget { Logger = _logger };
        }

        protected override void ProcessRecord()
        {
            base.ProcessRecord();

            // Works - run this cmdlet with -Verbose to see output
            _widget.PerformTest();

            // Does not work - throws the error detailed above
            Task.Run(() => _widget.PerformTestAsync()).GetAwaiter().GetResult();
        }
    }

Running the command provides this output:

PS> ipmo C:\Users\myUser\source\repos\PowerShellTest\PowerShellTest\bin\Debug\PowerShellTest.dll

PS> Test-Widget -Verbose
VERBOSE: Beginning test
VERBOSE: Completing test
Test-Widget : The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services.
At line:1 char:1
+ Test-Widget -Verbose
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [Test-Widget], PSInvalidOperationException
+ FullyQualifiedErrorId : InvalidOperation,PowerShellTest.TestWidget

I have confirmed this behavior on Windows PowerShell 5.1, PowerShell 6.2.2, and PowerShell 7.0.0 preview 3.

I've seen a few questions about this, with various proposed solutions, but they all require the entire structure of the cmdlet class to be built around their solutions.

One could argue that a PowerShell frontend shouldn't be exposing the internal logging from the module, but I'd argue that PowerShell's native output streams are perfect for that sort of thing. The user can see no logging info if they want, or they can pass -Verbose and/or -Debug if they want detailed logging data displayed.

With the popularity of asynchronous programming, I believe it would be appropriate for PowerShell to handle this case gracefully in order to better integrate with the rest of .NET.

[SeeminglyScience]

In this scenario, what is currently running in the pipeline thread? The engine cannot take control back from something that is not either script based, or is a method from the engine itself that is specifically checking for requests from another thread.

Something that makes that prospect a lot more complicated is that writing an object as output is directly tied to the invocation of the next command's process block/ProcessRecord method. Even streams other than output like error and verbose can be redirected as output. So whatever process is running to record messages from other threads would need to know when to output these messages in a way that preserves execution order.

[replicaJunction]

In this scenario, what is currently running in the pipeline thread?

The pipeline thread is blocked and waiting for the async item to complete. In my example above, it's the .GetAwaiter().GetResult() line at the very end. As far as the end user is concerned, this is still a synchronous process - it's just leveraging an async method under the hood. The method is async in the first place to allow for compatibility with other ecosystems (in my specific use case, the same library is also used for a WPF application).

The trouble is that the async item is still handled on a separate thread, even though the PowerShell engine is blocking and waiting for its completion. This causes the error message I detailed above, since it's not actually executing on the pipeline thread.

[SeeminglyScience]

The issue is that instead of GetAwaiter().GetResult(), you could be running some other code that accesses the thread static resources this exception is meant to protect. PowerShell has no visibility into what your code is doing, even if it's just waiting on code from another thread.

What you could do instead of blocking on the result, is set up a blocking collection that allows you to marshal calls back to the context of your cmdlet. Something like:

public class AsyncExample : PSCmdlet, IDisposable
{
    private readonly BlockingCollection<Action> _asyncRequests = new BlockingCollection<Action>();

    private readonly CancellationTokenSource _stopProcessing = new CancellationTokenSource();

    public void Dispose()
    {
        _asyncRequests.Dispose();
        _stopProcessing.Dispose();
    }

    protected override void StopProcessing() => _stopProcessing.Cancel();

    protected override void EndProcessing()
    {
        var task = Task.Run(() => _asyncRequests.Add(() => WriteObject("something")));
        try
        {
            while (_asyncRequests.TryTake(out Action asyncRequest, -1, _stopProcessing.Token))
            {
                asyncRequest();
            }
        }
        catch (OperationCanceledException)
        {
        }
    }
}

You need a way to gracefully "complete" and probably a few other things but that gives the general idea.

[vexx32]

@SeeminglyScience that needs to be in some official guidance for working with async in PS cmdlets!

[replicaJunction]

@SeeminglyScience - Understood on the thread safety. I like your example, but in a large project, this will become boilerplate code very quickly, especially if I need to implement this in more than one process block (both ProcessRecord and EndProcessing, for example). The intent of my feature request is that PowerShell implements and abstracts this code so a developer doesn't need to copy/paste it around in every cmdlet they write.

Perhaps there could be a new subclass of Cmdlet - something like PSAsyncCmdlet - that overrides BeginProcessing, ProcessRecord, and EndProcessing with something similar to your code?

[vexx32]

There actually _is_ an AsyncCmdlet class if I'm not mistaken?

I don't know how reliable it is or the limitations of using it, though.

[replicaJunction]

I don't think there's a current, official AsyncCmdlet class from Microsoft - or if there is, it's prohibitively difficult to find.

I found this one from the old OneGet documentation, but I can't find that information on anything current.

I also found a bare-bones unofficial one at OctopusDeploy, but that doesn't cover this use case.

Finally, here's another community version that's much more elaborate. I just found this as I was typing this comment and haven't gone through it in depth yet, so I'm not sure whether it meets this use case or not.

[BrucePay]

@replicaJunction I'm not sure I understand the motivation here. It sounds like you want to allow random threads to start writing error/verbose/information messages mixed in with what the user is actually doing. That sound's like a very confusing experience to me. On the other hand, if the goal is to support in-band user notification of asynchronous events, that is more interesting but might lead to a different approach.

[replicaJunction]

On the other hand, if the goal is to support in-band user notification of asynchronous events, that is more interesting but might lead to a different approach.

This is what I'm going for. Basically, I'd like an asynchronous, background thread to be able to call Cmdlet.WriteVerbose() and have the Cmdlet object handle any thread safety concerns transparently.

In my particular use case, I only need one callback at a time, and I block until that background worker is complete anyway. (This is the use case for which I provided a code example in the first post.) These are just implementation details, though.

I understand there would be no guaranteed order if there are multiple callbacks happening on multiple background threads - I see that as a "let the buyer beware" kind of situation. If a user wanted to have multiple background threads going at once, I don't see an issue with the warning that "these are not in a guaranteed order, and they could be confusing."