Powershell: PSNativeCmdlet class - wrapper for native applications

[iSazonov]

Motivations

Very long history and fears of inevitable breaking changes

https://github.com/PowerShell/PowerShell/issues/1995

7388

1908

https://github.com/PowerShell/PowerShell/issues/13068
https://github.com/PowerShell/PowerShell/issues/12491
https://github.com/PowerShell/PowerShell/issues/12975

10722

10741

https://github.com/PowerShell/PowerShell/issues/9006
https://github.com/PowerShell/PowerShell/issues/3996
https://github.com/PowerShell/PowerShell/issues/13393

13579

https://github.com/PowerShell/PowerShell/issues/13640#issuecomment-693435355 PSNativePSPathResolution
https://github.com/PowerShell/PowerShell/issues/10509#issuecomment-678951017

https://github.com/PowerShell/PowerShell-RFC/pull/90
https://github.com/PowerShell/PowerShell-RFC/pull/88

It seems to be not all

Proposal

Introduce new PSNativeCmdlet class as an successor of the PSCmdlet class which will work as a wrapper for native apps.

Capabilities

All traditional cmdlet capabilities:

• Parameter completion
• Strong typed parameters
• Argument validation attributes
• Transformation attributes
• Parameter sets
• Dynamic parameters

Additionally:

• input/output encodings
• specific error output handling
• specific error code handling
• specific argument parsing, escaping (requires a hook in Binder)
• output parser (for strong typed output to PowerShell pipeline) #10741
• having special wrappers for specific applications like cmd.exe or msiexec
• specific hook for Suggestion System
• alternative optimizations for native pipelines

Benefits

_The main thing is that this class can be integrated into the Engine subsystems selectively and gradually without total breaking changes._
And all subsystems themselves can be enhanced gradually.
All existing Engine possibilities remain. Wrappers can overlap its or bypass to its.

The class itself can also improve gradually.

We can add flags which subsystems a particular wrapper can support. For example say that only Completions. Or run it like a regular cmdlet. Or run it as a normal external command, but use the output parser.

We could add new option in Start-Process so that it uses capabilities of the new class too.

It is compiled and works fast. (We could think about script support too)

Easy adoption

Easy adoption for #1995 - We can implement @mklement0's Invoke-NativeShell internally in PSNativeCmdlet. Users can load a module with wrappers and get new behaviors or unload the module and get previous behavior. Or we could add flags in PSNativeCmdlet which turn on/off capabilities on the fly for the specific wrapper.
Git is a good example of gradual adoption. We could implement git wrapper to address argument parsing issues and it will work with posh-git module. Later we could enhance the wrapper with posh-git capabilities in some way.

Based on telemetry we will able to decide when to make the new behavior as default for all.

We can collect wrappers for the most popular commands in a separate _standard_ module and a separate PowerShell project repository. The community will be able to quickly add wrappers and improve them. Other vendors and communities may publish modules with their wrappers for their applications.

Additional thoughts

This approach could address questions in @JamesWTruher's blog posts about how PowerShell can take better advantage of native executables
https://devblogs.microsoft.com/powershell/native-commands-in-powershell-a-new-approach/
https://devblogs.microsoft.com/powershell/native-commands-in-powershell-a-new-approach-part-2/

Since all native applications are built without a single parameter schema, these wrappers seem to be the only universal solution.

[iSazonov]

Don't replicate ALL discussions to the issue - please continue discussions in specific issues.

[vexx32]

I'm not really fully clear what this all means, but inferring what I can, it sounds like the idea is to build a whole new command processor to handle this -- among several other related tie-ins to existing subsystems.

If I'm honest, I think this is probably massively complicated to approach in this manner. It may well be a more complete approach to resolving the problems we're seeing, but I think the necessary time to put this kind of change together and review it, and then figure out how to test it... would be pretty significant.

I'm not sure the slated benefit is worth the level of complexity being proposed here. 🤔

[iSazonov]

I'm not sure the slated benefit is worth the level of complexity being proposed here.

You see a complicity only because __all__ features from different issues we want implement are enumerated at once.
Really every feature will implemented separately - so it is the same complicity as implementation of one feature. The same for testing.
For example, a parameter completion is as simple as define primitive cmdlet. I think to create a demo for this.

[vexx32]

To be clear - the surface features I think are desirable.

But I don't think the proposed implementation is the way to go, there seem to be too many pieces to fix what really boil down to 2-3 much simpler requests:

1. Argument parsing
2. Tab completion
3. Alternate ways of handling stderr/stdout

In my own opinion, it's not worth duplicating significant portions of the engine code. These issues are, effectively, either bugs or feature requests that can be resolved in a much less expensive fashion -- _if_ we're willing to tackle the issue of them introducing breaking changes here and there.

It's my opinion that this approach is significantly more expensive in all respects than simply taking the necessary cost of a couple of breaking changes.

[iSazonov]

In my own opinion, it's not worth duplicating significant portions of the engine code.

Where do you see "duplications" in Engine? The proposal says about (1) step-by-step (2) injection in some Engine subsystems.

Why do you say about complicity the work?

• What about _simplicity_ for consumers? It is very simple for user to add one method in the class and get desired behavior without thinking about how Engine works.
• Maybe it is complex for me but not for MSFT experts. You know MSFT team has limited resources and as result postponed many useful features. But they could inject the new class in completion code and community get possibility to add completers for native commands. Then they could inject the class in pipeline output and community get possibility to create output parsers for native command. And so on. Mainly - small MSFT team efforts lead to significant ecosystem improvements.

[rkeithhill]

I can't speak to the complexity of the implementation of a PSNativeCmdlet type but the issues around using native exes have been around for 15 years. I don't want to see another "bandaid" (ahem --%) applied. So, whatever gets done, I'd love to see that adequate time & thought is given to the implementation the team comes up with. But please, let's not add yet another bandaid that will require additional fixes down the road and leave users with more bandaids to determine if they should use or not.

I realize you can't anticipate everything. I suspect the team working on v1 never anticipated running on macOS and Linux. But given everything we do know, let's make sure the solution is well thought out and implemented even if it winds up being complex. Honestly, I'd rather wait a bit longer to really nail these native exe issues. In the meantime, modules like @mklement0's native could help fill the gap.

[yecril71pl]

That would require command providers to bundle manifests that PowerShell can read.

[iSazonov]

I don't want to see another "bandaid" (ahem --%) applied.

@rkeithhill With the proposal end users will run native applications as is without any additional --% and cmdlets. They only have to load the module with wrappers.

[mklement0]

@iSazonov, _command-individual wrappers_ are not the right solution to this problem.

We don't need to _wrappers_ for external executables, we just need argument passing to work, fundamentally, with any executable.

To put it differently, quoting the first part of the linked blog series:

PowerShell is a great shell, it can execute any executable, the same way that any good shell can do. No change is needed, just run kubectl and you’re done!

It's precisely because that is _not_ true that we need a fundamental fix - and that is the focus of #1995.

https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-674445995 now lays out the fix in detail:

• which is trivial _and complete_ on Unix
• which would be trivial on Windows if we limited ourselves to supporting executables that adhere to the MS C/C++ conventions ...
  
  
  
  • ... but since calling calling batch files and msiexec-style CLIs, neither of which adhere to the convention, is common, we should accommodate them too
  
  
  • for any - then truly rare - edge cases not covered by these accommodations, there's --% or the more flexible ins (Invoke-NativeShell).
  
  

The only question is: how do we make this fix available, if a _direct_ fix is - at least for now - not an option?

To be clear: a direct fix is clearly the most desirable option, given that it is a severe handicap for a shell to not properly pass arguments to external executables.

Shipping an ie _function_ as a stopgap that can later become a no-op once a direct fix is implemented is a low-ceremony (opt-in, per-call) solution that is easy to use, publicize and document:

• Assuming that the ie function from the Native module stands up to rigorous tests, it could be shipped as-is with PowerShell, requiring no engine changes whatsoever.
  
  
  
  • That said, to also make the function work with && and ||, we should implement https://github.com/PowerShell/PowerShell/issues/10917#issuecomment-550550490, so that the function can set $? properly.
  
  
• This means it could even be shipped with Windows PowerShell v5.1 and v6.x and v7.0 maintenance releases.

---

As for the other areas of functionality mentioned in this proposal, as summarized by @vexx32 above:
They would be _enhancements_, as opposed to the urgently needed bug fix required for #1995:

1. Tab completion

You don't need a new, cmdlet-derived class to get tab completion with external executables; for instance, @bergmeister's posh-cli module bundles a growing number of tab-completion modules for popular CLIs.

1. Alternate ways of handling stderr/stdout

https://github.com/PowerShell/PowerShell-RFC/pull/88 should address the fundamental lack of integration of external executables with PowerShell's error handling.

13361 just fixed the problem with 2> redirections from external executables unexpectedly being sensitive to $ErrorActionPreference = 'Stop'; a related fix regarding $? is pending: #13393

4332 would give us the ability to capture stderr output in a variable.

Overall, I'm not convinced we need engine-level support for wrapping the _functionality_ of executables, such as parsing their text output into objects.

[iSazonov]

@mklement0 All you say about #1995 right in thoughts but... PowerShell lives with this over 15 years. And you know MSFT team said a day before - "we postpone this until 7.2". You can read this "We will think about it next year". It's a polite form to say NO. So you have two alternatives - (1) stop posting about this until 7.2 (and then wait 7.3, 7.4...) (2) find an compromise solution.
And again - I don't want to turn this issue into an endless discussion - they are already there.

This issue is also about in-depth refactoring. Today @daxian-dbw and MSFT team are thinking/working on SMA refactoring to loadable subsystems. As part of the work they could thing about refactoring these subsystems because this code is old and overgrown with extensions over the years. I suppose its partial refactoring is inevitable when creating subsystems. PSNativeCmdlets proposal is a way to make internal and external design simpler and clearer without breaking changes and the implementation faster and more compact. I believe this proposal has even more possibilities than announced.

[mklement0]

@iSazonov

PowerShell lives with this over 15 years.

Indeed, and any day longer is unacceptable.

(1) stop posting about this until 7.2

I don't think there's any reason to wait; if this really won't happen until 7.2, then we have more time for discussing and honing proposals, though I think the above proposal - which took a while to work out in detail - is stable now.

In general, I believe in describing what (debatably) _should_ be the right solution, even if the powers that be decide against it.

(2) find a compromise solution.

What I'm proposing _is_ a compromise solution - one that's easy to implement in the short term.
It specifically doesn't preclude a direct, breaking fix later and is even forward-compatible with one.
It doesn't break any existing code while giving users a very simple way to get the correct behavior.

This issue is also about in-depth refactoring.

I don't see any connection between this proposal and #1995.
The changes required to fix the latter are limited to core functionality in NativeCommandParameterBinderController.cs and NativeCommandProcessor.cs.