Powershell: Separate out Parser as independent nupkg

[bergmeister]

Problem statement:

• I would like to use a self-contained version of the Parser in an application that uses the full .Net Framework (4.7.2), however, due to the System.Management.Automation NuGet package being compiled and published against netcoreapp2.1, I cannot use it in full .Net, even when trying to use it in a .Net Standard project and publishing a self-contained version of it. I would like to have a compiled version of it to be able to run it in full .Net, even limited functionality (i.e. only having the Parser working but not being able to host a runtime) would be OK for my scenario.

Would it be feasible (with reasonable effort) and would a PR be accepted to allow compilation and publication against netstandard2.0? Or is there a better way of using and shipping a self-contained version of the Parser in full .net, i.e. to not having to depend on the installed version of PowerShell? @SteveL-MSFT

As far as I can see the dependent project Microsoft.PowerShell.CoreCLR.Eventing would need to be changed to target netstandard2.0 as well. A quick experiment shows that most compilation errors can be resolved by adding the System.Reflection.Emit, System.Reflection.Emit.Lightweight, System.Memory, System.Collections.Immutable and System.Runtime.Loader NuGet packages to SMA, the remaining compilation errors that would need to be resolved are only about the EnumerationOptions type not being available in .Net Standard and seemed to be not too difficult to handle via conditional compilation to use an overload of Directory.GetDirectories without EnumerationOptions the (and by looking at the history, those new API calls were only recently added by @daxian-dbw as an optimisation)

D:\PowerShell\src\System.Management.Automation\engine\Modules\ModuleUtils.cs(17,43): error CS0234: The type or namespace name 'EnumerationOptions' does not exist in the namespace 'System.IO' (are you missing an assembly reference?) [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\Modules\ModuleUtils.cs(24,43): error CS0234: The type or namespace name 'EnumerationOptions' does not exist in the namespace 'System.IO' (are you missing an assembly reference?) [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]

After resolving them, I suddenly got a lot more compilation errors, some I can resolve, but some about dynamics are a bit tricky and don't have a blame history other than the initial commit (not even in the source depot branch) to figure out how code was converted from full .net to netcore as a hint on how to make it netstandard compliant.

D:\PowerShell\src\System.Management.Automation\engine\ExperimentalFeature\GetExperimentalFeatureCommand.cs(103,50): error CS1061: 'Dictionar
Add' and no extension method 'TryAdd' accepting a first argument of type 'Dictionary<string, ExperimentalFeature>' could be found (are you m
\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\Modules\ModuleSpecification.cs(314,20): error CS0103: The name 'HashCode' does not exi
ion\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\Modules\ModuleCmdletBase.cs(2407,37): error CS1503: Argument 1: cannot convert from 'c
m.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\Modules\ImportModuleCommand.cs(643,53): error CS1503: Argument 1: cannot convert from
tem.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\namespaces\EnvironmentProvider.cs(201,36): error CS1061: 'Dictionary<string, DictionaryEntry>
TryAdd' accepting a first argument of type 'Dictionary<string, DictionaryEntry>' could be found (are you missing a using directive or an ass
em.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(431,42): error CS0656: Missing compiler requ
 [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\parser\PSType.cs(1237,56): error CS1061: 'TypeBuilder' does not contain a definition f
 argument of type 'TypeBuilder' could be found (are you missing a using directive or an assembly reference?) [D:\PowerShell\src\System.Manag
D:\PowerShell\src\System.Management.Automation\engine\parser\PSType.cs(1240,75): error CS1061: 'TypeBuilder' does not contain a definition f
 argument of type 'TypeBuilder' could be found (are you missing a using directive or an assembly reference?) [D:\PowerShell\src\System.Manag
D:\PowerShell\src\System.Management.Automation\engine\DscResourceSearcher.cs(136,25): error CS0656: Missing compiler required member 'Micros
\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(2871,46): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(2927,42): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3018,42): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3150,37): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3227,38): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3275,42): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3326,46): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3405,38): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3463,46): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(3526,38): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(4107,40): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\CommandCompletion\CompletionCompleters.cs(4623,32): error CS0656: Missing compiler req
' [D:\PowerShell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\PSVersionInfo.cs(647,34): error CS1503: Argument 1: cannot convert from 'char' to 'str
.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\PSVersionInfo.cs(647,59): error CS1503: Argument 1: cannot convert from 'char' to 'str
.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\PSVersionInfo.cs(647,84): error CS1503: Argument 1: cannot convert from 'char' to 'str
.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\runtime\Operations\MiscOps.cs(2707,29): error CS0656: Missing compiler required member
hell\src\System.Management.Automation\System.Management.Automation.csproj]
D:\PowerShell\src\System.Management.Automation\engine\runtime\Operations\MiscOps.cs(3017,32): error CS0656: Missing compiler required member
hell\src\System.Management.Automation\System.Management.Automation.csproj]

Is it easier to try to make it compile it directly net472?

[SteveL-MSFT]

The use case is valid, but currently it's probably not trivial to get SMA.dll and dependent assemblies to build targeting .Net Std 2.0. We should eventually see what that would take and add that as a supported runtime in the SMA nupkg. For now, unless you want to complete this work and submit as a PR, it would be easier to multi-target net472 and netcoreapp21 (assuming you want it to work in PSCore6.1).

[iSazonov]

System.Reflection.Emit and System.IO.Enumeration will in .Net Standard 2.1
https://github.com/dotnet/standard/tree/master/docs/planning/netstandard-vnext#major-features

We could wait .Net Standard 2.1 - with it the solution will be easier.

[bergmeister]

I tried compiling against net461 today (net472 was more difficult due to missing APIs), some code inside #if !CORECLR seems to not work any more due missing implementations, like PSSnapin (yes, I know, they are not supported any more), or InternalMISerializer, AdapterCodeMethods, etc., so I just commented some of that code out for a proof of concept (the work is here).
After some work I am now left with only 1 compilation error, which is due to the missing implementation of PowerShellAssemblyLoadContext in the ClrFacade here and I could not find an implementation of it in the code base for full .net (I also looked in the source depot branch for it). Any help/pointers would be appreciated.

[SteveL-MSFT]

The PowerShellAssemblyLoadContext code was originally wrapped by #if CORECLR pragmas as it wasn't needed by .NET Framework.

[iSazonov]

@bergmeister I guess you want to get only AST tree, do not you? In the case we could try to separate the Parser code and don't compile whole SMA.

[bergmeister]

@iSazonov Yes, just having the Parser only would be sufficient and I already tried briefly if I could extract that code only but it did not seem to be much easier (due to coupling with security and runtime classes) unless you have some specific ideas on how to approach it.
But being able to compile SMA against full .net or .net standard would be more beneficial to the project itself I guess (because someone else could take it further afterwards and compile also the rest against full .net or mono and create a new version of PowerShell)

I had some success with Steve's tip but after that was fixed, more compilation errors started to appear so I am working through those at the moment

[iSazonov]

I believe that the main direction of development is migration from .Net Framework to .Net Core. So I would not invest in compatibility with .Net Framework. Seem some of .Net Standard 2.0 is already lacking in .Net Framework. With .Net Standard 2.1, the outbreak will increase.
I expect that PowerShell Core will be consistent (compiled) with .Net Standard Future in a few years and again the outbreak with .Net Framework will increase.
Also if we'll add improvements in Parser with Span\, Memory\ and Buffers we can not compile the Parser in .Net Framework at all.

[bergmeister]

@iSazonov Yes, I can already see that .Net Core is ahead of .Net Framework in terms of API calls. Yes, .Net standard would be the more ideal solution but as Steve said, it would be quite tricky with netstandard2.0, therefore multi-targeting .Net Core and Full .Net is probably easier for me achieve my goal of getting it to run on full .Net, but I will see and report how I get along
Until .Net Core 3.0 is out, the full .Net framework will still be predominant.

[iSazonov]

@bergmeister MSFT announced .Net Core 3.0 Preview will be released later _this_ year. Seems .Net Standard too. I do believe that it is more profitable for you to start working in this direction now.

[bergmeister]

I have a working prototype in my branch here that needed special treatment only in a few places and some WIndows PowerShell Code to be brought back in order to compile against net472 AND not define the CORECLR constant (the latter is optional and is up for discussion if it should be defined in this case or not; as an alternative, one could've also just removed some #if !CORECLR sections that required those treatments, some of them like PSSnapins would even make sense to be completely removed).
Therefore I do not see much of a burden to re-enable compilation against net472 and fullclr with only a compilation check in CI. I saw only 5 method calls that used new .Net Core 2.1 APIs that are not available in net472 and for some of them I created adapters using extension methods, the rest (like e.g. Spans) could all be brought back to full .Net via NuGet packages. Not setting the CORECLR constant for compilation against full .net requires to either remove some fullclr code or to bring back some fullclr code that is just not compiled at the moment or in the source-depot branch. Compiling with the CORECLR could be possible as well as far as I have seen.
I do not expect full support from MSFT for an SMA package compiled against net472, but having it in the mainstream branch and it being published by MSFT would be helpful.
Since compilation against netstandard is not trivial and and netstandard2.1 is not even out yet, I'd agree with @SteveL-MSFT that multi-targeting for net472 (with the coreclr or fullclr option?) is the easiest option where I can issue a PR now
Separating out the Parser seemed to me to be quite difficult.
Can MSFT please state if it is OK to multi-target SMA for net472 for the moment and if the CORECLR compile constant should be set in this case.
cc @TravisEz13 @SteveL-MSFT

[SteveL-MSFT]

@bergmeister I don't think we want to bring FullCLR code back into the code base as it makes changes complicated and we don't validate the FullCLR code. It would be better if you need this to fork the project and maintain what you need until netstandard2.1 is available and (ideally) resolves the need to multi-target

[bergmeister]

@SteveL-MSFT OK, I will then not bring FullCLR code back and I am OK with this decision as I can understand the reasoning. But are you also saying that
a) Code that was written for the CORECLR would actually work with the fullclr that net472 uses under the hood? My initial thoughts were that some of the code is coreclr specific.
b) That a PR that multitargets netcoreapp2.1 and net472 would be acceptable if the compilation for net472 also defines the CORECLR constant, i.e. uses the coreclr code? Or would you still prefer to wait for netstandard2.1 in this case as well?

[SteveL-MSFT]

Until we have netstandard2.1 I can't say what work is needed to make it work under net472. There is another effort to refactor SMA.dll making it smaller for hosting PowerShell. This may be what you actually need although my initial thinking was for .NET Core, but perhaps we should consider .NET Framework as well (cc @BrucePay). I'm hoping an RFC for this will be published soon.

In general, we've been trying to clean up the #if CORECLR or #if !CORECLR out of the code base to make it more maintainable, so adding back changes to enable multi-targeting netcoreapp and net472 would be counter-productive.

[bergmeister]

@SteveL-MSFT Details about netstandard2.1 were recently announced here and the gist is that only .Net Core 3.0 will implement it but not the current 2.1 and 2.2 versions and neither will the full .Net Framework (even 4.8 was ruled out).
Since the current state of SMA is far away from being able to work using netstandard2.0 (due to special compiler binding calls and Spans), the only alternative is to multi-target net472. I already did a prototype a few months ago and there are actually not too many adaptions that are needed since most of the new features can brought to net472 via NuGet packages (e.g. Spans), in total there are only around 10 cases where a slightly different API call is needed or an extension method needs to be created to adapt for net472. Would you accept such a PR where I just multitarget SMA for net472 as well but use the PowerShell Core code (i.e. the CORECLR will be defined)? Alternatively, separating out the parser into its own project as discussed by @rjmholt here would be a viable alternative (the use case for me would be to be able to get rid of a dependency on WIndows PowerShell in a Desktop WPF app that will never be able to go to .Net Core due to a dependency on Workflow foundation)

[SteveL-MSFT]

@bergmeister unless there are other use cases to multi-target, it seems that separating the parser as it's own nupkg may be a better long term solution

[iSazonov]

I discover that the parser intersect with HelpSystem. We need to investigate how split them in better way. Can we get ExecutionContext (there HelpSystem is initialized) in the parser (ast)?

[bergmeister]

Ok, I'd be happy with a Parser NuGet package that compiles against netstandard2.0 or net472 but separating the Parser into its own project seems to be a non-trivial task that is probably too hard for people without in-depth knowledge/experience and thus requires a lot of effort. Just moving the engine/parser and engine/lang folders from SMA into a new project that references the same projects/packages as SMA results in more than 200 compilation errors due to missing types in SMA, some of them like LightCompiler or Interpreter should maybe (?) not be in there and other types in the engine folder and it seems quite a bit of refactoring is required...

[SteveL-MSFT]

@bergmeister for sure separating out the parser is a non-trivial task. Targeting net472 is problematic as we've be removing #if !CORECLR to have a cleaner code base targeting .NET Core and started using new types that aren't available in .NET Framework.

[iSazonov]

Over time, the gap between net standard and .net framework will only increase (last annouce is that .net framework will do not support netstatndard 2.1 but PowerShell Core could seems be compiled against one.). I think it will force PSAnalyzer to change its archetecture.

[lzybkr]

@bergmeister - I wouldn't move all files - e.g. compiler.cs isn't useful outside of the engine.

[rjmholt]

If we went down the path of putting the parser into its own project, I think it would be important to work out what the conceptual cohesion of that project would be.

I imagine that project's conceptual purpose would be to ingest PowerShell script input and output a PowerShell AST, which is then input to the engine. But that's just my quick imagining. @lzybkr, @BrucePay and @daxian-dbw might have a better conception of what belongs in that project.

[iSazonov]

I see DSC in the Parser and it creates Runspace and PowerShell objects. This makes it impossible to decouple the parser in a separate package.
Is DSC needed for PSSA and PSES?

[bergmeister]

@iSazonov PSSA has DSC specific rules and using the -SaveDSCDependency switch, it downloads required modules from the PSGallery by parsing occurrences of Import-DSCResource so that it can later load them into a runspace in order to be able to parse a DSC script, therefore I _think_ that PSSA does not need public Parser APIs for DSC but it probably needs the parser to 'work' when parsing a DSC resource.
https://github.com/PowerShell/PSScriptAnalyzer/blob/b1a81874fbe61ebaf1a0b9631f337b4f06296625/Engine/Generic/ModuleDependencyHandler.cs#L424

[iSazonov]

it can later load them into a runspace in order to be able to parse a DSC script

@bergmeister Who creates the runspace? If PSSA we could pass it as argument to the Parser.

[bergmeister]

@iSazonov PSSA creates the runspace here. What happens is that if calling Parser.ParseFile fails here, then it extracts the missing module names based on parsing extents of Import-DSCResource or the error message and then downloads them to a local temp folder here, which it temporarily adds to the PSModulePath environment variable here so that the Parser can pick up the downloaded dependencies

[iSazonov]

Looking the PSSA code I see that it utilizes _full PowerShell Engine_ and I wonder that you request standalone nupkg for Parser.

[bergmeister]

@iSazonov At the time being the request was for a company internal software that only needed the Parser for static script parsing, which was not as involved as PSSA. I'm not working for that company any more but the request for having a nupkg for the Parser in general is still valid I think and has multiple use cases.

[iSazonov]

@rjmholt Could you please share how PSES uses the Parser? Can it really benefit from the standalone Parser or we have the same as for PSSA?

[rjmholt]

PSES uses the parser to perform script file analysis, like reference discovery, but it also runs PowerShell. PSES integrates with the installed PowerShell on a system and won't run without it, so there's no strong need for a standalone parser, but it would add universality perhaps. I think PSSA has a much stronger need, since it's used to analyse scripts going to other PowerShell implementations.