Powershell: Make using NuGet packages installed with Install-Package easier to use - make Add-Type support NuGet packages

[mklement0]

https://github.com/PowerShell/PowerShell/issues/6679#issuecomment-384096971 shows how hard it is to _use_ a NuGet package installed via Install-Package in Powershell, for two reasons:

• You must manually determine the platform-appropriate *.dll file in the package's folder subtree and pass its full path to Add-Type -Path (e.g., runtimes/linux-x64/... on a 64-bit Intel Linux system)

• Even that by itself is not enough, if the package contains native helper libraries, because Add-Type -Path doesn't find them - see link above (whereas creating a C#-based .NET Core console application based on the same package installed with dotnet add package _does_).

Therefore, it would be great if Add-Type had built-in support for resolving both issues above, say with the following syntax:

## WISHFUL THINKING

# Install a NuGet package from the NuGet Gallery and add its types to the session.
Install-Package -Provider nuget Mono.Posix.NETStandard  | Add-Type

# Add the types of an already installed NuGet package to the session.
# Perhaps even download and install the package on demand.
Add-Type -Package Mono.Posix.NETStandard 

# Equivalent, via Get-Package.
Get-Package Mono.Posix.NETStandard | Add-Type

That is, when given a NuGet package [name], Add-Type would load the platform-appropriate assemblies and all their dependencies to make their types available to the current session.

Environment data

Written as of:

PowerShell Core v6.0.2

[thnk2wn]

Would love this. Coming from Cake Build this was so easy with https://cakebuild.net/docs/fundamentals/preprocessor-directives. Given the widespread usage of nuget and likely need to leverage that from PS Core scripts it would seem this would be supported as a first class citizen.

[iSazonov]

How it correlates with Install/Uninstall-Module?

[mklement0]

@iSazonov:

Install-Module is used for _PowerShell modules_ from the PowerShell Gallery, whereas NuGet packages are _.NET packages_ from the NuGet Gallery.

While PSGallery modules can be installed with Install-Package too, because the *-Package cmdlets are an _abstraction_ over all galleries / package sources, the inverse is not true: NuGet packages cannot be installed with Install-Module.

Is that what you were asking?

[iSazonov]

Thanks!
Seems this can resolve #7259 too:

Install-Package "SomePackage"  | Add-Type -PassThru | Import-Module

[mklement0]

@iSazonov:

While #7259 has the same goal in the abstract - making use of NuGet package easier - it is a _competing_ proposal in terms of implementation [_update_: it isn't; said issue is about PowerShell modules only - see here].

• 7259 suggests the use of Import-Module to load NuGet packages [containing _PowerShell modules_].

• This issue suggest use of Add-Type _only_ [for _non-PS-module_ NuGet packages].

I don't think it's appropriate to integrate the functionality into Import-Module, because NuGet packages are _.NET assemblies_, not _PowerShell modules_; hence the suggestion to use Add-Type, which is the cmdlet already used for loading assemblies.

In short: My vote is to implement this issue only, _instead_ of #7259.

[iSazonov]

.NET assemblies can contain cmdlets. We use this in our tests.

[mklement0]

@iSazonov:

Yes, all compiled cmdlets come as part of assemblies, but they're typically packaged as part of a PS module.
However, good point about the need to use Import-Module on an assembly containing a cmdlet that isn't packaged as part of a module in order to import such cmdlets; I assume you meant:

(Install-Package "SomePackage" | Add-Type -PassThru).Assembly | Import-Module

But it sounds like we're in agreement then, correct?:

No changes to Import-Module are required; it is sufficient to make Add-Type understand NuGet packages.

[mklement0]

I've started a conversation about which proposal to consider (or perhaps find a synthesis) at https://github.com/PowerShell/PowerShell/issues/7259#issuecomment-409319483

[jptillman]

Since this has been sitting on the back-burner for 7 months now with no comments or assignment, I'll ask the obvious: has anyone found a workaround solution that doesn't require waiting for this feature to be implemented?

[KallDrexx]

I would like to know this too. We have a bunch of local development functionality that would be good for us to implement, such as seeding our local Azure Storage Emulator with local development data and a bunch of other local-only functionality that would make a lot more sense as powershell scripts than a catalogue of random tools in our C# projects. Unfortunately Powershell's limitations with importing nuget packages into the namespaces makes this difficult and we just opted for spamming our solution with misc projects :(.

[aolszowka]

I just wanted to chime in and say we're run into this as well.

Our use case is that we use several binaries from SQL Management Objects (SMO) which as of SQL 2017 are ONLY shipped as a NuGet Package in our various scripts for maintaining SQL Databases. It'd be really convenient to have Add-Type extended to support that scenario.

Anyone got any good ideas?

[iSazonov]

@aolszowka We could use assembly context. Now _binary module_ developers can already use it.

[aolszowka]

@iSazonov Are you asking which Assemblies we use from the SMO package or is there something else I am missing?

• Microsoft.SqlServer.SMO
• Microsoft.SqlServer.SmoExtended
• Microsoft.SqlServer.ConnectionInfo
• Microsoft.SqlServer.SmoEnum

We primary use it to expose Microsoft.SqlServer.Management.Smo.BackupDeviceItem and its kin to our scripting environment.

I suspect that the problem will become worse as more and more of .NET Core is leveraged for example in other places in this script we use System.Data.SqlClient right now the older version that ships with the .NET Framework works just fine for us; but eventually I am sure we will need to ship the version that is posted on NuGet https://www.nuget.org/packages/System.Data.SqlClient/

[iSazonov]

@aolszowka My comment was about version conflits and how we could resolve this. PowerShell Core uses default .Net Core assembly context. All assemblies is loaded in the context including assemblies created by Add-Type. As workaround you could create custom context and load needed assemblies to it. Then you'll have to implement cmdlets which refer to the custom context.

[jzabroski]

@iSazonov I think this is an important feature, and you bring up a good point about assembly context. With the new .NET SDK project system, PowerShell has been completely left behind in terms of ease-of-use. @SteveL-MSFT recently asked me how I wanted New-ModuleManifest to work, and I rebutted that I don't just want functionality to change, I want entire _features_ to exist that make development easier.

For others (@mklement0 and @aolszowka ): To understand what @iSazonov is saying about "assembly loading context", I suggest reading @natemcmaster 's blog post: https://natemcmaster.com/blog/2018/07/25/netcore-plugins/ - so, when you're calling Add-Type, it loads it into the default context and there is no assembly isolation, so there is no way to load two versions of Newtonsoft.Json. Add-Type would need a -AssemblyLoadContext parameter to be able to say how to load the types from the dll.

[jzabroski]

@iSazonov The other problem I see is that psd1 file format does not specify an AssemblyLoadContext, either.

[iSazonov]

@jzabroski It could be in PowerShell engine like Import-Module -IsolatedContext

[jzabroski]

@iSazonov Yes, it could be in PowerShell engine via something like -IsolatedContext. Here are a list of the touch points I can think of throughout the PowerShell universe that should be considered:

1. psd1 PowerShell Module Manifest file "knobs"
   
   
   
   1. RootModule key can be a dll (although this fact is not documented _anywhere_, my testing discovered it), that dll in turn can load other dll(s) using the default assembly loading context. While it's possible for the author of the RootModule dll to use something like DotNetCorePlugins and achieve an isolated context for its dependencies,
   
   
   2. (Curiously) If RootModule is unspecified and NestedModules contains a single dll, that dll will become the entry point for the PowerShell module. (This was tested using RedGate.SqlClone.PowerShell module). This is my _observed behavior_ and may not be the fundamental behavior.
   
   
   3. If RequiredAssemblies key is specified, my _observed behavior_ (because PowerShell documentation is missing) is that Import-Module for the module forces the assemblies to load before Import-Module completes.* (* Asterisk denotes "before", because, as you may know, the module cache is asynchronous, so the behavior is actually undefined as to when it really gets loaded.)
      
      
      
      
      
      • In PowerShell 5, if you try to load two different dll versions of the same assembly via RequiredAssemblies, the second assembly silently fails. In PowerShell Core, it produces a _warning_, which is better but still not great.
      
      
      
   
   
2. Add-Type knob
   
   
   
   1. Like the psd1 PowerShell Module Manifest, it contains a -ReferencedAssemblies parameter, which I am guessing is meant to be isomorphic to the Module Manifest's RequiredAssemblies key. This "law of nature" is also not documented anywhere, but for the sake of this discussion and drawing a circle around "assembly loading context" design space, it is purposeful to highlight.
   
   
   2. Get-Help Add-Type -Examples returns nothing. Before we go adding new parameters to this cmdlet, I suggest we fix the documentation. What's the point in having a technical discussion if there is no specification and examples of the specification in action?
   
   
3. CLR System.Reflection knobs
   
   
   
   1. [System.Reflection.Assembly]::LoadFrom , the behavior for which is defined in Deploying .NET Framework Applications > How the Runtime Locates Assemblies > Initiating the Bind
   
   
   2. [System.Reflection.Assembly]::LoadWithPartialName deprecated
   
   
4. Please add additional knobs I am unaware of in your replies to this thread.

In this vein, I have found StackOverflow user _Bacon Bits_ answer to "How to load assemblies in PowerShell?" quite helpful, in that he delves into "How do I verify what I loaded?"

Further in this vein, I think it would be great to name named contexts. This could be essentially the same as "var jq = $.noConflict()" JavaScript trick the jQuery library uses: it releases the existing named context and replaces it with a new variable (ideally a locally scoped variable!). This is also not much different than how, in C#, you can specify a namespace alias, including the global namespace alias.

[jzabroski]

Note: I ran into an issue on PowerShell 5 where an assembling Binding Redirect was required, but since PowerShell doesn't have the concept of binding redirects, I can't just use Add-Type or -RequiredAssemblies, as explained in StackOverflow answer to Resolving assembly dependency references with powershell

I realize this GitHub project is devoted to PowerShell 6, but until PowerShell 6 gets more features, for the same of completeness, I'm going to bring this up as it's "the only way to get things done in PowerShell 5".

[iSazonov]

since PowerShell doesn't have the concept of binding redirects

It is not PowerShell feature. It is .Net Core feature and we get this back in .Net Core 3.0 as new API.

[jzabroski]

Please provide a link to explain your comments.

On Sat, May 25, 2019, 3:42 AM Ilya [e-mail] wrote:

since PowerShell doesn't have the concept of binding redirects

It is not PowerShell feature. It is .Net Core feature and we get this back
in .Net Core 3.0 as new API.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/6724?email_source=notifications&email_token=AADNH7JWJX6SN5EW4LSMANDPXDUXFA5CNFSM4E4KKOOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWHIFRQ#issuecomment-495878854,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADNH7IZLP4ZLCWSJPEW2UTPXDUXFANCNFSM4E4KKOOA
.

[iSazonov]

https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/assemblyloadcontext.md

Links from related discussion https://github.com/PowerShell/PowerShell/issues/9488#issuecomment-492650809
https://github.com/PowerShell/PowerShell/issues/9488#issuecomment-492726603

Update: https://github.com/dotnet/designs/blob/master/accepted/runtime-binding.md#rollforward

[MartinHBA]

hello guys, I am MS-SQL DBA and I use PowerShell on Windows and I used it also with System.Data.SqlClient namespace, however this won't get new features from now on and only Nuget Package Microsoft.Data.SqlClient will be shipped will all new features (decision by .NET team). This Nuget Package comes with 4 dependencies. Can someone please give me simple step by step guide how can use objects defined in Microsoft.Data.SqlClient in my PowerShell script? I've tried Add-Type but that doesn't work really.
edit: I finally get it done... only PS Core works

[iSazonov]

edit: I finally get it done... only PS Core works

@MartinHBA Since you mentioned this, please add a link to an example/gist.

[MartinHBA]

hi @iSazonov , you can find it here
edit: fixed link, thx

[mklement0]

@MartinHBA, there's an extra char. at the end of your link's URL - the correct link is:
https://gist.github.com/MartinHBA/86c6014175758a07b09fa7bb76ba8e27

[jzabroski]

The correct link is
https://gist.github.com/MartinHBA/86c6014175758a07b09fa7bb76ba8e27

On Sat, Nov 9, 2019, 5:08 PM Michael Klement [e-mail]
wrote:

@MartinHBA https://github.com/MartinHBA, did you accidentally create a
private Gist? The link yields a 404.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/6724?email_source=notifications&email_token=AADNH7JT2ZDCB2KSSAXNDLLQS4YGPA5CNFSM4E4KKOOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDUQJ2I#issuecomment-552142057,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADNH7NXW5AOK3EDGLSCIPLQS4YGPANCNFSM4E4KKOOA
.

[aolszowka]

Is this on the roadmap for the PowerShell team at anytime?

The need to load in NuGet Packages (in this case the one that seems to keep coming up is the SMO Tooling, in addition to this thread here is another one https://github.com/dotnet/SqlClient/issues/623) seems to be a pretty common scenario for a lot of developers.

The current work arounds (provided by @MartinHBA and @mklement0 in #11860) aren't super great. Massive kudos to you two though because you show up in many of these threads! Here's another public blog post that tries to work around the issue as well http://www.maxtblog.com/2020/03/streamlining-sql-server-management-objects-smo-in-powershell-7-revised/. While we appreciate that it is a way to move forward they all basically boil down to:

1. Copying the Appropriate DLL's out of the various referenced NuGet Packages (or creatively referencing them from their NuGet locations)
2. Performing an Add-Type for each of the dependency assemblies

This is a bit of a pain, especially because the Microsoft.Data.SqlClient.SNI.runtime package ships a native binary that will require you to determine if you're going to run x64 vs x86 and then have the native assembly sitting next to the managed assembly. This makes automating this in any type of pipeline wherein you do not have full control over the machine difficult.

The path of least resistance in these cases has been to build a "dummy" binary that pulls in all of the NuGet Packages into a single output and then commit this into version control. This completely subverts the powerful tooling that NuGet was supposed to provide to us to keep our repositories free from binaries.

Streamlining SQL Server Management Objects (SMO) In PowerShell 7 (Revised) – Max Trinidad – The PowerShell Front