Powershell: Question on common dependency binary cmdlet module

[arunswarnam]

We are working on developing few binary Cmdlet modules targeting PWSH Version 6 and above. we are trying to answer some questions (related to a common dependent module) that possible end users of our modules could face when they try out the Cmdlet modules we will be publishing to PSGallery. It would be great if you could advise us. For this example let us assume there are two modules Module A, Module B that provides different functionality but both Module A and Module B depends on some boiler plate code added in Module C (Common module). We can also assume that all Modules (Module A and Modules B) with version 0.1.0 will depend on Module C (Common Module) with the same version (0.1.0). We also intend to do periodic releases of all the modules. Now if an end user wants to do the following:
1) Use Module A - 0.1.0 (depends on Module C - 0.1.0) and Module B 0.2.0 (depends on Module C 0.2.0) in the same Powershell session. If the user imports Module A in the session and then tries to import Module B. Pwsh complains "assembly with the same name is already loaded". What can we possibly do let the user use different versions of Module A and module B in the same session?

1. This question is not a real problem (as we could open a new PWSH session) but still we are trying to understand why it is happening. We also notice that if we do Import-Module ModuleA -RequiredVersion 0.1.0 and Remove-Module ModuleA and later do Import-Module ModuleA -RequiredVersion 0.2.0. Pwsh complains the same error telling assemblies with the same name is already loaded. Is this because Remove-Module does not remove the loaded assemblies? Is there a work around to load the latest version of the module in the same PWSH session after doing a Remove-Module?

ModuleA.psd1 contents for version 0.X.0:

RootModule = 'assemblies/ModuleA.dll'

# Version number of this module.
ModuleVersion = '0.X.0'

RequiredModules = @(@{ModuleName = ‘ModuleC’; GUID = ‘xyz’; RequiredVersion = '0.X.0'; })

# Assemblies that must be loaded prior to importing this module
RequiredAssemblies = 'assemblies/AnotherDependencyForModuleA.dll'

```
ModuleC.psd1 contents for version 0.X.0:

Script module or binary module file associated with this manifest.

RootModule = 'assemblies/ModuleC.dll'

Version number of this module.

ModuleVersion = ‘0.X.0’

Modules that must be imported into the global environment prior to importing this module

RequiredModules = @()

RequiredAssemblies = 'assemblies/NLog.dll',
'assemblies/retrier.dll'

## Steps to reproduce

```powershell

Expected behavior

Actual behavior

Environment data

[vexx32]

It happens because the two modules of a different version try to load the same dependency with a different version (and public token). .NET doesn't really allow this without a good amount of finagling.

/cc @daxian-dbw @rjmholt this is one scenario that would be made possible if PowerShell had some support for separating AssemblyLoadContexts between modules.

[SeeminglyScience]

Related issue that @vexx32 is referring to: #12920

[viralmodi]

@vexx32 How about the scenario 2 above? Why is that not possible? Why do we force users to open a new powershell session when all they intend to do is upgrade to a latest version of a loaded assembly?

[vexx32]

@viralmodi I'm not clear on the precise specifics, but in general terms .NET isn't great at removing binary modules from memory. I know there were _some_ improvements made to that end in .NET Core 3, but to my knowledge it's not something PowerShell has been coded to work with yet. I'm sure there are limitations even so, but it's a bit beyond my ken.

[SeeminglyScience]

I know there were _some_ improvements made to that end in .NET Core 3

afaik that only applies to assembly load contexts, you still can't unload an assembly loaded into the default ALC (except for dynamic assemblies marked as collectible). And even then it's mostly impossible in PowerShell due to caching (can't unload an assembly if it's being referenced).

[viralmodi]

@vexx32 @SeeminglyScience thanks for your responses. Our team is a relatively new to PowerShell. Perhaps, it would be great help if you can point to some documentation which addresses typical usage of PowerShell for the use cases mentioned above? i.e. as a PowerShell user, if I have an installed and imported module in a PowerShell session and I need to upgrade its version, is the only option to close this shell window and open a new one? How do veteran PowerShell users deal with use cases 1 and 2 mentioned above? (Would they already know to close the PowerShell session and open new PowerShell to deal with these use cases because this is considered "normal behavior" in PowerShell world?) I, specifically, want to ensure that we (my team) understand/s this part better to be able to explain our Product's Module users in future, if and when, they report issues related to this behavior.
Again, thanks for your quick responses and help.

[iSazonov]

@viralmodi There is no solution for the cases mentioned above. #12920 is for discussion this.

Your team could create a temporary solution for your modules using ALCs as @rjmholt described in blog post (see the reference in #12920).

But it is much better to contribute in PowerShell if your team have resources. Resolving dependency conflicts, module isolation and module unloading is a related and complex problems and it's worth it to resolve them.

[SeeminglyScience]

i.e. as a PowerShell user, if I have an installed and imported module in a PowerShell session and I need to upgrade its version, is the only option to close this shell window and open a new one?

Yeah more or less. You can also install side by side, but you'll still need to open a new session before you can actually import the new version.

How do veteran PowerShell users deal with use cases #1

For dependencies, you have these options for the most part:

1. Target and bundle the lowest version you can for each module. When one module needs to target a later version, target it in all modules. This only really works if you control the dependency and every dependent module
2. Don't bundle the dependency with each module, put the dependency in it's own module, and require PowerShell to load the installed module instead of .NET. Not a lot of folks do this one because PowerShellGet doesn't handle it well currently
3. Put each module in it's own ALC, then it can load whatever it wants. Only choose this one if you fully understand how ALC's work. They make it incredibly challenging for external binary modules to interact with your module

Honestly most of us just don't take/make dependencies.

and #2 mentioned above? (Would they already know to close the PowerShell session and open new PowerShell to deal with these use cases because this is considered "normal behavior" in PowerShell world?)

Yeah it's pretty commonly hit limitation. Most veteran users will already have ran into it. Those that don't, just let them know they need to update prior to importing.

[arunswarnam]

It happens because the two modules of a different version try to load the same dependency with a different version (and public token). .NET doesn't really allow this without a good amount of finagling.

/cc @daxian-dbw @rjmholt this is one scenario that would be made possible if PowerShell had some support for separating AssemblyLoadContexts between modules.
I understood that it is because of a dependent assembly but what I am not sure of is this https://devblogs.microsoft.com/powershell/resolving-powershell-module-assembly-dependency-conflicts/#comment-398

PowerShell

Resolving PowerShell Module Assembly Dependency Conflicts | PowerShell

When writing a PowerShell module, especially a binary module (i.e. one written in a language like C# and loaded into PowerShell as an assembly/DLL), it’s natural to take dependencies on other packages or libraries to provide functionality. Taking dependencies on other libraries is usually desirable for code reuse.

Why PWSH allows older version of the same module to be loaded into the same ALC (PWSH default ALC) even though it has a newer version of the same module already loaded but not vice-versa?

[SeeminglyScience]

Why PWSH allows older version of the same module to be loaded into the same ALC (PWSH default ALC) even though it has a newer version of the same module already loaded but not vice-versa?

It doesn't, but if you reference version 0.1.0 of an assembly and you have 0.2.0 loaded, assembly resolution sees that and says "probably works". Most assemblies are backwards compatible, the problem is when you have already loaded 0.1.0 but specifically reference 0.2.0, because then there is probably an API that you need that isn't there.

Note that none of this really has much to do with PowerShell specifically, these are .NET rules. You're just less likely to run into the same problems outside of PowerShell because most C# projects get their own process. As a module in PowerShell, all of these different projects share the same process/appdomain/ALC.

[arunswarnam]

Why PWSH allows older version of the same module to be loaded into the same ALC (PWSH default ALC) even though it has a newer version of the same module already loaded but not vice-versa?

It doesn't, but if you reference version 0.1.0 of an assembly and you have 0.2.0 loaded, assembly resolution sees that and says "probably works". Most assemblies are backwards compatible, the problem is when you have already loaded 0.1.0 but specifically reference 0.2.0, because then there is probably an API that you need that isn't there.

Note that none of this really has much to do with PowerShell specifically, these are .NET rules. You're just less likely to run into the same problems outside of PowerShell because most C# projects get their own process. As a module in PowerShell, all of these different projects share the same process/appdomain/ALC.

Oh! So PWSH doesn't actually load the older version of the assembly and just relies on the backward compatibility of the assembly to work correctly?
Looking at all the answers loading dependent assemblies into an isolated ALC seems to be next step forward for our use case.

[SeeminglyScience]

Oh! So PWSH doesn't actually load the older version of the assembly and just relies on the backward compatibility of the assembly to work correctly?

Yeah, more specifically that's how .NET's assembly resolution works. PowerShell doesn't really have a hand in it afaik.

It's not perfect, it's possible for minor versions to have breaking changes like changing a method overload or something. In those cases it'll throw a JIT time exception which is very hard for the consumer to pin down. So if you own the dependency, try very hard to keep binary compatibility.

Looking at all the answers loading dependent assemblies into an isolated ALC seems to be next step forward for our use case.

Make sure to read through the linked issues and be sure that you understand the challenges it presents. Once you have it up and running make sure any type you return is either from the global ALC, or is not accepted by any other API. For example don't isolate newtsonsoft and then emit a JObject to the pipeline because it's type identity will not match the type identity that other assemblies expect.

Right now the only time I would personally recommend this approach is if your module is intended to be loaded into an environment where it's presence should be mostly invisible. For example, the module that the vscode-powershell extension loads must have as little impact on the process as possible or it's value diminishes (like if a script breaks purely because of the editor you're testing it in, that's problematic). That's significantly less true for most projects.

[msftbot[bot]]

This issue has been marked as answered and has not had any activity for 1 day. It has been closed for housekeeping purposes.