Powershell: Pure Pain: Preinstalled PS Modules - Please Rationalize Them with Package Management Installs

[DarwinJS]

Cross posting this so that you all can go vote on it here: https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/20398699-pure-pain-preinstalled-ps-modules-please-ration

I am trying to update Pester on Windows 2016 to 4.x because the preshipped Pester 3.4.0 emits a note about the depreciation of the -quiet switch when I use the quiet switch. This get's scooped into stdout when executing via AWS SSM remote commands. Version 3.x complains it does not understand "-Show None".

I want to update to 4.x - in which I ran into all the problems documented here: https://github.com/OneGet/oneget/issues/215 (created by @jaykul)

But ended up with "Install-module pester -force -SkipPublisherCheck".

So far so bad - but when I go to remove the 3.4.0 version (with admin rights) from: 'c:\program files\windowspowershell\modules\pester\3.4.0' I get errors because only TrustedInstaller has permissions.

If this scenario wasn't horrible enough, putting preinstalled modules under Windows Resource Protection really throws it over the top.

Could someone please make sure that preinstalled modules can be replaced cleanly? Or don't preinstall them?

[daxian-dbw]

Can you please also open an issue about this on https://github.com/PowerShell/PowerShellGet?

[iSazonov]

It seems we could make preinstalled modules optional.

[RichardSiddaway]

I would be better if they were installed as if they'd come from the gallery. That leaves them updateable and removable if required. I've been through the pain of upgrading Pester and its not an experience I'd wich on anyone

[iSazonov]

TrustedInstaller is fundamental Windows protection - we shouldn't ignore this. Right fix is in PowerShellGet to support this.

[SteveL-MSFT]

Perhaps what we need is to not distribute the modules, but use the suggestion api to suggest installing from the gallery on first use? For any inbox modules, they will always be owned by TrustedInstaller since it's copied during OS install.

[iSazonov]

Thinking about corporate environment I would prefer to have frequently used modules in distributive. And have possibility to customize distributive to install/update additional modules from local folder/repo.

[DarwinJS]

@daxian-dbw - From the responses I got from uservoice it seems like that might be the right place since being part of the base Windows build is what puts preshipped modules in the odd state of not being able to remove them. Thanks.

[DarwinJS]

I also put some possible implementation suggestions on the uservoice article - such as having a scheduled 'run on first boot' powershell script that installs them so that are there on new OS loads - but not technically installed during OS build.

[DarwinJS]

My last comment about the uservoice "run on first boot' is exactly along the same lines as @RichardSiddaway comment. My intent was to essentially find a way to actually install it from the gallery in every usage scenario.

The downside being that in many circumstances that first boot won't have internet access - hence the additional uservoice suggestion to have the modules stashed locally and installed on first boot from a local location.

[SteveL-MSFT]

We've already removed Pester as part of PSCore6 package, but we've been discussing internally about removing Pester 3.4 from Windows PowerShell. We originally made the decision to ship Pester inbox in Windows to help build a community around a common test framework for PowerShell as well as encourage more testing in general. I believe we've succeeded in that effort so there's no longer a requirement to ship Pester inbox and it makes sense for users to be accustomed to getting the latest Pester from PSGallery.

A less desired option is to update the inbox version of Pester to 4.x. This is problematic for two main reasons:

1. 4.x is not backwards compatible with 3.4 so it's a breaking change
2. There's a ton of process work to ship the new version in Windows

I'd like to get some community feedback on this. Keeping Pester 3.4 is not an option as it's only getting more and more out of date.

[markekraus]

@SteveL-MSFT I don't have a good solution to this, but one problem when pester stops shipping with Windows PowerShell will be its disappearance from CI's that keep their PS updated. I agree that keeping 3.4 is not an option. Adding 4.0 is a breaking change. Removing Pester is a breaking change.

My preference would be that it not ship with Windows PowerShell instead of shipping a Pester 4.0 I can't easily downgrade from. If a Pester 4.0 can be included that I can easily downgrade from, then I guess that is OK too.

[Jaykul]

If it's going to "TrustedInstaller" locked and catalog signed with a cert that the Pester team doesn't own, I'd _rather have it not be included_, just because of the frustration of updating it, and the certainty that it's getting older. I'm actually dealing with a situation where we're slightly locked into the old version -- partly because it's preinstalled, and taking a dependency on the new version in one of our modules seems like it would break installing that module.

However, I have to ask whether we're establishing a precedent. Will you apply the same logic to PSReadLine? Not having _that_ preinstalled would be more painful.

I wonder if there could be a meta-module that takes dependencies on those so that people could just Install-Module WindowsPowerShellCompatibilityPack and it would install those modules to preserve compatibility? And then 😉 I find myself wondering whether that module would install the _latest_ or the original, fully-compatible version 😕. But in either case, at least the modules would get installed, and they'd be upgradable with PowerShellGet

[essentialexch]

Get rid of it. Not one single admin-scripter knows what it is. Same for PSReadLine.

Who is the target of the default install? It darn well ought to be the admin-scripter. Not someone in DevOps. Not a PowerShell MVP (or equivalent).

[it-praktyk]

Pester shipped with PowerShell is "little" outdated and will be deprecated soon (?) - the pull request https://github.com/pester/Pester/pull/947 waits to be merged.

Please read: RFC - Pester v. 3.x - an maintenance plan.

CC: @nohwnd, @dlwyatt

[DarwinJS]

I agree with @Jaykul that if pre-shipping means the completed TrustedInstaller lock down it's a bit silly.

I don't agree with @swngdnz that admin-scripters are the primary target of shipped-in-box PowerShell goodies. I did only admin scripting for 15+ years and now I do 100% DevOps automation on Windows - pester is super useful for all scenarios.

[essentialexch]

OK, @DarwinJS, tell me how many times you used Pester as an admin scripter? If you say it's non-zero, I want specific details. Honestly, I won't believe you. It is NOT an admin scripter utility/technology/whatever.

[essentialexch]

By the way, @DarwinJS, I've been working with PowerShell since it was Monad for Exchange 2007 (in 2006). You cannot truthfully claim to be using PowerShell as an admin scripter for 15+ years. Again, your claim isn't believable.

[DarwinJS]

I said "I did only admin scripting for 15+ years and now I do 100% DevOps automation on Windows" - I didn't limit that to PowerShell :)

In regard to your other question I am responsible to build our Windows VM template for AWS. At various stages in the process I run Pester tests to look for common misconfigurations that we've experienced and to validate that hardening (such as SSL cipher disablement) is in place and a ton of other things. Before this was all done with monkey testing (Click-Next) and many times things were missed. Now when the automation runs I am 95% sure that the image creation automation was done correctly. The pester tests also work without modification with Server Core 2012 R2 and 2016 - where there is no GUI to click-next test on.

[essentialexch]

I wrote my first book on admin-scripting with VBScript with Exchange in 2003.

You @DarwinJS, still didn't answer the question: as an admin-scripter, how often did you use Pester? And how?

And if you are claiming to have used other admin-scripting technologies on Windows pre-PowerShell, how often did you use the WSD? And how?

[agarstang]

While I can think of several admin tasks where I've used or seen Pester used, I'd say remove it. Anyone who's going to use Pester knows what they want and knows where to find it in my opinion.

[MaximoTrinidad]

I think, removing Pester will be on the PScore side and not Windows PowerShell.
Right!

[markekraus]

@MaximoTrinidad from https://github.com/PowerShell/PowerShell/issues/4497#issuecomment-359861559

@SteveL-MSFT :

We've already removed Pester as part of PSCore6 package, but we've been discussing internally about removing Pester 3.4 from Windows PowerShell.

This is specifically about removing pester from Windows PowerShell. It has already been removed from PS Core.

[SteveL-MSFT]

@MaximoTrinidad Pester is already no longer shipped with PSCore. This is specifically about Windows PowerShell.

Regarding PSReadline, we made a decision early on for PSCore6 to use PSReadline as our default interactive readline experience so there are no plans to not have PSReadline by default (the basic readline support in pwsh is extremely basic). The upgrade inconvenience of PSReadline should be solved by PSGet team in my opinion.

[MaximoTrinidad]

Thanks @SteveL-MSFT !! I'm trying to keep up.
:)

[iSazonov]

If preinstalled modules is protected by TrustedInstaller then MSFT should distribute feature updates (Windows Update, WSUS). For Pester it can be optional update with Pester 4.
If in future PowerShell Core will replace Windows PowerShell then we get new policy automatically - we should change nothing in Windows PowerShell.

[SimonWahlin]

Pester is a great tool for any PowerSHell user and including it in Windows PowerShell was a great move at the time. Today we have a good and working module-management solution in PowerShellGallery and installing your version of choice is really simple.

That being said, I'd rather see Windows PowerShell shipping without pre-installed modules than going through the hassle of working around the trusted installer lock.

If modules are shipped within PowerShell they need to be kept up to date, which would include breaking changes as mentioned above and to be honest I think that would cause more harm than good.

[johlju]

I thought it was a good idea to distribute Pester with Windows. The adoption of Pester would improve since the framework was already present (and signed by Microsoft) and it was more "safe" to use without the need for a security review that might be needed by companies when downloading something from PowerShell Gallery. Removing it would be a major breaking change - adding the new one would be a breaking change in certain scenarios. If we could remove or add the new one with the next version (build) of Windows then that breaking change, either removing or changing version, would not have as much of impact - going to that build would be more of a breaking change than which Pester version is used. If we talking about installing an optional update for WMF 5.x and Pester would be removed, then that is up to each IT-department to evaluate the impact of installing an update. It we are talking of doing this in a non-optional update (like security update), then that's worse... we shouldn't do that - scripts could break.

[SteveL-MSFT]

Keeping Pester 3.4 in Windows is not an option as indicated by @it-praktyk as the Pester team wants people to move to 4.x and not continue to support 3.x and deprecate it.

[DarwinJS]

Can it be distributed as it - but let us delete it if we have upgraded it? I know how to hack at it to get it out from under TrustedInstaller protection - but maybe this could be built-in for PowerShell modules that pre-ship?

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this. The expectation is that Windows PowerShell (and associated modules shipped in Windows) are complete. These form a stable contract that people can depend upon for the foreseeable future. We don't want to introduce any changes that churn that contract. The current experience installing Pester 4.x on Windows PowerShell should be addressed by PowerShellGet.

[DarwinJS]

In my original post I wasn't necessarily interested in not shipping it - but that the technical aspects of upgrading don't create such an out-of-box rabbit hole experience.

I was also interested in possibly formulating an approach for any other pre-ships with any version of PowerShell - that don't unwittingly utilize the same rabbit hole.

[iSazonov]

@DarwinJS What is a status of the Issue? Can we close it? We removed all external modules from PowerShell Core distributive.

[stuartluscombe]

@iSazonov Removing the module from PSCore doesn't address the original issue though. Until a Windows update is put out that removes external modules I don't see how this issue can effectively be resolved.

@SteveL-MSFT In my mind I think removing the modules via an updates, which would then force admins to use PSGallery in order to re-add them will be less of a headache than having to force install modules such as Pester and have 2 different versions installed.

[KevinMarquette]

It may be best to move this conversation over to PowerShellGet. The issue with it being installed is that PowerShellGet makes updating it a lot of work and that does not need to be the case.

Unless I am missing something, there should not be any issues with leaving the old one in place. The newer version should get loaded if it is installed.

[DarwinJS]

@KevinMarquette - the initial issue does cite a similar issue in the oneget repo, however:

• I feel your statement "there should not be any issues with leaving the old one in place" should be backed by testing in all execution scenarios - under a service, as the system account, etc.
• this is a windows problem due to the level of lockdown of almost anything that is pre-shipped.
• I am building machine templates, and at least for that scenario I would like to have the one internally supported version be the only one available.
• I think any preinstalled modules should simply be easy to uninstall - just like the Window's feature on demand. When these features are on a system they have been qualified by MS and are locked in tight which is cool - but I can also remove them.

[mburszley]

@KevinMarquette I don't know about that. I just tested installing 4.4.2 (current, scope: allusers) and launching a new session. When I load up pester (ipmo pester), it's loading 3.4 instead of the latest available.

It wasn't until I installed it under the CurrentUser scope that it actually loads it first.

[SteveL-MSFT]

@TheIncorrigible1 the order of PSModulePath should be: user, all users, $PSHOME. So if you had 3.4 installed under your user folder, it will find that first unless you specify a specific version.

[mburszley]

@SteveL-MSFT on my win10 LTSB box, Pester 3.4 is installed under the AllUsers root and is taking priority over a later version of the module in the same root.

[SteveL-MSFT]

@TheIncorrigible1 if you're talking about Pester 3.4, you're talking about Windows PowerShell and not PowerShell Core, correct? We don't handle Windows PowerShell issues in this repo. However, on my Win10 box which comes with 3.4 installed for AllUsers and 4.3.1 installed for myself, it uses 4.3.1 just fine.

[mburszley]

We don't handle Windows PowerShell issues in this repo

@SteveL-MSFT Since that's the case, does this thread have a purpose anymore?

However, on my Win10 box which comes with 3.4 installed for AllUsers and 4.3.1 installed for myself, it uses 4.3.1 just fine.

My point here is that the prior version (3.4) is being selected over a later version of the same module if they're in the same scope (AllUsers).

[SteveL-MSFT]

@TheIncorrigible1 just tried with 4.4.2 and 3.4.0 installed for AllUsers on Windows PowerShell and 4.4.2 got loaded. If you can repro this with PSCore6, we can investigate

[msftbot[bot]]

This issue has been marked as answered and has not had any activity for 1 day. It has been closed for housekeeping purposes.