Powershell: Decouple the bundled modules to seperate Repo's

[kilasuit]

Now that we have PowerShell Open Sourced we should really move to decouple all of the modules that are shipped "inbox" into their own repos so that updates to these can be make separately to the core engine and can also be pulled from the PowerShell Gallery

However this brings a number of additional challenges that warrants a detailed discussion and a plan of action as not to break the current build process.

Modules that should be decoupled and pulled in as submodules

• [x] - PowerShellGet Priority
• [x] - PackageManagement (this may already be out there in the OneGet Repo)
• [x] - Microsoft.PowerShell.ODataUtils
• [x] - PSReadline (already decoupled just needs to become a submodule)
• [x] - PowerShell.Archive (as above)
• [x] - Pester (as above but submodule to point to Pester not current submodule)
• [ ] - CimCmdlets
• [ ] - Microsoft.PowerShell.Diagnostics
• [ ] - Microsoft.PowerShell.LocalAccounts
• [ ] - Microsoft.PowerShell.Management
• [ ] - Microsoft.PowerShell.Security
• [ ] - Microsoft.WSMan.Management
• [ ] - PowerShell.Utility
• [ ] - PSDiagnostics
• [ ] - PSScheduledJob
• [ ] - PSWorkflow
• [ ] - PSWorkflowUtility

This is especially more important as in the differing subfolders in https://github.com/PowerShell/PowerShell/tree/master/src/Modules there are a number of code duplications due to the modules residing in multiple folders which is rather unnecessary and adds unneeded complexity for management of this repository going forward.

[lzybkr]

We have definite plans to do this for some of the modules like PackageManagement, PowerShellGet, Pester, and PSReadline.

Utility and Management might not be worth the hassle because of implicit or explicit circular dependencies.

[kilasuit]

I thought it may have been "on the plans" but couldn't see an issue for it so felt it best to raise one for visibility

[Carringguns]

Thanks im not free and clear yet. Little cleanup still. But want go to the source and offer you the data. and its running state im keeping in a loop.

[Carringguns]

Its design to eexplore when mission complete. Love this one. Better than most ive seen

[vors]

• [ ] PSReadLine @andschwa
• [ ] Archive @vors
• [ ] Pester @vors #1371

The current proposition is the following:
Make sure that upstream version is suitable as a drop-in replacement.
If not, massage it to the point, when it is.

Replace code by a git submodule.

cc @andschwa @lzybkr
cc @dlwyatt (for Pester)

[andschwa]

Oh yeah. I can help save the history for PSReadLine (and fix incoming changes on the upstream repo).

[vors]

@andschwa sweet, then I will assume you will take care about PSReadLine

[andschwa]

Okay, sounds good.

[SteveL-MSFT]

ODataUtils should be split out as well

[dragonwolf83]

It is probably a bit late to change it, but Is there a better way to organize all the repos under https://github.com/PowerShell to handle splitting the code base more?

It makes perfect since to break out the modules, but it is a bit messy right now with all of the individual DSC repos + Documentation + RFC. There are 6 pages of repos already. I can imagine a dozen more springing up to clean up the code base and dozens more for new things.

Something to keep in mind for those of us watching the Repos! A landing page dedicated to source would probably be enough of it is self-maintaining (or just maintained).

[RamblingCookieMonster]

@dragonwolf83 - Agreed, probably worth opening an issue for this, although not sure where you would do that, given that it's an org-level issue : )

Could go as simple as a readme that points to the various repos, with a little organization... or even something like GitHub Pages. Ideally, the community could submit PRs and issues, so IMHO a Microsoft hosted site (unless fed by CI/CD from GitHub) would be less appropriate.

Cheers!

[vors]

A landing page dedicated to source would probably be enough of it is self-maintaining (or just maintained).

I don't think it's worth the effort to be honest.

This is a generically applicable statement for any big github organization.
Take https://github.com/Microsoft or https://github.com/google as examples.

How people are finding repos in these cases?
Trying to list all the repos is not practical. There are projects that only a handful of people are interested in. With modern search engines (and also GitHub search) it's easy to find a particular project or explore interesting projects.

Also https://github.com/dotnet has a github-pages hosted site for the org, but there is no list of all projects there.

[vors]

Talking about lists, there are a lot of lists under "awesome" umbrella.

https://awesome.re
https://github.com/alebcay/awesome-shell

Maybe awesome-powershell would be something that community is interested in?

[kilasuit]

I'll update the Issue to include a list of modules that I think should be decoupled where reasonably possible and then if we raise a new issue for each module then we can track it properly as well with tasklists to see the progress

[kilasuit]

@vors @lzybkr @andschwa @SteveL-MSFT - Please see the list of modules I think that need to be decoupled from this repository is now updated at the top of this issue in my initial comment

[kilasuit]

@vors @lzybkr @andschwa @SteveL-MSFT - any further thoughts/comments to the suggested list at all?

[SteveL-MSFT]

@kilasuit thanks for taking the lead on this. the list looks good

[vors]

I think we should be clear about the trade-offs and goals here.
Things like Pester must be separate because it's a separate project, it just happened to be very central for out testing and we are using our fork temporarely.

So essentially for anything that is makes sense to be shipped on the gallery and updated from the gallery, we can do the decoupling. I.e. PSReadLine.

Decoupling _can reduce complexity, but it can also create it_.
I don't see value in separating core modules like Microsoft.PowerShell.Management: would it be intended to update from the gallery? If I need to fix a bug that spans across SMA and Microsoft.PowerShell.Management in this decoupling, it would be way harder.

I think these module should stay in the repo:

• PowerShell.Utility
• Microsoft.PowerShell.Diagnostics
• Microsoft.PowerShell.Management
• Microsoft.PowerShell.Security
• Microsoft.WSMan.Management
• PSDiagnostics
• CimCmdlets

These modules are for the Full CLR builds and hence again should stay in the repo

• PSScheduledJob
• PSWorkflow
• PSWorkflowUtility

[SteveL-MSFT]

Microsoft.WSMan.Management and CimCmdlets should be separate since those two are related and likely used together should probably be in same repo.

[andschwa]

@vors you mention:

Replace code by a git submodule.

But we're currently under the impression of bringing most of these pieces in via packages (PSReadLine, PowerShellGet, and Pester at least). We want to reduce submodules. Is that okay with you?

[kilasuit]

@andschwa as @lzybkr mentioned in #2405 pulling them in via packages is the right way long term as this will allow for a much better side by side story & could also make way for a better downlevel story too

[KirkMunro]

I generally agree with @vors about which modules should stay in the PowerShell/PowerShell repo.

My line in the sand to decide what stays vs what is submoduled would be whether or not module updates are already or may become later available to downlevel versions of PowerShell. If module updates do apply to downlevel versions of PowerShell, then use a submodule so that the module can be maintained independently as one that works for multiple versions of PowerShell, allowing for updates to be leveraged by all supported versions. If not, then keep it in the PowerShell/PowerShell repo because it is bound to that project.

PowerShellGet already has downlevel version support (although I don't think the way downlevel support is done in that module right now is the way to go, so there's some work to do there). PSScriptAnalyzer has downlevel version support. Microsoft.PowerShell.Archive is outside of PowerShell/PowerShell already, and currently requires version 5+, but it will have downlevel version support shortly once I submit my pull request. Other core modules listed above in @vors post should stay in the main project IMHO.

[lzybkr]

I see some value in breaking out all modules - e.g. the recent improvement to Join-Path in Microsoft.PowerShell.Management and Get-Credential in Microsoft.PowerShell.Security are both useful and shouldn't require updating to a newer version of PowerShell.

That said, decoupling a couple of the core modules has some risk, so I'd consider that lower priority.

[vors]

PSGet and OneGet are separated in #2711

[kilasuit]

@joeyaiello - please have a look over this issue from 2016.

Is there any chance could we get some update/traction on this to make things easier/better going forward PowerShell 6.1 onwards

[iSazonov]

The only thing left to complete is PSReadline.
I agree with @vors and don't see any benefit in moving out the rest of the modules. No one is actively developing these modules now. What prevents? Does someone have a big additions to (ex.) Odatautil in their repositories? Why can not commit here?

I would have seen more benefits in moving very popular cmdlets to this repository after they became stable.

[kilasuit]

Bumping this for a review as PSReadline was recently decoupled and there is still a checklist (checked off PSReadline just now) that needs to be confirmed/denied if/when there are plans to do so

[iSazonov]

ODataUtils and Pester is in separate repos.

[kilasuit]

updated the checkboxes - thanks @iSazonov

[iSazonov]

I think we can close the issue. All works is done. The remaining modules will still be in this repository as discussed.
/cc @SteveL-MSFT