Powershell: Improving the discoverability of PowerShell variables

[iSazonov]

Continue #4216 to discuss improving the discoverability of PowerShell variables.

Problem: It is hard for users to discover PowerShell automatic (system) variables and get help (their description).

Idias:

1. Enhance cmdlets to provide better discoverability, tab-completion, etc.
   1.1. Get-Variable -AutomaticVariable/-PowerShell/-System/-Special / -PreferenceVariable
   1.2 Get-Help -AutomaticVariable/-SpecialVariable / -PreferenceVariable
2. Add new namespace $pspref or $psvar
3. Enhance the variable: provider with additional properties IsAutomatic and IsPreference.

[SteveL-MSFT]

Also enhance the variable: provider with additional property IsAutomatic

[mklement0]

@SteveL-MSFT:

Good idea; while we're at it: IsPreference would be useful too.

Programmatic discovery of preference variables has been suggested before, on uservoice.com.

[mklement0]

I still think the introduction of namespaces is worth considering, given that, with the current approach of sticking all automatic / preference variables in the same namespace as user-defined variables, any introduction of new automatic / preference variables comes with the risk of breaking existing scripts that happen to use variables of the same name.

Technically, a single namespace such as $ps: would suffice, containing the union of preference variables and automatic variables, including the proposed platform-abstracted "environment" variables (PS-internal variables that provide cross-platform information about the environment, such as the location of the directory for temporary files).

Given how namespace notation currently works, an underlying ps: drive would be required, though a conceivable departure could be to instead use Get-Variable with switches such as the new ones proposed in the initial post for filtered discovery of _all_ variables, including those in the $ps: namespace.

[iSazonov]

Internally we use term AutomaticVariables in different way then public.

Should we use SpecialVariables that is used internally in public too?

[iSazonov]

We have 7 well-known preference variable - I think there's no point in highlighting them.

[mklement0]

Re _automatic_ variables: While this discrepancy is unfortunate, _automatic variable_ is a well-established and documented user-facing term.

It sounds to me that the way to resolve this is to make a note in the source code or change the variable name(s), such as to AutomaticEngineVariable.

Re _preference_ variables:

Similarly, the about_preference_variables help topic lists many more preference variables than the 7 named as such in the source code:

> (Get-Help about_preference_variables) -split '\r?\n' | Select-String '^\s+(\$\w+)\s{2,}' | % { (-split $_)[0] }

$ConfirmPreference
$DebugPreference
$ErrorActionPreference
$ErrorView
$FormatEnumerationLimit
$InformationPreference
$LogCommandHealthEvent
$LogCommandLifecycleEvent
$LogEngineHealthEvent
$LogEngineLifecycleEvent
$LogProviderLifecycleEvent
$LogProviderHealthEvent
$MaximumAliasCount
$MaximumDriveCount
$MaximumErrorCount
$MaximumFunctionCount
$MaximumHistoryCount
$MaximumVariableCount
$OFS
$OutputEncoding
$ProgressPreference
$PSDefaultParameterValues
$PSEmailServer
$PSModuleAutoLoadingPreference
$PSSessionApplicationName
$PSSessionConfigurationName
$PSSessionOption
$VerbosePreference
$WarningPreference
$WhatIfPreference

On a side note: $OFS is currently documented as _both_ an automatic and a preference variable, whereas I think it should only be the latter - see https://github.com/PowerShell/PowerShell-Docs/issues/1566

I'll address the need for discoverability of preference variables separately.

[iSazonov]

Similarly, the about_preference_variables help topic lists many more preference variables than the 7 named as such in the source code

It's a misleading internal/public terminology again.

[mklement0]

As for the need for programmatic discoverability of automatic / preference variables:

PowerShell (as well as .NET) has a strong tradition of "self-awareness" through reflection.

_Generally_, you'd want to know about special attributes / behavior of variables: whether a variable is defined by PowerShell and whether a variable modifies PowerShell's behavior is certainly worth knowing about.

• This is especially true currently, given that these variables live in the same namespace as _user_ variables, but the argument applies in general.

• Again, my vote is to move these variables into a _separate_ namespace, while grandfathering in the current variables for backward compatibility.

In _particular_, with respect to _preference variables_, #4568 shows why a programmatic way to discover all preference variable is important (though I wish this particular need would go away via a _built-in_ way to honor all caller preference-variable values in the context of functions inside script modules).

[DTW-DanWard]

@mklement0 My 2 cents, if it helps: many years ago I created a PowerShell module framework for a client and ran into this issue. The environment was very heterogeneous; OK, not cross-OS like Core but every flavor of Windows that supported PS v1 and above. Using a separate namespace - an in-memory-only $psf: drive - solved the problem perfectly:

• clean abstraction separate from all other types of PS variables, 3rd party modules, user-specific settings, etc.
• good discoverability: I created a function to walk the $psf: tree and output all variables and values; you could also manually walk the tree if you wanted using tab completion.
• easy to override per machine if necessary: user could run $psf:Folders.Temp = <whatever> in the shell to update for just that session or update their $profile to permanently override the value set when bootstrapping.
  
  
  
  • easy to customize and maintain. The tree was basically a structure of hash tables so it worked well in PowerShell, including tab completion. The tree could be as deep and / or wide as needed.
  
  

All that said: this solution did (and still does) run on 100+ machines, not the millions you have to consider. Plus there is backward compatibility to think about. But if you are interested in making a fresh start with Core, a separate namespace / settings drive would be nice. You could also add some structure, improving organization and discoverability, i.e. $psf:Folders.Home , $psf:Folders.Temp , $psf:Preferences.Verbose , etc.

[mklement0]

Thanks, @DTW-DanWard - good stuff. So you created an in-memory $psf: drive by implementing your own provider?

I agree that a separate namespace makes sense - and PowerShell has the sophistication in terms of features to pull it off (unlike other shells).
The backward-compatibility issue could be solved by continuing to surface the defined-so-far PS-controlled variables in the user-variable namespace while also exposing them via the new namespace.

[DTW-DanWard]

Yep @mklement0 a custom provider. It was a great way to provide a default location for settings that could be quite different based on the machine: dev machines vs. servers, web servers vs. databases vs. , dev environment vs. UAT vs prod, etc. And even though I typically only used it with machine-level settings (not app-level), it grew quickly enough that I still had to organize the contents into 'folders': $psf:Folders, $psf:Applications, $psf:Applications.Database, $psf:Applications.Database.SqlCmd = <path>, etc.

[Jaykul]

I really don't like the idea of creating namespaces for this stuff. Please don't do that.

1. There are a lot of automatic variables already
2. There are a lot of preference variables already
3. PowerShell has already reserved for itself "variables starting with PS"
4. Modules can add variables which they consider "preference" variables (e.g. $PSEmailServer _ought_ to be a preference variable exported by the Microsoft.PowerShell.Utility module where Send-MailMessage is)

For all of these reasons, we cannot really "move" these things. We need to just annotate them where they already are.

Could we not just add "Automatic" (and even "Preference") to the ScopedItemOptions enumeration and set it accordingly? Certainly it would be _much easier_ to do that than to add properties to the variable class or implement a new provider or namespace.

[mklement0]

Re 1. and 2.: The proposal is to grandfather those in and parallel them in the new namespace - and define future automatic / preference variables only in the new namespace.

Re 3: That currently applies to only 20% of the automatic / preference variables. A separate namespace is a conceptually cleaner and more robust solution.

Re 4: Modules - if they have preference variables at all - will have to find their own mechanism, which could be prefix-based; $PSEmailServer seems to be the only current preference variable that is specific to one cmdlet as opposed to pertaining to the session (scope) as a whole or to the engine.

[Jaykul]

@mklement0 your response to 3 is a terrible point: Separate things are conceptually cleaner, but even the ones we have don't get used/followed properly, so let's create another?

Would your new ... thing ... be scoped? All of those preference variables are currently _scoped_, and if you copy them out of the variable drive, the new thing still needs to have variable scope, which would mean your proposed $ps:EmailServer would be introducing people to the dreaded double-drive syntax: $ps:global:Email

Adding a double-implementation of core preference variables sounds like a performance hit, and a HUGE source of confusion. I mean, even if it could be implemented without a performance hit, you'd still have the original variables floating around, and the syntax for the new ones would be confusing!

You would be complicating script analyzer rules and code reviews _for everyone_ --for ever.

I think the whole idea is even worse, if it's implemented in a way that's engine-specific and only PowerShell itself can create variables in it. In that scenario you're would be creating a lot of code and architecture to basically just replace an about_ doc

[vexx32]

Maybe something simpler is in order?

$_ is a well-known automatic variable. Perhaps it's worth setting a convention that automatic variable names should all start with $_ (so we'd end up with things like $_ErrorActionPreference, for example.

Regarding issues with changed names, perhaps it would be wise to simply have the existing automatic variable names point to the new ones by reference (can that be done?). This way, if someone wants to use one of the old names, they're just overriding the for-legacy-purposes reference.

Given that older versions of PS are still in fairly widespread use, we would probably need to put together a PSSA warning or something about scripts developed for older versionsusing the new vars.

A single character change like this would be an easy fix, as well - a simple regex replace like -replace '(?<=$)_(?=[^\s|\n])' could be used to turn them back into their backwards-compatible versions. Then perhaps just warn against use of the older names for legacy reasons, etc., Etc.

There isn't a perfect solution here, because this design wasn't really built with expansion in mind, at least in terms of the number of automatic variables PS uses. They were more brought in as needed. Resulting in strangely semi-standardized things with grey areas.

Case in point: $ProgressPreference
It's not an actual stream like the others, there's really only one cmdlet that utilises it, and as a result it is often mishandled by both users and the engine (cmdletbinding doesn't seem to affect it quite the same as the other preference vars, for example).

[RichardSiddaway]

I don't see the benefit of $_xxx. if you keep the old variable around to avoid breaking tons of code you'll increase the confusion level.

[vexx32]

Agreed. But I think that although I would _like_ to just change the names, somehow I doubt that that is a solution that can be accepted at this point in time.

[mklement0]

@Jaykul:

even the ones we have don't get used/followed properly, so let's create another?

Yes, so as to resolve the existing inconsistencies, once and for all.
In a separate namespace, there is no need for a prefix, so that $PSSessionConfigurationName could become $ps:SessionConfigurationName for instance (I'm leaving aside the question of whether preference vs. automatic variables should occupy distinct namespaces).

The payoff:

• First and foremost: _No more name collisions on introducing new preference / automatic variables_ - no one else can create variables in that namespace.
• No inconsistently applied prefixes.
• Simple, focused discoverability - see scopes discussion below.

Adding a double-implementation of core preference variables [...] and a HUGE source of confusion.

Confusion after transitioning to a different, better solution is unavoidable - it's the price of improving things while maintaining backward compatibility.
Proper documentation and PSSA rules would ease the pain of the transition.
Ideally the rules would be silenced for code explicitly tagged for compatibility with older versions and, conversely, code with no backward-compatibility burden would trigger a warning when using the legacy names.

I think the whole idea is even worse, if it's implemented in a way that's engine-specific and only PowerShell itself can create variables in it.

Considering preference and automatic variables the domain of PowerShell exclusively makes sense to me: they are engine- and whole-session/scope-related (outlier $PSEmailServer notwithstanding).

As an aside: @vexx32, with respect to $ProgressPreference are you referring to the fact that there's no corresponding -ProgressAction _common parameter_? In what way is it mishandled (should a new issue be created)?

---

Would your new ... thing ... be scoped?

Good point - yes, it would have to be scoped.

That concept is currently not a part of the PS provider framework as such, but it could be implemented via a _dynamic_ -Scope parameter - though perhaps that isn't necessary, given that that wasn't even done for the Variable provider in its current form: Get-Item variable:foo only ever gives you the variable _visible or defined_ in the _current_ scope, i.e., the _effective_ value.

Thus, perhaps it is sufficient to integrate with Get-Variable for the full functionality, just as Get-Variable is already needed for full access to existing variables.

Specifically (note: I'm floating ideas here - many details would have to be worked out, and I'm aware that this is a big change):

The Variable provider could be made _hierarchical_ (2-level hierarchy) to support _containers_ that effectively act as _sub-namespaces_ of the current all-in-one variable namespace.

Expressed in namespace notation, that would mean:

Get-Item variable::foo  # user-space variable $foo, as before
Get-Item variable::ps:OutputEncoding  # PS-maintained $ps:OutputEncoding preference variable.

Note that I'm proposing : as the provider's path separator here, which would be a departure from the existing providers (note that the trying to support both \ and / as the universal separators is already problematic).

Now, for namespace notation $ps:OutputEncoding to be consistent with current usage, that would require defining a PS _drive_ as follows, and analogously for all other sub-namespaces:

New-PSDrive ps -Provider Variable -Root ps  # ps: drive with direct access to all $ps:... vars.

On the plus side, this would make the preference variables that are in effect in a given scope discoverable with Get-ChildItem ps:, for instance (analogous to Get-ChildItem variable:)

If proliferation of drives is a concern, perhaps variable-provider containers could be special-cased and be surfaced _implicitly_ as "drives" in the context of namespace notation.

While PowerShell itself would exclusively claim the ps container name (and potentially others, TBD), this would give user code the ability to create variable namespaces too (though name collisions are definitely possible, including with custom _filesystem_ drives).

As for integration into Get-Variable (and the other *-Variable cmdlets): The -Name parameter could be made to accept _paths_:

Get-Variable ps:OutputEncoding # or Get-Variable -Namespace ps OutputEncoding

As for sub-namespace discovery:

Get-ChildItem variable:: | ? PSItemType -eq 'Container'

would work, but is a little clunky, so perhaps adding a parameter such as -ListNamespaces to Get-Variable is called for.

Also, there's the issue of container names such as ps potentially colliding with existing regular variables.
A simple way around that is to allow a variable and a container to have the same name - awkward, but not unheard of: in the Windows registry, a given key can have a _value_ and a _subkey_ with the same name.
The alternative is to stick _everything_ into containers, with the current variables put into default, which would only require redefining the variable: drive slightly, with -Root default, and making Get-Variable look in container default, if a name without namespace prefix is given.
Hypothetically, legacy code that used the _provider prefix_ variable:: rather than the _drive name_ variable: would be impacted, but that sounds unlikely.
On the plus side, Get-ChildItem variable:: would then list all available sub-namespaces (only).

Generally, as hinted at before, due to use of namespace notation, the names of variable sub-namespaces would themselves share a namespace with PS drives of _any_ type (provider), including custom _filesystem_ drives, so existing code with a custom ps: drive is hypothetically a problem.

would be introducing people to the dreaded double-drive syntax: $ps:global:Email

First, accessing preference variables in a different scope strikes me as a rare use case.
If needed, $global:ps:EmailServer seems acceptable (the scope modifier should come first), and
Get-Variable ps:EmailServer -Scope Global is always an option, and needed for relative-scope access anyway.

[vexx32]

@mklement0 RE: $ProgressPreference -- it has been my experience that setting $ProgressPreference is only reliable in the current scope. I'm not sure if other cmdlets simply routinely modify it themselves for some reason, or if there are issues in the underlying code that handles the variable.

This, combined with the very... clunky... nature of Write-Progress for many purposes which we would typically want to use it for, results in me simply... not... using it, at all. Write-Progress is great when it works, but by and large it is much more difficult to use than one would expect, and the effort one needs to input to get it working satisfactorily is typically far more than the task is worth.

[SteveL-MSFT]

@vexx32 seems like you should open a separate issue on Write-Progress and suggestions on how to make it more usable

[Jaykul]

@mkelement0 you keep doubling down and making it even more complicated...

I think your proposed benefits are just an illusion. The name collisions and inconsistently applied prefixes are all still there, because as everyone keeps saying: you can't get rid of the current variables without breaking the world. People have already been told There's not better discoverability, you've actually buried the things in a folder ... people would need to read a document to even learn about that and this document would have to explain the old values as well as the new provider, because most people need to write code that works in older versions of PowerShell ... like 6.1

To me, it seems obvious that a simple property (or additional value in the options enum) is a much simpler option that keeps the actual code backwards-compatible, while still enhancing visibility (especially paired with a change to the view).

P.S. using : as a separator for the _variable_ namespace would be really confusing, because it's already the separator for scopes. Don't you write $global:ProgressPreference? I never see code using the variable cmdlets just to access global or script (or local) scope -- it's usually only used when you need to set options, or access a scope _by index_.

P.P.S. I'm strongly opposed to adding hierarchy to a provider that's already scoped - even a new provider would be better.

[mklement0]

@Jaykul

The name collisions and inconsistently applied prefixes are all still there, because as everyone keeps saying: you can't get rid of the current variables without breaking the world.

They will continue to be there for backward compatibility.
They will be documented as deprecated (without fear of removal), and users will be encouraged to use the new namespace(s) going forward.
PSSA will provide guidance, as suggested.

There's not better discoverability, you've actually buried the things in a folder

Something like Get-Variable ps:* or Get-Variable -Namespace ps seems pretty discoverable to me.

To me, it seems obvious that a simple property (or additional value in the options enum) is a much simpler option

Tagging variables is fine, but it does not address the primary problem with how preference / automatic variables are currently maintained: _every time a new one needs to be introduced, existing scripts may break_.

people would need to read a document to even learn about that

Yes, when a new feature is introduced, people need to learn about it, if they want to avail themselves of it (they don't have to, thanks to backward compatibility, though they'll hopefully see the benefits of the new behavior).

most people need to write code that works in older versions of PowerShell .

By that reasoning, all innovation in PowerShell should be put on hold.

That improvements may take a while before they are usable in production is no reason not to introduce them.

using : as a separator for the variable namespace would be really confusing

I think it would be just fine in practice, given that : _already_ does double duty as the scope-modifier separator ($global:LASTEXITCODE) and as the drive-name separator with namespace notation ($env:HOME).
The only thing new would be _combining_ the two, but all you'll need to remember is that the scope modifier goes first As is already the case^[1], the namespace (drive name) goes first (and creating drives / sub-namespaces named global, script or local would have to be prevented).

I'm strongly opposed to adding hierarchy to a provider that's already scoped - even a new provider would be better.

You needn't think of it as a hierarchy; think of it as partitioning into sub-namespaces.
That is, you'd access variables in the default namespace as before ($foo) or with a namespace qualifier in a given namespace ($ps:OutputEncoding).

Re new provider: obviously, most of the logic is already in the Variable provider and we're talking about simply partitioning the set of variables.

---

^{[1] Even though it's not supported consistently, the current namespace notation requires that the scope modifier _follow_ the drive name; e.g., $function:global:help returns the definition of the help function in the global scope.}

[mklement0]

Correction re sequence of namespace (drive name) and scope modifiers: as is already supported (albeit inconsistently), the namespace goes _first_ (e.g., $function:global:help) - I've updated my previous comment accordingly.

[romero126]

This is a pretty rough read honestly. Given that there is this big conversation regarding creating variable pathing. My question is how does the proposed arguments add to and not obfuscate the base idea that $scope:variable exists.. as well as not muddy the waters diverging too far away from how variables work within the confines of C# and other scripting languages.

I believe adding a path would detract so much from the language that it would confuse a lot of people about how scoping works. Especially if path'ing would exist in the global space. Making Garbage Collection a little bit more murky.

Instead I recommend adding an Attribute to the Variable. and the ability to look up the variable by AttributeType. [PSAutomaticVariableAttribute].

So example code would be
Get-Variable * -AttributeType [PSAutomaticVariableAttribute]
Which would return $_, $PSItem, $Error.

I believe this would also avoid an entire rewrite of PSVariable module and help keep the language similar to other languages (C#/Python/Ruby/etc) and help implement a Pretty Awesome Feature.

[vexx32]

I agree. Having variable paths is a nice idea in concept, but in practice it just means burying them further down out of the immediately accessible values.

Maintaining the PS prefix for any automatic variables like this would seem to be the best option, supplemented with the ability to specifically search for them within the variable provider by adding these designation(s) as a property or attribute of the variables themselves, per @romero126's suggestions.

[mklement0]

@vexx32: As someone once said:

I think that while by and large PowerShell tends to backwards compatibility there are cases where it makes sense to break a few eggs to make a better omelet. 😄

And here's the good news: No eggs need harming in the implementation of my proposal:
Grandfathering in the existing variables means old code won't break.

Maintaining the PS prefix for any automatic variables like this would seem to be the best option

Yes, _going forward_ you can commit to PS-prefixing future automatic variables, but that can still only be a _convention_, if you want to preserve backward compatibility.

Yes, implementing sub-namespaces would be a _new concept_ that users will have to learn, but I think it's worth it in the long run.
It also fits in well with namespace variable notation (e.g., $env:HOME), which isn't currently well-known as a general concept, but deserves to be.