Powershell: Variables with names that include scope modifiers cannot be removed

[vexx32]

Steps to reproduce

$Green = @{ForegroundColor = 'Green'}
Write-Host "Create a variable named literally 'Global:MyVar' in the local scope" @Green
New-Variable -Scope Local -Name Global:MyVar -Value 123
$V = Get-Variable | Where-Object Name -eq Global:MyVar
$V
Write-Host "Try to remove variable 'Global:MyVar'"
$V | Remove-Variable
Write-Host "The variable should no longer be present." @Green
Get-Variable | Where-Object Name -eq Global:MyVar

Expected behavior

Create a variable named literally 'Global:MyVar' in the local scope

Name                           Value
----                           -----
Global:MyVar                   123

Try to remove variable 'Global:MyVar'
The variable should no longer be present.

Actual behavior

Create a variable named literally 'Global:MyVar' in the local scope

Name                           Value
----                           -----
Global:MyVar                   123
Try to remove variable 'Global:MyVar'
Remove-Variable : Object reference not set to an instance of an object.
At line:8 char:6
+ $V | Remove-Variable
+      ~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Remove-Variable], NullReferenceException
+ FullyQualifiedErrorId : System.NullReferenceException,Microsoft.PowerShell.Commands.RemoveVariableCommand

The variable should no longer be present.
Global:MyVar                   123

Environment data

Name                           Value
----                           -----
PSVersion                      6.2.0
PSEdition                      Core
GitCommitId                    6.2.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[kvprasoon]

IMO, New-Variable should not accept variable special chars, because such variables are not possible without ${}

[fMichaleczek]

@kvprasoon
${bla:foo} = 1 => Cannot find drive. A drive with the name 'bla' does not exist
${variable:bla:foo} = 1 => OK

[kvprasoon]

Yes, It works only that way. But New-Variable accepts it and creates nothing. Hence the Bug is in New-Variable as well, or it should allow to create such variables, hence it can be retrieved using ${} notation.

[fMichaleczek]

@kvprasoon

The problem is only with variable with a scope modifier prefix (like global: or local:) and remove-variable

New-Variable -Name global:foo -Value 1
Get-Variable -Name global:foo
${variable:global:foo}
Remove-Variable -Name global:foo => Object reference not set to an instance of an object.

With your previous example, everything works :

New-Variable -Name bla:foo -Value 1
Get-Variable -Name bla:foo
${variable:bla:foo}
Remove-Variable -Name bla:foo

${} notation is for all PSProviders and not only variable, if we use a variable prefix that contains ":", it can't determine which PSProvider you want to use. (see provider path in namespace variable notation inconsistency

[kvprasoon]

Understood, but when New-Variable -Name 'bla:foo' -Value 1 works $bla:foo doesn't return it and is possible only via Get-Variable and namespace notation.

[vexx32]

So, a variable name itself should not be able to have any : characters in it, really. These are used for scope modifiers, and while there are cases where it _appears_ that a given variable might have that as part of its name, almost all usable cases result in that portion of the name being taken as a scope modifier or PSDrive accessor, as intended. What _should not_ be possible is the creation of a variable in the variable: drive where the actual variable name contains a : character.

There isn't really a good reason to allow this, in my opinion, and it makes more sense to expressly forbid it due to the unavoidable issues it causes thanks to its use as a drive or scope qualifier.

[fMichaleczek]

We are all agree that allowing any : character in variable has no sense.

But exception of this issue, it works if you provide the full syntax.

If you want examples of inconsistency with : character, :

New-Variable -Name : -Value 1
New-Variable -Name :: -Value 2 => A variable with name '::' already exists.
Get-Variable -Name :: => Cannot find a variable with the name '::'
${::}
Get-Variable -Name :

[vexx32]

It works _right up until_ you try to remove the variable again.

So.. sort of. 😄

[fMichaleczek]

I wrote "But exception of this issue", example "bla:foo" works !

[vexx32]

Ah, I missed that, sorry! Good point to be aware of... I guess that makes sense. If it doesn't resolve a usable scope name it probably isn't as problematic for the provider / cmdlets to figure out.

[bpayette]

The variable cmdlets certainly appear to have problems but namespace qualified operations work properly:

PS[1] (32) > $variable:global:myvar = 123
PS[1] (33) > $variable:global:myvar
123
PS[1] (34) > rm variable:global:global:myvar
PS[1] (35) > $variable:global:myvar
The variable '$variable:global:myvar' cannot be retrieved because it has not been set.
At line:1 char:1
+ $variable:global:myvar
+ ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (variable:global:myvar:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

So it seems that the issue can be addressed by fixing the variable cmdlets.

[vexx32]

Sure, but does it even make sense to allow the drive/scope delimiter to be usable in a variable name given the additional confusion and disparity it causes in terms of how to define and how to retrieve values when it's used in this way?

[msftrncs]

FYI, access to such variable is possible via a scope, $local:global:myvar, no namespace syntax required.

[msftrncs]

Also, $:global:myvar works. The first colon (in $ syntax) must always be used to describe a scope or drive, the first colon is never part of the variable name. And two colons can only be used together if they are the first two in the $ syntax. Additional : require ${} syntax

[msftrncs]

Talk about weird stuff.

new-variable -name global:myvar -value 1
$:global:myvar # there it is
remove-variable -name global:myvar # oops, error
$:global:myvar # still there
clear-variable -name global:myvar
$:global:myvar # still there
remove-variable -name global:myvar # no error
$:global:myvar # not gone yet!!!
clear-variable -name global:myvar -scope local
$:global:myvar # still there, but empty now
remove-variable -name global:myvar -scope local # error again
$:global:myvar # empty, but still here

I seem to get different results each time I run. Also, test-path variable:global:myvar seems to fail to work as well. Also, if you use clear-variable and remove-variable back to back, it seems you do not get the error message. (but you have to NOT supply the scope parameter to clear-variable!)

[vexx32]

... yep. Looking at that, I can only say that it makes no sense to allow variable names themselves to include a colon character. 😌

[LarryWeiss]

Are there any other characters that are not allowed in a variable's name?

[vexx32]

You can't include { or } in a variable name without escaping them with a backtick, because they're part of the variable name-enclosing syntax:

PS> ${{a} = 1
At line:1 char:4
+ ${{a} = 1
+    ~
Use `{ instead of { in variable names.
+ CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : OpenBraceNeedsToBeBackTickedInVariableName

Note: while } will work, it just ends the variable name if it's not escaped. Neither are usable in a non-enclosed variable name. However, _both_ are currently usable with Set-Variable without escaping, which I am inclined to consider an oversight, as it highly complicates retrieving these variables again.

I think treating : similarly makes sense, as it too is intrinsic to the syntax of how variable naming _works_. If you really want to use it as part of a name instead of a scope modifier, you can escape it with a backtick. Then, we just need to add handling on the back-end to ensure it's handled correctly instead of being interpreted as a scope.

Trying to set such a variable should error out if the : is not escaped and there's already a drive name or scope modifier in the name, or with Set-Variable if -Scope is specified.

[msftrncs]

Lets not forget, that using the ${ } notation, ALL characters are valid, including control characters! Also the same applies to the variable -name argument of the variable CMDLETs, all characters are valid (though wildcards may need to be escaped). The only places there are restrictions are with the $ sigil.

Lets explain, to the best of my knowledge, what is going on in the tokenizer:

• Meets a $, possibly variable or substatement
• Valid second characters are $, ^, {, :, ? or w (all the letter characters in Unicode, which includes _ and numeric digits)
• $ and ^ end the variable name immediately
• { begins a expanded character set name, escaping is required for some characters, but any character is otherwise valid, the name ends with an un-escaped }
• otherwise the name is any : (as long as the next character is also not :), ? or w, until a non allowed character is met.
• If none of the above matches, then an alternate second character is (, which then tokenizes a substatement.
• not a substatement, then the $ is included in any already ongoing token, and that ongoing token continues.

Only after the name is tokenized does drive and scope get decoded, using the first : regardless of the method used to tokenize. A blank drive or scope is allowed, but the first : is then removed with the process, making it not part of the variable name.

So it this point it would complicate tokenizing to change the current behavior. The : becomes an allowed character in the variable name because it must be accepted by the tokenizer in the basic name category in order to capture the drive or scope.

The variable CMDLETs need to be fixed. They neither seem to actually allow a scope to be specified in the -name parameter (which I don't think they should), nor do they work if you do try to, which indicates they are broken, even though they work for all other variable names that include :, plus get-variable / set-variable work correctly. Then the issue would only be a small issue of confusion by new users as to how to specify the scope part to the variable CMDLETs. Do note that the variable CMDLET's accept all characters for variable names, since the variable name is an argument.

I do think its weird that in some places the variable: drive accepts scopes and in some places it doesn't. In the variable namespace notation, it doesn't, but in certain CMDLETs it does, such as 'remove-item' and 'get-content'

EDIT: I previously was in error regarding \p{L}. The actual regex construct equivalent would have been \w.

[vexx32]

We don't need to change tokenizing beyond maybe adding a case similar to what exists for { (e.g, ${a{b} = 2 throws a very intentional error prompting you to use a backtick to escape the brace). I think that may occur in the parser, I'm not sure, would have to look.

Beyond that, we should add checks to ensure the code paths are followed appropriately so we get no errors when we do intentionally try to make such variables. Not sure if that's actually a fault in the cmdlets or if it's more pertaining to the variable provider itself, since I believe similar difficulties arise using the provoder-agnostic *-Item cmdlets. Either way, yeah we need to fix that.

[bpayette]

The tokenizer is here if you want to see how things actually work.

Remember that "variable syntax" is not just about variables. It applies to all providers and providers have different rules about which characters are valid in a variable path. This is why ${...} allows any character. The parser/tokenizer cannot determine that a path is valid, only the associated provider can.

And there are other special cases to consider like:

PS /tmp> $function:global:foo = { "Hi there" }
PS /tmp> foo
Hi there