Powershell: Path is not parsed correctly if '[' is present in the path

[krishnayalavarthi]

Steps to reproduce

$sess = New-PSSession -ComputerName localhost
Copy-Item -Path E:\test.txt -Destination 'e:\[test\' -ToSession $sess

Expected behavior

Should copy the file E:\test.txt to E:\[test directory

Actual behavior

Failed as '[' is seen as regex

Environment data

Applicable to all versions

[iSazonov]

It looks like Path vs LiteralPath. If users types special char in path they should escape it or use LiteralPath. It is simple and predictable UX. If a path comes from file system it should be considered as literal.

[mklement0]

@iSazonov, note that the problematic path is passed to the -Destination parameter, where no distinction between literal and wildcard-based paths is offered.

In general, for _output_ paths, it only every makes sense to interpret them as _literal_ paths , so this is worth fixing - just as we're about to do for the -OutFile parameter of Invoke-WebRequest / Invoke-RestMethod (see #11701 - incidentally, please change that PR's title).

[krishnayalavarthi]

@iSazonov @mklement0 , agree that -Destination parameter should be treated as literal. Right now it doesn't work with c:\[test directory, and hence needs to be fixed to make it work as literalpath.

Also, even if we consider regex scenario, having [ alone in the directory is not a regex and should be seen as literal path. In both cases it makes sense fixing it.

[mklement0]

As an aside, @krishnayalavarthi, we're talking about _wildcard expressions_, not regexes.

Agreed, given that when remoting isn't involved, something like Copy-Item foo.txt [foo.txt seems to quietly fall back to a literal interpretation of the path.

It's implied by your repro steps, but to spell it out: the problem only arises in the context of _remoting_ (even though background jobs use the same serialization infrastructure, I don't see it there).

Also note that remoting still targets _Windows PowerShell_ by default, so it's better to start investigating with a command that also targets PowerShell Core:

$sess = New-PSSession -ComputerName localhost -ConfigurationName PowerShell.7

[krishnayalavarthi]

All the below commands have issue with this invalid wildcard in the path parameter.
PR #12170 solves the below failures.

PS C:\WINDOWS\system32> Test-Path E:\[test

Test-Path : Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test At line:1 char:1 Test-Path E:\[test
~~~~~~~~~~~~~~~~~~ CategoryInfo : InvalidArgument: (:) [Test-Path], ParameterBindingException FullyQualifiedErrorId : GetDynamicParametersException,Microsoft.PowerShell.Commands.TestPathCommand

PS C:\WINDOWS\system32> Get-Item E:\[test -ea SilentlyContinue
Get-Item : Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test At line:1 char:1

Get-Item E:\[test -ea SilentlyContinue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CategoryInfo : InvalidArgument: (:) [Get-Item], ParameterBindingException FullyQualifiedErrorId : GetDynamicParametersException,Microsoft.PowerShell.Commands.GetItemCommand

PS C:\WINDOWS\system32> Remove-Item E:\[test -Force -ea SilentlyContinue
Remove-Item : Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test At line:1 char:1

Remove-Item E:\[test -Force -ea SilentlyContinue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CategoryInfo : InvalidArgument: (:) [Remove-Item], ParameterBindingException FullyQualifiedErrorId : GetDynamicParametersException,Microsoft.PowerShell.Commands.RemoveItemCommand

PS C:\WINDOWS\system32>

[mklement0]

@krishnayalavarthi, these are examples of _input_ paths, which by default bind to the -Path parameter, which _does_ expect wildcard expressions (and must continue to do so).

To pass literal paths, use the -LiteralPath parameter (its short alias in PS Core is -lp).

That said, I cannot reproduce your symptoms, neither in direct invocation nor via remoting.

[krishnayalavarthi]

I just installed Powershell 7. Below are repro steps.

I agree that by-default if -Path is not given, the path binds to the -Path which should accept wildcard expressions.

Here we have the scenario of "invalid wildcard". If just [ is given in the path (in this case E:\[test without completion bracket ]), the path should be treated as literalpath and should continue, instead of failing with the below errors.

PS C:\Program Files\PowerShell\7> Test-Path -Path 'e:\[test'
Test-Path: Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test

PS C:\Program Files\PowerShell\7> Test-Path -LiteralPath 'e:\[test'
True
PS C:\Program Files\PowerShell\7>

[mklement0]

Good point, @krishnayalavarthi - the reason I didn't see the symptom is that these commands behave differently _if no such file exists_; since I have no E: drive, I actually got a meaningful error message.
If the drive exists, but not the path as a whole, Test-Path works as expected, but Get-Item returns nothing and doesn't complain.

That it would fail _if the file actually exists_ is confusing and unhelpful.

I agree that falling back to a literal path is the more sensible behavior.

As a minor side effect, something like Get-Item C:\[NoSuch would start complaining (it currently doesn't), but I think that too is more sensible behavior.

[krishnayalavarthi]

@mklement0 I didn't notice this behavior earlier. Agree with you.

PS C:\Users> test-path E:\[test
Test-Path: Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test

PS C:\Users> test-path E:\[nosuch
False

PS C:\Users> Get-Item E:\[test
Get-Item: Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test

PS C:\Users> Get-Item E:\[nosuch
PS C:\Users>

@iSazonov / @mklement0 : Do you think I can use the #12170 for this issue? This PR will check if the path has invalid wild card (just [), in such case it will treat the path as literalpath and proceed. This will fix the above issue. Your comments pls.

[iSazonov]

This PR will check if the path has invalid wild card (just [), in such case it will treat the path as literalpath and proceed.

We already have follow fix in other places: check the path exists as literal, if no process as wildcard.

[krishnayalavarthi]

We already have follow fix in other places: check the path exists as literal, if no process as wildcard.

Sorry, I didn't get the point. You are saying fix for "check the path exists as literal, if no process as wildcard." . In yes, we should not be seeing the below error... right?

PS C:\Users> Get-Item E:\[test
Get-Item: Cannot retrieve the dynamic parameters for the cmdlet. The specified wildcard character pattern is not valid: [test

As the path has invalid wildcard char [ (just single [ instead of []), this should have been treated as literalpath and reported True as the path already exists in my file system.

[iSazonov]

The fix I mentioned is for WorkingDirectory. There is still another scenario fixed in the same way.

[mklement0]

@iSazonov, the behavior that @krishnayalavarthi is proposing (if I understand correctly) makes for consistent behavior:

• Something that _cannot be_ a wildcard expression, because it is _syntactically invalid_, should be treated as a _literal_ path.

As stated, that means that a command such as Get-Item C:\[NoSuch, which currently fails silently, would then complain, because no file _literally_ named C:\[Nosuch is then found, which would trigger the Cannot find path '...' because it does not exist. error.

• Are you concerned about this aspect of the proposed fix for backward compatibility reasons?

• Are you proposing that Get-Item C:\[NoSuch continue to fail silently (as opposed to successfully targeting a file C:\[DoesExist literally even with (implied) -Path)?

Note that the fixes you mention, if I understand correctly, sound like a _deliberate inconsistency_ for when wildcard patterns - notably including syntactically _valid_ ones - happen to match an existing file _if interpreted as literals_. An inconsistency, I presume, that was intended to ease the pain of previously having to use -LiteralPath explicitly in order to unambiguously pass paths as literals.

We've run into this before: #6733 proposed that an invalid wildcard pattern _consistently_ be treated as an error, which would be another way to resolve this - but it is at odds with the fixes you mention.

Either way, two fixes are required:

• Fix the always-inappropriate wildcard interpretation of _output_ file parameters such as -Destination - they should always be considered literal paths.

• In line with the fixes you mention, make a wildcard pattern that _as a literal_ matches an _existing item_ fall back to a literal, for convenience (even though in the case of _valid_ wildcard patterns that amounts to an inconsistency).

_Consistently_ falling back to literals is what makes the most sense to me, not least because that's how it has always worked with paths that contain no wildcard metacharacters at all: Get-Item NoSuch is effectively treated as _literal_ and therefore results in Cannot find path '...' because it does not exist. - even though it is _technically_ also a valid wildcard expression.

Of course, the horribly treacherous combination with Get-ChildItem -Recurse is the regrettable exception: see #5699.

[iSazonov]

Are you concerned about this aspect of the proposed fix for backward compatibility reasons?

Yes, it looks like a breaking change.

Are you proposing that Get-Item C:[NoSuch continue to fail silently

No. Suggestion is to do (1) Get-Item -LiteralPath , then to do (2) Get-Item -Path
Then I'd agree that Get-Item C:\[NoSuch could return Get-Item: Cannot find path 'C:\[NoSuch' because it does not exist. but I think we can do without complicating wildcard analyzes for this.

We've run into this before: #6733 proposed that an invalid wildcard pattern consistently be treated as an error

In general it is impossible to ensure by parsing that a path is valid - only underlying file system can return reliable result. So the proposed steps (Literal, then Path) is best that we can do.

Since you mentioned #5699 I guess there is a common code that brings the behavior. In the case a fix should be common too and it will huge breaking change.

[iSazonov]

For reference #4076

[mklement0]

Suggestion is to do (1) Get-Item -LiteralPath , then to do (2) Get-Item -Path

That's what @SteveL-MSFT also just proposed in https://github.com/PowerShell/PowerShell/issues/6733#issuecomment-604136181 (and @vexx32 agreed), and it makes sense, given the history of this problem and inconvenience of having to _know of_ and _explicitly use_ a separate parameter for literal paths (-LiteralPath).

Then I'd agree that Get-Item C:[NoSuch could return Get-Item: Cannot find path 'C:[NoSuch' because it does not exist. but I think we can do without complicating wildcard analyzes for this.

Glad to hear it: As for complicating the wildcard analyzer: All we need to know up front is whether a
given input is a _syntactically valid_ wildcard pattern, and only a few edge case are _not_:

In short: Only patterns that either contain [] or have only a [ without a closing ] (but not vice versa) are invalid (e.g., test[].txt, test[.txt).

[WildcardPattern]::new('C:\[test') doesn't throw an exception _on construction_, but calling the .IsMatch() function does: [WildcardPattern]::new('C:\[test').IsMatch('foo')

In general it is impossible to ensure by parsing that a path is valid

That is not the goal here.

---

So the proposed steps (Literal, then Path) is best that we can do.

As stated, it's probably the best compromise under the circumstances, but let's be clear on the tradeoffs:

It means that passing something _intended_ to be a wildcard can situationally and unexpectedly be treated as a literal.

For instance, I'm getting the wrong match if I'm looking for -Path file[12].txt to match files file1.txt and/or file2.txt, and a file _literally_ named file[12].txt happens to be present - and there is _no workaround_.

To only way to avoid this pitfall is to _consistently_ treat any -Path value as a wildcard - and complain if it's syntactically invalid - and to treat _only_ -LiteralPath values as literals.

Needless to say, that is inconvenient, and a perennial pitfall, especially to those who may not even be aware of the presence of the -LiteralPath parameter.

POSIX-like shells have solved this problem more succinctly: if the argument is _unquoted_, it is a wildcard; if it's _quoted_, it is a literal (that it is the shell itself rather than the target command that does the matching is a separate issue); e.g., file[12].txt asks for wildcard matching, 'file[12].txt' asks for a literal.

In a manner of speaking: it is the _data type_ of the argument that unambiguously selects the desired behavior, akin to how /..../ is a regex literal in JavaScript, distinct from '...' (strings).

Note that the POSIX-shell approach _is_ already implemented, but only when passing arguments to _external programs_ on _Unix_ (of necessity, for compatibility): /bin/echo *.txt expands *.txt and passes the resulting file names as individual arguments, whereas /bin/echo '*.txt' passes *.txt _verbatim_.

As a general solution, we've previously discussed adopting the same distinction _generally_ or doing the _inverse_ :
Given that we're stuck with the wildcard behavior as the _default_ (positional binding to -Path), we could provide a succinct way to signal the desire to _escape_ the pattern in order to treat it _literally_, which sigil sequence @` could do (with the ` suggesting _escaping_); e.g.,
Get-Item @`file[12].txt would ask that the pattern be _escaped_ and would effectively be the equivalent of Get-Item -LiteralPath file[12].txt.
The same approach could be extended to _regex_ contexts, including in expression mode, so that you could write 'foo|bar' -replace @\'foo|' as the succinct equivalent of 'foo|bar' -replace [regex]::Escape('foo|')

[iSazonov]

It means that passing something intended to be a wildcard can situationally and unexpectedly be treated as a literal.

Get-Item -Path E:\abc[test].txt

If we will check literal and the file exists what is a priority: return a result for the literal or process wildcard? Taken into account -Path parameter I think we should process wildcard and if result is nothing then check the literal. If the literal does not exist too what is an error we should return? Path not found?

To only way to avoid this pitfall is to consistently treat any -Path value as a wildcard - and complain if it's syntactically invalid

I still believe we have no need for wildcard complain. User will get "Path not found" and can review its input.

[mklement0]

Taken into account -Path parameter I think we should process wildcard and if result is nothing then check the literal.

I agree.

If the literal does not exist too what is an error we should return? Path not found?

• If it is a syntactically _valid_ wildcard expression, _nothing_ should be output, as before.

• If it is a syntactically _invalid_ wildcard expression, there is no reason to even _attempt_ wildcard matching, and only literal matching should be attempted - and that will give you the "Path not found" error, if no such item is found.

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this, our recommendation:

• if the wild card is invalid, treat as literal
• if the path expects wild card matching, then wild card matching happens first and if nothing matches, then treat as literal
• document that to escape wildcards, one needs to escape the escape character when using double quotes:

write-host "``["
`[

or using single quotes

write-host '`['
`[

• for cmdlets that don't respect the single escaping (e.g. set-location), we should investigate a fix so that you don't need to escape it multiple times