Powershell: Test-Path -Path with wildcards may miss hidden items

[nightroman]

Test-Path -Path with wildcards may miss hidden items.

Steps to reproduce

# make a directory "z" and a hidden item "hidden" in it
$item = mkdir z\hidden -Force
$item.Attributes = $item.Attributes -bor [System.IO.FileAttributes]::Hidden

# True, expected
$r = Test-Path z\hidden
"Test-Path z\hidden = $r"

# False, unexpected
$r = Test-Path *\hidden
"Test-Path *\hidden = $r"

Expected behavior

Test-Path gets true in both cases.

Actual behavior

For the existing hidden item "hidden" Test-Path *\hidden gets false.

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      6.0.2
PSEdition                      Core
GitCommitId                    v6.0.2
OS                             Microsoft Windows 6.3.9600
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[BrucePay]

@nightroman In general, this is how wildcarding works: hidden files are not matched by default (exact paths do match). If you want to match hidden files, then you (typically) have to specify -Force. Now Test-Path doesn't have a -Force parameter so that could be considered a bug. Alternatively, Test-Path could be changed to always pass the force flag to the underlying APIs (which would be a gray area breaking change). Finally you could argue that Test-Path shouldn't do wildcarding at all since it's really intended to test for specific paths. In that case, if you wanted to see if group of files exist you could/would use Get-Childitem. What do you think?

[nightroman]

I realize that it is complicated. I am not suggesting anything. But I was caught by the current behavior. I would be happy if it behaves as I described in "expected", breaking or not.

[nightroman]

Test-Path -Path finds hidden items: (1) without wildcards; (2) with wildcards specified for items, not folders. So it is not completely true that "In general, this is how wildcarding works". Test-Path finds hidden items with wildcards sometimes. It depends on a wildcard and this is too complex for understanding. If it works for some wildcards then it should work for all.

P.S. Turns out I am suggesting something after all.

[BrucePay]

(2) with wildcards specified for items, not folders.

I'm not sure I understand quite what you mean by this. Can you provide an example? Here's what I tried:

After creating a non-hidden file in the hidden directory, I tried

PS[1] (91) > test-path z/hidden/file.txt
True

It's an exact path so it returns true as expected.

PS[1] (92) > test-path z/hidden/*.txt
True

This also returns true also as expected - the directory component 'hidden' doesn't contain wildcards (i.e. the path component is exact) so it succeeds.

PS[1] (93) > test-path z/*/file.txt
False

This returns false because the path component corresponding to "hidden" has wildcards and therefore doesn't match.

So far, everything is working as expected.

[nightroman]

Sorry, I answered in a hurry. I thought Test-Path *.txt returned true for hidden *.txt files (i.e. not folders). No, it did not. So it is consistent with wildcards.

[nightroman]

In general, this is how wildcarding works: hidden files are not matched by default (exact paths do match).

At least the help of Test-Path could mention this wildcarding feature.

I was not aware of it, actually. Where else is it used in practice?

[nightroman]

I think Resolve-Path has the same "issue".

[iSazonov]

I think all FileProvider cmdlets should have the same behavior.

/cc @mklement0 Could you please look the Issue?

[BrucePay]

@iSazonov Just to be clear: the behavior for wildcards wrt hidden files is implemented by the FileSystemProvider not by cmdlets. Any cmdlet that uses providers will have this behavior including all of the "core" cmdlets (*-Path, *-Item, *-ChildItem).

@nightroman The help for Test-Path does mention wildcard support in the parameter descriptions e.g. for the -Path parameter:

Specifies a path to be tested. Wildcard characters are permitted. If the path includes spaces, enclose it in quotation marks.

but that's not really what you want. The semantics for how a path is resolved are up to the underlying provider, so this should probably be covered in the FileSystemProvider documentation (it's not).

[mklement0]

Now Test-Path doesn't have a -Force parameter so that could be considered a bug.

I agree. -Force should be added for symmetry with other FileProvider cmdlets - see below.

Finally you could argue that Test-Path shouldn't do wildcarding at all since it's really intended to test for specific paths.

It's definitely useful to use Test-Path with wildcards (e.g.: Are there any text files in this dir? Test-Path *.txt Is there a v4.x .NET framework installed? Test-path C:\WINDOWS\Microsoft.Net\Framework64\v4*), and given that it has a -Path parameter distinct from a -LiteralPath parameter,
wildcards should definitely (continue to) be supported.

Below is my attempt to summarize the status quo; do let me know if got something wrong / missed something.

Overview of the current behavior of the filesystem cmdlets and their inconsistencies

• If there are _no_ wildcards in _any_ path components:

  • Hidden items ARE found by _literal paths_ (whether passed to -Path or -LiteralPath) by Test-Path, Convert-Path, Resolve-Path, Move-Item, Rename-Item, Copy-Item, New-Item, Get-Content, Set-Content, Add-Content, Clear-Content, Split-Path -Resolve, Join-Path -Resolve.

  • Hidden items are NOT found by Get-Item / Get-ChildItem unless -Force is also specified (applies to -LiteralPath too).

  • Note, however, that Get-ChildItem will find (visible) _children_ in a literally matched hidden dir.; e.g., in @nightroman's example, Get-ChildItem z/hidden

• If there are wildcards in _any_ path component:

  • Hidden items ONLY match if -Force is also specified.

  • Given that they lack a -Force switch, the following cmdlets are currently fundamentally incapable of finding hidden items via wildcards:

  • Test-Path, Convert-Path, Resolve-Path, Split-Path -Resolve, Join-Path -Resolve, Invoke-Item

  • New-Item fundamentally lacks wildcard support - only literal paths are supported (and do find hidden items); thus, New-Item's -Path parameter should always have been -LiteralPath instead

  • Additionally, the help topic currently erroneously claims that wildcards _are_ supported - it's incorrect at least with respect to the filesystem provider (haven't tried others).

Note: Set-Item and Clear-Item were not considered above, because the filesystem provider doesn't implement them.

[iSazonov]

I think we could expand the Issue on Test-Path, Convert-Path, Resolve-Path, Split-Path -Resolve, Join-Path -Resolve, Invoke-Item cmdlets and -Force.

[mklement0]

@iSazonov: I've created #6501 and #6502