Powershell: Windows (NTFS): Inconsistent support for the long-path prefix (\\?\) in filesystem paths

[mklement0]

Note: This is a generalization of #4439. @SteveL-MSFT, if you agree, please close the latter.

Prefixing full, normalized, native filesystem paths with \\?\ allows targeting filesystem items whose path is longer than the legacy limit of 259 characters.

Update: Prefix \\?\ is _never_ needed in PS Core (.NET Core altogether, so long paths just work as-is, even if not enabled system-wide. However, old code may still use it, and even new code may have to, when creating cross-edition scripts.

As an aside: In Windows 10 you can now opt in system-wide to support long paths, but individual applications must _also_ opt in (PowerShell does) - see https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-4-6-2-and-long-paths-on-windows-10/

As of PowerShell Core 7.0.0-preview.4, \\?\-prefixed paths _are_ supported if you use the -LiteralPath parameter (as opposed to (implied) -Path) with cmdlets such as Get-ChildItem and Get-Item (this is a regression from Windows PowerShell).

_Update as of PowerShell Core 7.0.0-rc.2_: Get-Item -LiteralPath is now broken, too (Get-ChildItem still works).

However, \\?\-prefixed paths are _not_ supported in the following scenarios:

• Using the -Path parameter (which is implied if omitted) with cmdlets such as Get-ChildItem and Get-Item (but, curiously, it does work with Remove-Item; -LiteralPath works).

  • This is consistent with @jeffbi's comment at https://github.com/PowerShell/PowerShell/issues/4439#issuecomment-319578300, "It looks to me like this is more in the globbing code than in the provider itself." - i.e, the problem is related to interpreting the argument as a _wildcard_ expression.

• Using > / >>, which implicitly behave like Out-File -Path - see #4726

• In the same vein, _invocation of an executable_ - whether with or without & - is broken (possibly related to #4726 as well).

• Start-Process

• New-Item

• Set-Location

Note:

• Some of these problems are regressions from Windows PowerShell, where only the invocation / Start-Process tests fail and > only with a _new_ file.

• I haven't looked into whether invoking an _executable_ with an overly long path is supported in principle by the underlying APIs.

Steps to reproduce

Run the following from a Pester test script (*.Tests.ps1) on Windows:

Describe "Support for long paths, via \\?\" {
  BeforeAll {
    Push-Location (Convert-Path TestDrive:\)
    $dirPath = $PWD.ProviderPath
    $PrefixedDir = "\\?\$dirPath"
    $fileName = ('a' * 248 + '.cmd')
    $fileNameAlt = ('b' * 248)
    $PrefixedFullName = "$PrefixedDir\$fileName"
    $PrefixedFullNameAlt = "$PrefixedDir\$fileNameAlt"
    # Create the file with the overly long path using .NET,
    # to avoid issues with New-Item
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "Get-ChildItem -LiteralPath" {
    Get-ChildItem -LiteralPath $PrefixedFullName | % FullName | Should -Be $PrefixedFullName
  }
  It "Get-ChildItem -Path" {
    Get-ChildItem -Path $PrefixedFullName | % FullName | Should -Be $PrefixedFullName
  }
  It "Remove-Item -LiteralPath" {
    { Remove-Item -LiteralPath $PrefixedFullName } | Should -Not -Throw
    # Recreate the file.
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "Remove-Item -Path" {
    { Remove-Item -Path $PrefixedFullName } | Should -Not -Throw
    # Recreate the file, if necessary
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "> with new file" {
    { '' >  $PrefixedFullNameAlt } | Should -Not -Throw
  }
  It "> / >> with existing file." {
    { '@echo Hi.' >  $PrefixedFullName } | Should -Not -Throw
    { 'REM ' >> $PrefixedFullName } | Should -Not -Throw
  }
  It "Invocation / &" {
    & $PrefixedFullName | Should -Be 'Hi.'
  }
  It "Start-Process" {
    Start-Process -FilePath $PrefixedFullName
  }
  It "New-Item" {
    { New-Item -Force -Type File $PrefixedFullName } | Should -Not -Throw
  }
  It "Set-Location -LiteralPath" {
    { Set-Location -EA Stop -LiteralPath $PrefixedDir } | Should -Not -Throw
  }
  It "Set-Location -Path" {
    { Set-Location -EA Stop -Path $PrefixedDir } | Should -Not -Throw
  }
  AfterAll {
    # Use .NET to remove the overly long path, so that Pester itself doesn't 
    # fail on trying to remove the dir. underlying TestDrive:
    [IO.File]::Delete($PrefixedFullName)
    [IO.File]::Delete($PrefixedFullNameAlt)
    Pop-Location
  }
}

Expected behavior

All tests should pass.

Actual behavior

All tests but the first one fail, with various error messages.

Environment data

PowerShell Core 7.0.0-preview.4

[iSazonov]

LocalGlobbing code is very sensitive and it seems it is not covered by tests explicitly - I think it is main stopper of a progress here.

[mklement0]

As of PowerShell Core 7.0.0-rc.2 Get-Item -LiteralPath \\?\C:\Windows is now broken too (complains about not finding the path; Get-ChildItem -LiteralPath \\?\C:\Windows still works).

I've noticed that the \\?\ prefix is never _needed_ in PS Core. Still, people may use it in cross-edition scripts.

[iSazonov]

I've noticed that the \?\ prefix is never needed in PS Core.

Long paths is Windows feature, not PS Core. It seems we have a test for long path support.

[mklement0]

Yes, it only applies _on Windows_, as hopefully clearly implied by the issue title. PS Core should support the long-path prefix there too, and support has degraded from Windows PowerShell, and even further since v7.0.0-preview.4

What I was trying to say: In PowerShell Core _on Windows_, you can seemingly access paths longer than 259 chars. even _without_ the \\?\ prefix, irrespective of whether long-path support is turned on system-wide.

[mklement0]

It seems we have a test for long path support.

You can run the tests from the initial post; for me, 7 out of the 11 tests fail as of v7.0.0-rc.2

[iSazonov]

What I was trying to say: In PowerShell Core on Windows, you can seemingly access paths longer than 259 chars. even without the \?\ prefix, irrespective of whether long-path support is turned on system-wide.

I don't understand how PowerShell Core can work with long paths if the feature is not enabled on Windows 10.

[mklement0]

That is a fair question, @iSazonov - I haven't looked into the why and how, but you can try it yourself:

First, open an _elevated_ session and run the following, to turn system-wide support for long paths off:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 0

Then, open a regular session and run the following:

Set-Location; 'hi' > ('x'*(259 - "$pwd".length)); (Get-Item x*).FullName.Length; Remove-Item x*

You'll get no error message and 260 as the output, which is the longer-than-259-chars. path of the temporarily created file.

Running the same code in _Windows PowerShell_ fails.

Perhaps \\?\ is being applied behind the scenes?

[iSazonov]

@mklement0 Please share your Windows 10 version. I guess latest versions can have the feature enabled by default.

[mklement0]

It's Microsoft Windows 10 Pro (64-bit; Version 1909, OS Build: 18363.592), but I don't think that is the issue, because the very same code also works on Windows _7_ (where system-wide support for long paths didn't even exist).

[iSazonov]

@mklement0 If you replace 259 in your test with more large value (what about 400?) you get an error.

[mklement0]

That is an unrelated constraint: even in long paths the _individual components_ must not be longer than 255 chars., as far as I know. You'd have to design the test differently to get longer paths.

[iSazonov]

Ah :-) my bad.
Get-Item really doesn't process long paths - it split path and works with leaf (the individual component < 256) but returns FileInfo object with FullName which is > 255 (266 chars). Perhaps just the fact confuses us.
Then I tried to Set-Location and create subdirectory but get error on Set-Location. So long paths do not works.

[mklement0]

I don't see the behavior you're seeing; try the following code, which creates a directory path that is itself longer than 259 chars., then a file in it, targets the directory with both Get-Item and Get-ChildItem and the file with Get-Item, changes to the directory with the overly long path with Set-Location, removes the file with the overly long path, then cleans up - it all seems to work fine, both on Windows 7 and Windows 10.

Push-Location ($dir = mkdir -force ($env:TEMP + "\$pid\" + 'x' * 250)); $file = "$pwd\test.txt"; 'hi' > $file; "dir path length: $($dir.FullName.Length)`nfile path length: $($file.length)"; Get-Item $dir; '###'; Get-ChildItem $dir; '###'; Get-Item $file; '###'; Set-Location $dir -passthru; Pop-Location; Remove-Item $file; Remove-Item (Split-Path $dir) -Recurse

Output on my Windows 10 machine:

dir path length: 288
file path length: 297


    Directory: C:\Users\jdoe\AppData\Local\Temp\5816

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           1/24/2020  4:59 PM                xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                                  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
###

    Directory: C:\Users\jdoe\AppData\Local\Temp\5816\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           1/24/2020  4:59 PM              4 test.txt
###
-a---           1/24/2020  4:59 PM              4 test.txt
###

Drive        : C
Provider     : Microsoft.PowerShell.Core\FileSystem
ProviderPath : C:\Users\jdoe\AppData\Local\Temp\5816\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Path         : C:\Users\jdoe\AppData\Local\Temp\5816\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[iSazonov]

@mklement0 I found why I don't understand you :-)
https://github.com/PowerShell/PowerShell/pull/3960/files#diff-52b1a915619c71b288b3f92f944924c4R1156-R1162

GitHub

Allow use of long paths by jeffbi · Pull Request #3960 · PowerShell/PowerShell

Fixes #3891 When calling Windows native API to determine if an item exists, ensure the path is prepended with \\?\ to allow for paths > 260 characters.

[mklement0]

😁 - thanks for digging deeper.

[SteveL-MSFT]

.NET Core itself supports long file paths implicitly, while .NET Framework requires app opt-in. I get that users may want to use the \\?\ legacy syntax, but it's not clear to me how impactful that is to motivate spending the time to fix this.

[mklement0]

Good to know that .NET Core supports long paths by default - seemingly even without the system-wide opt-in.

The code that @iSazonov linked to seems to no longer be part of the repo (except for the tests, which don't use the \\?\ prefix).

I get that users may want to use the \?\ legacy syntax

• They _may have done so in the past_, which means that old code with \\?\ prefixes that would otherwise be compatible breaks.

• They may _have to going forward_, if they want to write cross-edition code.

Given that the current behavior is clearly a regression from Windows PowerShell, I suggest either:

• fixing the behavior (which you seem disinclined to do)

• documenting the _breaking change_.

[iSazonov]

I am not sure we can fix this now. I think it _must_ necessarily be in FileSystem provider V2.

[mklement0]

Understood, @iSazonov.

With the intent to fix this _eventually_, the categorization shifts from _breaking change_ to _(known) bug_, so please label this issue accordingly.

[iSazonov]

Meta bug label :-)

[pjaclark]

This should be listed as a Known Issue causing breaking changes until resolved.

[musm]

Is this related to why I can't create very long paths?

> mkdir ('x' * 290)
New-Item: The filename, directory name, or volume label syntax is incorrect ....

[mklement0]

@musm, no, you're seeing a fundamental NTFS limitation that applies even when long-path support is enabled: _individual file/directory names_ must not be longer than 255 chars (as opposed to the length of a _path_ as a whole).

While you could argue that the error message _technically_ applies, it is certainly vague.