Vcpkg: Windows builds fail when path is too long

[Deadpikle]

Describe the bug

I'm building boost-system using a custom triplet named x86-windows-static-v141. The build fails with the following error: failed to write output file '[Really long path]\vcpkg\buildtrees\boost-system\x86-windows-static-v141-rel\boost\build\e84828337f108533f7982c73b179c026\libboost_system-vc141-mt-s-x32-1_70.lib.rsp'!. This path with the part I left out is 265 characters long, which is longer than the old file system path limit of 260 characters. If I build without using the custom triplet, things work, presumably because the resulting path is shorter.

Working command: ./vcpkg install boost-system
Not working command: ./vcpkg install boost-system:x86-windows-static-v141

I have LongPathsEnabled set to 1 in the registry.

Environment

• Windows 10 Version 10.0.18362 Build 18362 (Windows 1903)
• Building within Powershell
• Compiler: Visual Studio 2017 (15.9.16) (I assume?)
• Long paths is enabled in the registry
• Trying to build boost-system v1.70.0
• vcpkg version: 2019.09.12-nohash

In case it is important, here is the triplet:

set(VCPKG_TARGET_ARCHITECTURE x86)
set(VCPKG_CRT_LINKAGE static)
set(VCPKG_LIBRARY_LINKAGE static)
set(VCPKG_PLATFORM_TOOLSET v141)

To Reproduce
Steps to reproduce the behavior:

1. Nest vcpkg really deeply on your hard drive (realistically, due to project submodules or similar) / on a long file path
2. ./vcpkg install boost-system
3. See error -- build fails

Expected behavior
The build should succeed (or at least tell me my file paths are too long).

Failure logs
failed to write output file '[Really long path]\vcpkg\buildtrees\boost-system\x86-windows-static-v141-rel\boost\build\e84828337f108533f7982c73b179c026\libboost_system-vc141-mt-s-x32-1_70.lib.rsp'!

Call Stack (most recent call first):
  installed/x86-windows-static-v141/share/boost-build/boost-modular-build.cmake:278 (vcpkg_execute_required_process)
  ports/boost-system/portfile.cmake:14 (boost_modular_build)
  scripts/ports.cmake:91 (include)

Additional context
This might be more of a problem with boost's build system than vcpkg...?

[degski]

@PhoebeHui

In order to mitigate this, someone has to change the command line to b2.exe to add the flag --abbreviate-paths. This will mitigate the problem.

@Deadpikle

For the moment you can just fix the problem by allowing Windows 10 to use long paths. Attached are some .reg files to change or restore the 260 character limit:

Restore 260 Character Path Limit (Default).reg.txt
Remove 260 Character Path Limit.reg.txt

In order to 'install' those, download to some location, remove the .txt extension, right-click in explorer and select merge. Re-boot your system.

[Deadpikle]

@degski I already had LongPathsEnabled set. I rebooted just in case, but that did not fix the issue.

I tried to add --abbreviate-paths to ports/boost-modular-build-helper/boost-modular-build.cmake in the COMMAND "${B2_EXE}" calls, but that was also unsuccessful, unless I did something wrong, which may have happened given that the error log still has quite long paths.

[degski]

@Deadpikle

This path with the part I left out is 265 characters long, which is longer than the old file system path limit of 260 characters.

Iff you merged the correct .reg, it should work. You must have done something wrong. Check with the reg file whether the key is in the right place and has the correct value.

There is also the --hash option, one can add to the command-line. The link also mentions --abbreviate-paths and what it does.

[Deadpikle]

@degski The following gif shows my understanding of how to setup that registry file (without showing a reboot of course). Is this correct?

This is the command that is failing, as shown in the command line:

[vcpkg-path]/vcpkg/installed/x86-windows-static/tools/boost-build/b2.exe 
    --stagedir=[vcpkg-path]/vcpkg/buildtrees/boost-system/x86-windows-static-rel/stage 
    --build-dir=[vcpkg-path]/vcpkg/buildtrees/boost-system/x86-windows-static-rel 
    --user-config=[vcpkg-path]/vcpkg/buildtrees/boost-system/x86-windows-static-rel/user-config.jam 
    --abbreviate-paths 
    -j8 
    --debug-configuration 
    --debug-building 
    --debug-generators 
    --disable-icu 
    --ignore-site-config 
    --hash 
    -q 
    -sZLIB_INCLUDE=[vcpkg-path]/vcpkg/installed/x86-windows-static/include 
    -sBZIP2_INCLUDE=[vcpkg-path]/vcpkg/installed/x86-windows-static/include 
    threading=multi threadapi=win32 runtime-link=static link=static address-model=32 architecture=x86 toolset=msvc 
    -sZLIB_BINARY=zlib 
    -sZLIB_LIBPATH=[vcpkg-path]/vcpkg/installed/x86-windows-static/lib 
    -sBZIP2_BINARY=bz2 
    -sBZIP2_LIBPATH=[vcpkg-path]/vcpkg/installed/x86-windows-static/lib 
    variant=release debug-symbols=on

Working Directory: [vcpkg-path]/vcpkg/buildtrees/boost-system/src/ost-1.70.0-6140d37d5b/build

Error code: 1

Note that this is with my edit for the --abbreviate-paths option, and I'm trying a different triplet. The failed file is still 260 characters long: [path]\vcpkg\buildtrees\boost-system\x86-windows-static-rel\boost\build\07e493aef27bab0531bc692c6dff6000\libboost_system-vc141-mt-s-x32-1_70.lib.rsp. --hash is also being used.

I appreciate your assistance. Are there additional debug flags or anything I can turn on to get more information on why things are failing?

Is it possible the cause is not actually the file limit, but something else? This would be odd since things run fine from my desktop with a fresh vcpkg copy (aka shorter path).

[Deadpikle]

build-x86-windows-static-rel-out.log

Here is the rel-out.log file if that is helpful.

[degski]

@Deadpikle Yeah, you're doing it right. But, from the link in the .reg file:

There is one caveat. This new setting won’t necessarily work with every application out there, but it will work with most. Specifically, any modern applications should be fine, as should all 64-bit applications. Older 32-bit applications need to be manifested in order to work, which really just means that the developer has indicated in the application’s manifest file that the application supports longer paths. Most popular 32-bit apps should experience no problem. Still, you don’t risk anything by trying the setting out. If an application doesn’t work, the only thing that will happen is that it won’t be able to open or save files that are saved in places where the full path exceeds 260 characters.

Is your OS 32-bit?

[Deadpikle]

@degski I'm running 64-bit Windows 10. For fun, I tried running vcpkg with cmd, Powershell, and a git bash terminal. Didn't work on all of those. 🤔

Edit: I don't want to be a stick in the mud -- I realize I can build on some other path and do integrate install or move the files or similar, but that is just a workaround and every dev would need to do it (if their paths demanded that fix). 😅

[degski]

@Deadpikle It's a mystery. I've got Boost compiled and installed, and I guess many people with me [with whatever is included in vcpkg]. I guess your b2.exe is 32-bit, but mine must be as well. I'm running same OS, same build and use VS2019 (but I doubt that matters at all).

PS: I've got vcpkg installed in the root of a partition.

[PhoebeHui]

@degski, adding --abbreviate-paths to b2 to keep paths from becoming longer than the filesystem supports, it reduces each element to no more than five characters. For example, link-static becomes lnk-sttc. The --hash option reduces the path to a single directory using an MD5, Referred from --abbreviate-paths, the issue here is VCPKG Root directory is too long, so that option limit to solve the issue.

Setting LongPathsEnabled in registry seems only works in Powershell, but still not in explore, so after enable long, I noticed that I was able to create file and folder by Powershell, and long path is more than 260 characters. However, it still not fix the problem, it start failed with 'The input line is too long. LINK : fatal error LNK1104: cannot open file ''', we don't have a solution for this.
Refer from https://social.technet.microsoft.com/Forums/windows/en-US/4e471b83-10c6-49d4-acb5-6654640b9dff/long-path-setting-not-working?forum=win10itprogeneral

I will update boost-system to give a warning to user when the path is too long.

[degski]

@PhoebeHui

... the issue here is VCPKG Root directory is too long, so that option limit to solve the issue.

I don't understand that, the OS has no 'knowledge' of vcpkg. Surely the complete path is the only thing that matters.

Setting LongPathsEnabled in registry seems only works in Powershell, but still not in explore, so after enable long, I noticed that I was able to create file and folder by Powershell, and long path is more than 260 characters.

I'm [always] using simply cmd, and have no problem so far. I do, on the other hand, have vcpkg in the root of a partition, so maybe I'm not hitting the barrier. Having said that, I do have Boost-1.70 installed with the 'old' config, so, I really don't get it.

Well appreciated you're looking into this, btw!

[JackBoosY]

@degski See documentation.

[degski]

@JackBoosY

Thanks, reading a bit more, there seems to be a mix of bugs (in the OS) and apps [including, but not limited to explorer, cmd, b2 and vcpkg] that are not up-to-speed yet [and will they ever be?]. This feature simply seems to be half-baked [in Win10].

The API's exist.

[PhoebeHui]

The PR merged.