Vcpkg: Incorrect current path when parsing `vcpkg install --overlay-ports=` parameters.

[qis]

Description

Vcpkg parses all paths provided by the user using the --overlay-ports= command-line parameter as relative to vcpkg root.

Environment

• OS: Windows 10, Ubuntu 18.04.4 LTS in WSL, etc. (all affected)
• Compiler: (does not apply)

To Reproduce

Steps to reproduce the behavior:

1. .\vcpkg\vcpkg.exe install .\some-ports-overlay
2. Observe how the port overlay is not found or not used (in case it overrides an official vcpkg port).

Expected behavior

Relative paths should work according to the documentation.

Additional context

The PathsPortFileProvider uses fs::stdfs::canonical in the constructor to convert paths provided by the user using the --overlay-ports= command-line option to absolute paths. This does not work because it happens after vcpkg changes current path in vcpkg.cpp.

Relative paths are officially supported by vcpkg in the documentation and this functionality is apparently important to people who want to GitHub actions with vcpkg.

Thank you @vadi2 for mentioning this bug!

P.S.: Why does it use _wchdir(paths.root.c_str())/_wchdir(paths.root.c_str()) instead of a more consistent fs::stdfs::current_path(paths.root);?

P.P.S: Shouldn't there be a VCPKG_OVERLAY_PORTS environment variable with a syntax similar to PATH?

[vadi2]

Note that getting the path "wrong" will result in a crash:

Computing installation plan...
vcpkg.exe has crashed.
Please send an email to:
    [email protected]
containing a brief summary of what you were trying to do and the following data blob:
Version=2020.02.04-unknownhash
EXCEPTION='filesystem error: cannot canonicalize: No such file or directory [../lua] [/Users/runner/runners/2.168.0/work/Mudlet/Mudlet/3rdparty/vcpkg]'

[Voskrese]

Environment
OS: Windows 10, Ubuntu 18.04.4 LTS in WSL, etc. (all affected)
To Reproduce
.\vcpkg\vcpkg.exe install .\some-ports-overlay

ubuntu + wine?
why run Windows app from under WSL ?

[Voskrese]

No such file or directory [../lua] [/.../3rdparty/vcpkg]'

ls /Users/runner/runners/2.168.0/work/Mudlet/Mudlet/3rdparty/lua ?

[vadi2]

../lua was relative to the response file which was inside /Users/runner/runners/2.168.0/work/Mudlet/Mudlet/3rdparty/own-vcpkg-ports

[qis]

ubuntu + wine?
why run Windows app from under WSL ?

@Voskrese I have no idea how you jumped to this conclusion. It just means that every platform is affected and I tested it on Windows and Linux.

No such file or directory [../lua] [/.../3rdparty/vcpkg]'

ls /Users/runner/runners/2.168.0/work/Mudlet/Mudlet/3rdparty/lua ?

1. User provides relative path correctly expecting vcpkg to know his current path.
2. Vcpkg changes current directory to vcpkg root before looking at parameters.
3. Vcpkg does not find relative path UNLESS the path matches relative to vcpkg root.

[PhoebeHui]

@qis @vadi2, I can't repro this issue on window 10. On wsl, vcpkg crashed, however, that only happen when you access the custom-ports folder which created by windows system. since the folder does not exsit for linux system. If you use the folder that created by linux system, that should be fine.

~/vcpkg$ ./vcpkg install lua --overlay-ports=../custom-ports/lua
Computing installation plan...
The following packages will be built and installed:
lua[core]:x64-linux -- /home/ubuntu18/custom-ports/lua
Starting package 1/1: lua:x64-linux
Building package lua[core]:x64-linux...
-- Installing port from location: /home/ubuntu18/custom-ports/lua
-- Using cached /home/ubuntu18/vcpkg/downloads/lua-5.3.5.tar.gz
-- Using source at /home/ubuntu18/vcpkg/buildtrees/lua/src/lua-5-292bcd51ee
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /home/ubuntu18/vcpkg/packages/lua_x64-linux/share/lua/usage
-- Performing post-build validation
-- Performing post-build validation done
Building package lua[core]:x64-linux... done
Installing package lua[core]:x64-linux...
Installing package lua[core]:x64-linux... done
Elapsed time for package lua:x64-linux: 28.43 s

Total elapsed time: 28.44 s
Use this package via the module FindLua that comes with CMake. To use in your CMakeLists.txt:
include(FindLua)
find_package(lua REQUIRED)
target_link_libraries(main PRIVATE ${LUA_LIBRARIES})
target_include_directories(main PRIVATE ${LUA_INCLUDE_DIR})
For more information about the variables set by this module, please see:
https://cmake.org/cmake/help/latest/module/FindLua.html

On windows:
PS F:\vcpkg\src> ./vcpkg install sqlite3:x64-windows --overlay-ports=..\custom-ports\sqlite3
Your feedback is important to improve Vcpkg! Please take 3 minutes to complete our survey by running: vcpkg contact --survey
Computing installation plan...
The following packages will be built and installed:
sqlite3[core]:x64-windows -- F:\vcpkg\custom-ports\sqlite3
Starting package 1/1: sqlite3:x64-windows
Building package sqlite3[core]:x64-windows...
-- Installing port from location: F:\vcpkg\custom-ports\sqlite3
use custom-ports sqlite
-- Using cached F:/VCPKG/src/downloads/sqlite-amalgamation-3310100.zip
-- Using source at F:/VCPKG/src/buildtrees/sqlite3/src/3310100-b37a578ce1
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Performing post-build validation
-- Performing post-build validation done
Building package sqlite3[core]:x64-windows... done
Installing package sqlite3[core]:x64-windows...
Installing package sqlite3[core]:x64-windows... done
Elapsed time for package sqlite3:x64-windows: 21.66 s
Total elapsed time: 21.67 s
The package sqlite3:x64-windows provides CMake targets:
find_package(sqlite3 CONFIG REQUIRED)
target_link_libraries(main PRIVATE sqlite3)

[vadi2]

I am not sure you reproduced it correctly - you used a path relative to the vcpkg root, didn't you?

I am using a response file, and in that response file I have to use a path relative to the vcpkg root, not relative to the response file, which is not intuitive: https://github.com/vadi2/Mudlet/blob/add-macos-action-ci/3rdparty/our-vcpkg-dependencies/.vcpkg-osx-dependencies

[PhoebeHui]

@vadi2, thanks for your quick response! see the steps that I tried:

ubuntu18@phoebe:~$ ls
Phoebe  vcpkg
ubuntu18@phoebe:~$ mkdir custom-ports
ubuntu18@phoebe:~$ ls
Phoebe  custom-ports  vcpkg
ubuntu18@phoebe:~$ cp -R ./vcpkg/ports/lua ./custom-ports
ubuntu18@phoebe:~$ cd vcpkg
ubuntu18@phoebe:~/vcpkg$ ./vcpkg install lua --overlay-ports=../custom-ports/lua
Computing installation plan...
The following packages will be built and installed:
    lua[core]:x64-linux -- /home/ubuntu18/custom-ports/lua
Starting package 1/1: lua:x64-linux
Building package lua[core]:x64-linux...
-- Installing port from location: /home/ubuntu18/custom-ports/lua
-- Using cached /home/ubuntu18/vcpkg/downloads/lua-5.3.5.tar.gz
-- Using source at /home/ubuntu18/vcpkg/buildtrees/lua/src/lua-5-292bcd51ee
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /home/ubuntu18/vcpkg/packages/lua_x64-linux/share/lua/usage
-- Performing post-build validation
-- Performing post-build validation done
Building package lua[core]:x64-linux... done
Installing package lua[core]:x64-linux...
Installing package lua[core]:x64-linux... done
Elapsed time for package lua:x64-linux: 30.67 s

Total elapsed time: 30.71 s

Use this package via the module FindLua that comes with CMake. To use in your CMakeLists.txt:

    include(FindLua)
    find_package(lua REQUIRED)
    target_link_libraries(main PRIVATE ${LUA_LIBRARIES})
    target_include_directories(main PRIVATE ${LUA_INCLUDE_DIR})

For more information about the variables set by this module, please see:
    https://cmake.org/cmake/help/latest/module/FindLua.html

[vadi2]

Right, the problem lies with ./vcpkg - so your --overlay-ports=../custom-ports/lua is relative to the vcpkg binary.

But in my case, I do something like:

./vcpkg install @response_file

The reponse_file is not in the same folder as vcpkg. This is important.

That @response_file then mentions a path to overlay ports as --overlay-ports=. for example - meaning "look for overlay ports in the same directory as the response file", and that doesn't work.

[PhoebeHui]

Thanks, I can repro it now.

From https://github.com/microsoft/vcpkg/blob/master/docs/specifications/ports-overlay.md#3-proposed-solution, it seems this is not support now:

a file listing paths to the former two.

NOTE: Reading paths from a text file is not available in the current implementation, some revisions to this part of the specification are being made and will be implemented in a future date.

@vicroms, could you help double confirm?

Repro steps:

1. Create .\vcpkg\custom-ports\sqlite3
2. Create .\vcpkg\ports-repo.txt with 
..\vcpkg\custom-ports\sqlite3
3. ./vcpkg install sqlite3:x64-windows --overlay-ports=..\ports-repo.txt
Your feedback is important to improve Vcpkg! Please take 3 minutes to complete our survey by running: vcpkg contact --survey
Computing installation plan...
Error: Path "F:\vcpkg\ports-repo.txt" must be a directory

[qis]

@PhoebeHui Please try calling vcpkg from anywhere outside of the vcpkg directory. Use ./vcpkg/vcpkg instead of ./vcpkg.

I would like to urge you to reconsider the tag. In my opinion, it's a bug. Either vcpkg is wrong, or the documentation is wrong.

[PhoebeHui]

@qis, thanks for bringing this up.

@ras0219-msft, could you help confirm?

[ras0219-msft]

Yes, this is absolutely a bug -- thank you @qis for the concise description in the original post.

Additionally, I'd like to call out the subtle difference in @vadi2's case of using a response file -- we do not currently plan to interpret relative paths in response files as relative to the response file's directory (instead using the cwd). We would be very interested if there is existing practice around response files changing the meaning of relative paths!