Vcpkg: Suggestion: Automatically add the bin folder that contains the dll files to PATH environment variable

[jasjuang]

Currently I still need to manually add the bin folder (c:\vcpkg\installed\x64-windows\bin) that contains the dll files to PATH environment variable in order to execute my program successfully. In order to make the entire process more automated, is it possible to add the bin folder to PATH environment variable automatically? For example, during the process of the initial bootstrap?

[ras0219-msft]

Ideally, you shouldn't need to add the bin folder to the PATH environment variable -- all required DLLs should be copied into the output folder of your build. This avoids several issues with using the DLLs in-situ (for example, upgrades couldn't take place while the program is executing).

Could you go into more detail about how/why the app-local copying doesn't work for you?

[jasjuang]

@ras0219-msft It worked but that's even more manual friction for development. For app-local copying approach I think upgrades couldn't take place either if the program is executing. Could you elaborate more on the disadvantages a little bit more? What kind of big disadvantage does it have that will outweigh the reduced friction of development?

[ras0219-msft]

even more manual friction for development.

Could you go into more detail about how this causes more friction? It should be performed automatically as part of your builds and have zero workflow impact -- if it impacts your workflow, we should fix that!

For app-local copying approach I think upgrades couldn't take place either if the program is executing.

Since the application has its own copy of all DLLs, the "sources" in Vcpkg can be upgraded while the application is executing. Additionally, after the upgrade of the Vcpkg versions, your application will continue to work because it has the versions of libraries it was originally built against. When you rebuild the application, it should then get the newer versions corresponding to the new headers (though we may have bugs since this isn't as well tested!).

Additionally, relying on path-based DLL lookup is problematic if you have any other applications on your path. For example:

PS D:\src\vcpkg> where.exe qt5core.dll
C:\Program Files\CMake\bin\Qt5Core.dll

While prepending our path would work for a time, if you ever install another application which prepends itself before us, it will suddenly break _everything_ as it pulls randomly versioned DLLs from around the filesystem.

Because of the above, adding the bin folder is much more perilous than the app-local deployment and so we far prefer the second. If you experience friction using it, we should fix that!

[jasjuang]

@ras0219-msft Thanks a lot for the explanation. I thought I had to copy the dlls from vcpkg manually into my local project bin folder. How is it performed automatically? Is there a custom command from vcpkg I can call to copy the dlls into my local project bin folder? Or do I have to figure out some combination of cmake commands to make it happen?

[bagong]

May I join in here, as this is an interesting issue coming up again and again if I understand correctly. The initial misunderstanding of of what "app-local copying" actually means shows nicely that there is some functionality involved here that many of us (including me) seem not to be aware of. I still have no good answer to the following questions, and wonder if vcpkg can give us some additional help (on top of it being a mere library repo, which is already marvelous):

• how should dlls get copied to output-folder/target folders/install locations? I can think of three ways atm: a) manually copying (a nuisance and very error prone), b) making cmake do it in a post build command (inflexible and/or relatively complicated), c) using a tool like cmake bundle-utilities (flexible, context-aware but relatively complicated and actually not failsafe either). It feels like there is another way, that might be the better than any of these alternatives, so self evident for proper VS users that one doesn't feel the need to mention it? ;) Something along the logic of: well, if it's needed in the build, it's needed in the output folder too, so let's copy it there without making much fuss. Does the full fledged use of vcpkg (including the toolchain file and "install integration") contribute to this? I guess many users use vcpkg in "unix" logic, i.e. look at it as a single "sysroot", looked into both at build and runtime. Myself I only address vcpkg via cmake_prefix_path in "outer" builds. If I include the vcpkg toolchain file I actually get problems with my old build system which sofar I haven't had time to analyze.

• how can indirect dependencies be identified (dlls needed by a dll used in a build). This is the strength of cmake bundle-utilities, it can detect nesting library dependencies. But it uses search algorithms independent of the builds search algorithm, so you sometimes end up with mismatched libraries too.

• how about plugins and "optional" dlls, i.e. dlls that are not required, but provide a functionality if available (f.e. the openssh libraries provide ssl functionality to webkit if found/available. If they're not available, webkit won't complain, but it won't open ssl pages either. Second example: gstreamer can process certain multimedia-formats if the respective dll's/plugins are available. If not, it won't complain, but it won't produce sound, video or image either ;))

These are pretty basic questions, likely too generic to be considered specific vcpkg questions. But if vcpkg actually provides ways to solve any of these problems better than old style approaches, it would be lovely to see them described in a well visible space.

[ras0219-msft]

The app-local copying process is performed by scripts\buildsystems\msbuild\applocal.ps1[1]. It uses dumpbin to walk the full closure of all dependencies and copies every DLL from Vcpkg that's used. This means it actually does the right thing if you use a header-only boost library; your executable won't have any symbols into the DLL (since it isn't used), so it won't be listed by dumpbin and won't get deployed.

This script is automatically added as a post-build action by our integration mechanisms. In MSBuild, it's an extra target[2], and in CMake it's done by overriding add_executable[3].

I guess many users use vcpkg in "unix" logic, i.e. look at it as a single "sysroot", looked into both at build and runtime.

We definitely recognize this and try to make things work _almost_ like that as long as you don't look too closely :). However, unfortunately Windows' model for libraries means that you really _can't_ have the *nix sysroot implementation because you can't upgrade libraries while they are in use.

As an aside, this is one of the reasons why Windows needs to reboot when you install certain things: they want to upgrade a DLL that's used by the system or some other program, so they need you to restart to ensure all programs release their locks on the file.

how about plugins and "optional" dlls, i.e. dlls that are not required, but provide a functionality if available

This is definitely a difficulty of the current approach, because we can't analyze "LoadLibrary". In the case of Qt, I've reimplemented the windeployqt.exe program as a powershell script to handle that deployment logic[4]. I'd like to improve this in the future to somehow enable libraries to more generically express plugin dependencies -- from the examples I've seen, I think a simple mapping of DLL -> patterns should work.

If I include the vcpkg toolchain file I actually get problems with my old build system which sofar I haven't had time to analyze.

If you can get us a repro, I'd love to fix that! I'm sure someone else will also have your problem and we want everyone to be able to use the toolchain file :)

[1] https://github.com/Microsoft/vcpkg/blob/cedaaa195637296783f67a3f7cc161797ee7d562/scripts/buildsystems/msbuild/applocal.ps1
[2] https://github.com/Microsoft/vcpkg/blob/cedaaa195637296783f67a3f7cc161797ee7d562/scripts/buildsystems/msbuild/vcpkg.targets#L62-L81
[3] https://github.com/Microsoft/vcpkg/blob/cedaaa195637296783f67a3f7cc161797ee7d562/scripts/buildsystems/vcpkg.cmake#L112-L128
[4] https://github.com/Microsoft/vcpkg/blob/cedaaa195637296783f67a3f7cc161797ee7d562/ports/qt5/qtdeploy.ps1

[jasjuang]

@ras0219-msft Thanks for the detailed explanation. I looked into the bin folder of my project again and realized that some of the dlls from vcpkg did get copied over automatically but some of them did not. For example, boost dlls did not get copied over. Does that mean there is some bug in the portfile of boost? I can track down the rest if that is the case.

[ras0219-msft]

For example, boost dlls did not get copied over.

This is actually the desired behavior, if you aren't actually using those DLLs! Large parts of boost are actually header-only and don't require the runtimes. By analyzing your binary via dumpbin, we should be able to discover the "true" runtime requirements and make your application slimmer 😄!

However, if your application fails to run because it's missing pieces of boost, then that's probably a bug in our "applocal.ps1" script. Here are some things to check:

First, try running dumpbin on your executable to determine its true dependencies.

PS D:\src\cmake-test\build2\Debug> dumpbin /dependents .\main.exe /nologo

Dump of file .\main.exe

File Type: EXECUTABLE IMAGE

  Image has the following dependencies:

    boost_filesystem-vc141-mt-gd-1_64.dll
    boost_system-vc141-mt-gd-1_64.dll
    MSVCP140D.dll
    VCRUNTIME140D.dll
    ucrtbased.dll
    KERNEL32.dll

  Summary

        1000 .00cfg
        1000 .data
        2000 .idata
        3000 .rdata
        1000 .reloc
        1000 .rsrc
        A000 .text

Second, you can run the applocal script yourself in verbose mode:

PS D:\src\cmake-test\build2\Debug> D:\src\vcpkg\scripts\buildsystems\msbuild\applocal.ps1 .\main.exe -installedDir D:\src\vcpkg\installed\x86-windows\debug\bin -verbose
VERBOSE: Resolving .\main.exe...
VERBOSE:   boost_filesystem-vc141-mt-gd-1_64.dll: Copying D:\src\vcpkg\installed\x86-windows\debug\bin\boost_filesystem-vc141-mt-gd-1_64.dll
VERBOSE: Resolving D:\src\cmake-test\build2\Debug\boost_filesystem-vc141-mt-gd-1_64.dll...
VERBOSE:   boost_system-vc141-mt-gd-1_64.dll: Copying D:\src\vcpkg\installed\x86-windows\debug\bin\boost_system-vc141-mt-gd-1_64.dll
VERBOSE: Resolving D:\src\cmake-test\build2\Debug\boost_system-vc141-mt-gd-1_64.dll...
VERBOSE:   MSVCP140D.dll: D:\src\vcpkg\installed\x86-windows\debug\bin\MSVCP140D.dll not found
VERBOSE:   KERNEL32.dll: D:\src\vcpkg\installed\x86-windows\debug\bin\KERNEL32.dll not found
VERBOSE:   VCRUNTIME140D.dll: D:\src\vcpkg\installed\x86-windows\debug\bin\VCRUNTIME140D.dll not found
VERBOSE:   ucrtbased.dll: D:\src\vcpkg\installed\x86-windows\debug\bin\ucrtbased.dll not found
VERBOSE: Done Resolving D:\src\cmake-test\build2\Debug\boost_system-vc141-mt-gd-1_64.dll.
VERBOSE:   MSVCP140D.dll: previously searched - Skip
VERBOSE:   ADVAPI32.dll: D:\src\vcpkg\installed\x86-windows\debug\bin\ADVAPI32.dll not found
VERBOSE:   KERNEL32.dll: previously searched - Skip
VERBOSE:   VCRUNTIME140D.dll: previously searched - Skip
VERBOSE:   ucrtbased.dll: previously searched - Skip
VERBOSE: Done Resolving D:\src\cmake-test\build2\Debug\boost_filesystem-vc141-mt-gd-1_64.dll.
VERBOSE:   boost_system-vc141-mt-gd-1_64.dll: previously searched - Skip
VERBOSE:   MSVCP140D.dll: previously searched - Skip
VERBOSE:   VCRUNTIME140D.dll: previously searched - Skip
VERBOSE:   ucrtbased.dll: previously searched - Skip
VERBOSE:   KERNEL32.dll: previously searched - Skip
VERBOSE: Done Resolving .\main.exe.

The above commands need to be run from a developer prompt (because they use dumpbin). One way to get there is to run vcpkg env, which will enter a cmd sub-shell identical to the internal build environment.

[jasjuang]

@ras0219-msft I use dumpbin to check my application and it shows something like

``````
Dump of file mytest.exe

File Type: EXECUTABLE IMAGE

Image has the following dependencies:

my_lib.dll
opencv_core330.dll
glog.dll
MSVCP140.dll
VCRUNTIME140.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
KERNEL32.dll

Summary

    3000 .data
    1000 .gfids
    3000 .pdata
   15000 .rdata
    1000 .reloc
    1000 .rsrc
   2C000 .text
    1000 .tls

``````

But my project is clearly using boost functions and when I attempt to execute it, it fails because it wasn't able to find the boost dlls. Does that mean there are bugs in dumpbin?

EDIT: I am using find_package( Boost COMPONENTS system filesystem REQUIRED ) in my project and target_include_directories(${PROJECT_NAME} PUBLIC ${Boost_INCLUDE_DIR}) and
target_link_libraries(${PROJECT_NAME} ${Boost_LIBRARIES})

[ras0219-msft]

Hmm, while technically possible I highly doubt dumpbin is incorrect here. Are the calls to boost inside my_lib.dll? Dumpbin will only show the direct dependents.

Otherwise, is it possible to reduce this down to a dozen-line cpp file?

[jasjuang]

@ras0219-msft Yes the calls to boost is inside my_lib.dll. You are also right that it seems like dumpbin only show direct dependencies.

But, my_lib also has dependencies to gflags and tinyxml2 as well. gflags and tinyxml2 are also not shown in dumpbin but the dlls actually got copied over to my project bin folder by vcpkg. Is vcpkg purely relying on the info from dumpbin to decide which dlls to copy over?

[atkawa7]

@ras0219-msft @jasjuang @bagong Could you guys help me understand app local behavior. Are they any documentations online where I could learn this ? I remember @ras0219-msft was talking about using it in Linux in #57

[ras0219-msft]

@atkawa7 It's somewhat buried, but the Dynamic-Link Library Search Order[1] article on MSDN describes the precise method the loader uses to search for DLLs. One of the locations it looks in is the folder containing the executable, which means you can deploy all your dependencies within that folder ("app-locally") and they'll be found at load time.

@jasjuang Vcpkg looks _recursively_ at dumpbin's results. We do a DFS through the dependency tree and copy everything for every dependent. For DLLs that aren't ours (like my_lib.dll), we recurse into it if it's present and ignore it if it's not. It will probably be clearer if you run the applocal.ps1 script in verbose mode, which prints out each step taken.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx

[atkawa7]

@ras0219-msft Thanks for the lead. So if we were to port vcpkg to linux how would you mimic the same behavior

[ras0219-msft]

One way is to output a wrapper executor script that runs the program like:

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )".
LD_LIBRARY_PATH=$DIR:$LD_LIBRARY_PATH ./executable

This should add the script directory to the loader search path.

Alternatively, you might be able to use rpath [1] to bake the executable directory into the application binary. Specifically, in the recommendations section:

$ORIGIN: On Linux/Solaris, it's probably a very good idea to specify any RPATH setting one requires to look up the location of a package's private libraries via a relative expression, to not lose the capability to provide a fully relocatable package. This is what $ORIGIN is for. In CMAKE_INSTALL_RPATH lines, it should have its dollar sign escaped with a backslash to have it end up with proper syntax in the final executable. See also the CMake and $ORIGIN discussion. For Mac OS X, there is a similar @ rpath, @loader_path and @executable_path mechanism. While dependent libraries use @ rpath in their install name, relocatable executables should use @loader_path and @executable_path in their RPATH. For example, you can set CMAKE_INSTALL_RPATH to @loader_path, and if an executable depends on "@rpath/libbar.dylib", the loader will then search for "@loader_path/libbar.dylib", where @ rpath was effectively substituted with @loader_path.

[1] https://cmake.org/Wiki/CMake_RPATH_handling

[atkawa7]

Thanks that certainly helps @ras0219-msft I have been thinking about helping port vcpkg to Linux. However there are currently many dependencies on Windows. The one thing that is frustrating is the wstring/string issue. Since most of the functions on Windows work with wchars which is not the same with the Linux counter parts. What are yout thoughts on defaulting to wstring on windows then string on Linux. Something like

#ifdef _WIN32
#define CustomString std::wstring
#define Char wchar_t
#elif __linux__
#define CustomString std::string
#define Char char
#endif

[jasjuang]

@ras0219-msft Here is the result after I run powershell C:\dev\vcpkg\scripts\buildsystems\msbuild\applocal.ps1 mytest.exe -installedDir C:\dev\vcpkg\installed\x64-windows\debug\bin -verbose

``````
VERBOSE: Resolving mytest.exe...
VERBOSE: my_lib.dll: C:\devvcpkg\installed\x64-windows\debug\binmy_lib.dll not found
VERBOSE: opencv_core330.dll: C:\devvcpkg\installed\x64-windows\debug\bin\opencv_core330.dll not found
VERBOSE: glog.dll: already present
VERBOSE: Resolving C:\Users\jasju\Desktop\testrepo\build\bin\Release\glog.dll...
VERBOSE: gflags.dll: already present
VERBOSE: Resolving C:\Users\jasju\Desktop\testrepo\build\bin\Release\gflags.dll...
VERBOSE: SHLWAPI.dll: C:\devvcpkg\installed\x64-windows\debug\bin\SHLWAPI.dll not found
VERBOSE: KERNEL32.dll: C:\devvcpkg\installed\x64-windows\debug\bin\KERNEL32.dll not found
VERBOSE: MSVCP140.dll: C:\devvcpkg\installed\x64-windows\debug\bin\MSVCP140.dll not found
VERBOSE: VCRUNTIME140.dll: C:\devvcpkg\installed\x64-windows\debug\bin\VCRUNTIME140.dll not found
VERBOSE: api-ms-win-crt-runtime-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-runtime-l1-1-0.dll not found
VERBOSE: api-ms-win-crt-string-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-string-l1-1-0.dll not found
VERBOSE: api-ms-win-crt-heap-l1-1-0.dll: C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-heap-l1-1-0.dll
not found
VERBOSE: api-ms-win-crt-convert-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-convert-l1-1-0.dll not found
VERBOSE: api-ms-win-crt-environment-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-environment-l1-1-0.dll not found
VERBOSE: api-ms-win-crt-stdio-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-stdio-l1-1-0.dll not found
VERBOSE: Done Resolving C:\Users\jasju\Desktop\testrepo\build\bin\Release\gflags.dll.
VERBOSE: KERNEL32.dll: previously searched - Skip
VERBOSE: MSVCP140.dll: previously searched - Skip
VERBOSE: VCRUNTIME140.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-runtime-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-string-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-convert-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-environment-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-stdio-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-math-l1-1-0.dll: C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-math-l1-1-0.dll
not found
VERBOSE: api-ms-win-crt-filesystem-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-filesystem-l1-1-0.dll not found
VERBOSE: api-ms-win-crt-time-l1-1-0.dll: C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-time-l1-1-0.dll
not found
VERBOSE: api-ms-win-crt-heap-l1-1-0.dll: previously searched - Skip
VERBOSE: Done Resolving C:\Users\jasju\Desktop\testrepo\build\bin\Release\glog.dll.
VERBOSE: MSVCP140.dll: previously searched - Skip
VERBOSE: VCRUNTIME140.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-runtime-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-heap-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-convert-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-environment-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-stdio-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-string-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-filesystem-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-time-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-math-l1-1-0.dll: previously searched - Skip
VERBOSE: api-ms-win-crt-locale-l1-1-0.dll:
C:\devvcpkg\installed\x64-windows\debug\bin\api-ms-win-crt-locale-l1-1-0.dll not found
VERBOSE: KERNEL32.dll: previously searched - Skip
VERBOSE: Done Resolving mytest.exe.
VERBOSE:
Name Value

---- -----

api-ms-win-crt-locale-l1-1-... True

api-ms-win-crt-heap-l1-1-0.dll True

api-ms-win-crt-convert-l1-1... True

my_lib.dll True

api-ms-win-crt-stdio-l1-1-0... True

SHLWAPI.dll True

api-ms-win-crt-filesystem-l... True

MSVCP140.dll True

api-ms-win-crt-runtime-l1-1... True

KERNEL32.dll True

VCRUNTIME140.dll True

api-ms-win-crt-string-l1-1-... True

api-ms-win-crt-math-l1-1-0.dll True

opencv_core330.dll True

gflags.dll True

api-ms-win-crt-time-l1-1-0.dll True

api-ms-win-crt-environment-... True

glog.dll True
``````

For some reason it is expecting my_lib.dll to be in the vcpkg bin folder but it is actually not. I think that's the reason why it wasn't able to find the indirect dependencies

[ras0219-msft]

@atkawa7 To avoid straying too far in this thread, I've opened #1449 to track any questions about making the codebase more cross-platform.

@jasjuang Thanks a bunch! I think you're exactly right -- we appear to have a bug where we aren't properly recursing through non-vcpkg dependencies. I'll try to reduce that down and get a fix, this is a big help!

edit: Could you confirm that my_lib.dll is in the same output folder as mytest.exe? I think the bug may be caused by different output folders.

[atkawa7]

@ras0219-msft Thanks

[jasjuang]

@ras0219-msft Thanks, look forward to the fix!

[bagong]

This was an extremely instructive thread for me, a model for the merits of open source development :smile:. Thank you all for your great attitude and competence!

[ras0219-msft]

@jasjuang Alright, pushed the fix to master. Could you confirm that this resolves the issue?

Thanks again for reporting it and engaging in the back-and-forth 👍!

[jasjuang]

@ras0219-msft Works like magic, this is such an amazing change!