Vcpkg: Ability to have "slim" installs with release only libraries

[adam4813]

As the title suggests the ability to install only release libraries instead of debug and release we be very useful for CI caching and systems that have limited storage.

[ras0219-msft]

The reasoning for always deploying debug builds is that they are on the same order of size as the release builds, so by dropping them you decrease the total deployment by somewhere around half for non-header-only-libraries. I'm skeptical that halving the space used here is worthwhile, because if that made a difference then you'd run into the limit again once you double the number of libraries depended upon.

Do you have a concrete case where this would be really invaluable (with some target disk footprints)?

Another avenue that could decrease space usage by far more on CI servers is to clean out the buildtrees\ directory. This needs to be kept around on a developer machine to have source-level debugging, but for CI servers it can be removed after the build is complete.

Some numbers from expat (randomly selected package):

• Entire layout (packages\expat_x86-windows): 430 KB
• Debug directory of layout (packages\expat_x86-windows\debug): 258 KB
• Intermediate objects (buildtrees\expat\x86-windows-*): 3.04 MB
• Sources (buildtrees\expat\src): 2.12 MB

Similar numbers for opencv:

• Entire layout (packages\opencv_x86-windows): 227 MB
• Debug directory of layout (packages\opencv_x86-windows): 120 MB
• Intermediate objects (buildtrees\opencv\x86-windows-*): 1.32 GB
• Sources (buildtrees\opencv\src): 276 MB

A feature we have not implemented yet but intend to look into is hardlinking files between the packages\<name>\ directory and installed\. This will hopefully halve the disk space used for libraries with large binaries (header-only libraries have such small files that they will be mostly unaffected).

Apologies for the wall of text, we've thought about this issue before so we had some evidence at hand :). Please feel free to continue discussing, we may have overlooked something.

[adam4813]

Check protobuf for what I am getting at. The libraries alone are 166MB for the debug (all 3 combined) but only 23MB for the release (combined again.

This only becomes truly useful for caching the install libraries on a CI platform that limits cache size such as appveyor (200MB limit).

[ras0219-msft]

I see, that could be a significant amount of savings if the difference here can't be reduced.

According to https://www.appveyor.com/docs/build-cache/, I'm seeing a 100MB or 1GB limit for free plans and that the archives are compressed. Testing with 7z/zip/fastest (the settings listed in the link), I get the following results after compression is applied (note that I fixed protobuf to use DLLs here, so YMMV a bit):

• Protobuf_x86-windows full layout (zipped): 22 MB
• Protobuf_x86-windows "slim" (zipped): 11 MB

So it looks like the debug libraries (while bloated on disk) are very compressible and should have a total impact of ~2x in the specific case of AppVeyor.

[adam4813]

Yes but adding up over 20 libraries it could become a make or break.

This sort of feature doesn't seem like it would break anything and could perhaps be part of a triplet setting?

[barcharcraz]

it seems to me that doing both release and debug builds is going to increase the build time by ~1.5-2x (debug builds may be faster than release). I don't really mind this too much when I'm installing a package for my own use but when writing package files it gets super annoying. It would be really nice to be able to test only the debug mode while iterating on a package.

[ras0219-msft]

Hmm, we could do this through a triplet setting without impacting the reproducibility of the system. Would you want this to also eliminate PDBs?

[adam4813]

pdb tend to be optional for release builds, so yes I would think that is
fine.

[SergioRAgostinho]

Sorry to revive this one. Was there any updates on this?

I'm setting up a CI in appveyor and I was interested in limiting the installation process of dependencies to release versions only, in order to cut down on the build times.

[ras0219-msft]

No problem, thanks for chiming in!

I've definitely seen more people requesting this, so I've checked in an initial set of changes towards this goal. If you add

set(VCPKG_BUILD_TYPE release)

to a triplet file, most ports will avoid building the debug version. This is certainly going to have holes in it, but I'd be happy to accept PRs fixing those when they do arise.

[SergioRAgostinho]

It worked pretty good, thanks for the pro tip :+1:

On a side note, I wonder how scalable this triplet file approach is. Under a very simplistic assumption of 2^n combinations with n being the number of parameters, things can go quickly into a lot of files. So far there's target platform, desktop/windows store, static/dynamic linkage, with the addition of release/debug at some point. Well... you can see where this is going :sweat_smile: .

Thanks for the tip ;)

[ras0219-msft]

No problem :)

The triplet system is directly designed to combat that problem, actually. There's such an incredibly large space of possibilities, but you actually only ever want a small number (perhaps different per individual though). That's why we record those settings in a file and give them a nice name so they don't get lost 😄!

We obviously won't be adding 2^N built-in ones though; instead we keep those limited to a reasonable common set that works for 90%.

[adam4813]

I updated the appveyor documentation to show how to easily cache vcpkg
installs.

[ras0219-msft]

Implemented as part of #2120.

[SergioRAgostinho]

Edit: Ignore this guys. I made a dumb mistake and was passing "Release" instead of "release"

Sorry for resurrecting this but for a while now, setting

set(VCPKG_BUILD_TYPE release)

stopped working for me in zlib. It constantly aborted on the POST_BUILD_CHECKS. More info on the error output here

The snippet bellow exemplifies how I was setting the option. This worked ok before.

# cache:
  # - c:\tools\vcpkg\installed\

configuration: Release

environment:
  VCPKG_DIR: C:\Tools\vcpkg
  matrix:
    - generator: "Visual Studio 14 2015"
      APPVEYOR_BUILD_WORKER_IMAGE: "Visual Studio 2015"
      Platform: x86
    - generator: "Visual Studio 14 2015 Win64"
      APPVEYOR_BUILD_WORKER_IMAGE: "Visual Studio 2015"
      Platform: x64
    # #Uncoment this block once MSVC2017 is patched.
    # - generator: "Visual Studio 15 2017"
    #   APPVEYOR_BUILD_WORKER_IMAGE: "Visual Studio 2017"
    #   Platform: x86
    # - generator: "Visual Studio 15 2017 Win64"
    #   APPVEYOR_BUILD_WORKER_IMAGE: "Visual Studio 2017"
    #   Platform: x64

# Runs before cache setup
init:
  - cd %VCPKG_DIR%
  - git pull
  # - git checkout 94bd9dd66e9db88f965c8b270ea58f927685a317
  - echo.set(VCPKG_BUILD_TYPE %CONFIGURATION%)>> %VCPKG_DIR%\triplets\%PLATFORM%-windows.cmake
  - .\bootstrap-vcpkg.bat
  - vcpkg version
  - cd %APPVEYOR_BUILD_FOLDER%

install:
  - vcpkg install boost:%PLATFORM%-windows
                  eigen3:%PLATFORM%-windows
                  flann:%PLATFORM%-windows
                  qhull:%PLATFORM%-windows


build:
  parallel: true

build_script:
  - cd %APPVEYOR_BUILD_FOLDER%
  - mkdir build
  - cd build
  - cmake -G"%GENERATOR%"
          -DCMAKE_TOOLCHAIN_FILE=c:/tools/vcpkg/scripts/buildsystems/vcpkg.cmake
          -DPCL_BUILD_WITH_BOOST_DYNAMIC_LINKING_WIN32=ON
          -DPCL_BUILD_WITH_FLANN_DYNAMIC_LINKING_WIN32=ON
          -DPCL_NO_PRECOMPILE=ON
          -DBUILD_simulation=ON
          -DBUILD_global_tests=OFF
          -DBUILD_examples=OFF
          -DBUILD_tools=OFF
          -DBUILD_apps=OFF
          ..
  - cmake --build . --config %CONFIGURATION%

[Raffaello]

even better would be nice to have a CLI option flag like --debug-only or --release-only or something similar to use with vcpkg install [...] --triplet [...] --debug-only as eg.

[khrykin]

I currently call this on my CI machine before first vcpkg install (appends set(VCPKG_BUILD_TYPE release) to every triplet file):

 find ./vcpkg/triplets/* -type f -exec sh -c "echo \"\nset(VCPKG_BUILD_TYPE release)\n\" >> {}" \;

Would be nice to have a CLI flag though.