Vcpkg: Running more than one instance of vcpkg is racy

[janisozaur]

Running more than one instance of vcpkg concurrently causes race conditions on file system and can make the tool fail.

Observed races can occur (at least) in following cases:

• Updating package state:
  https://github.com/Microsoft/vcpkg/blob/6e653d88f685e30fb9c4b82a440a0cc8fb826f71/toolsrc/src/vcpkg/vcpkglib.cpp#L38-L86
• Downloading dependencies:
  https://github.com/Microsoft/vcpkg/blob/6e653d88f685e30fb9c4b82a440a0cc8fb826f71/scripts/fetchDependency.ps1#L14-L113

Without more granular verbs than install, it is impossible to prevent such races from external tools

[ras0219-msft]

Hi @janisozaur, thanks for posting this issue!

At the moment, it's not a goal for vcpkg to function concurrently with itself (however it _is_ a goal to avoid corruption upon process termination). Could you go into more detail about what you mean by "external tools"?

[janisozaur]

Sure!

Let me state the problem I'm facing first:

TL;DR: AppVeyor offers 2 cores, but vcpkg uses only one and runs out of time. Using two cores produces full set of libraries well under limits.

Long version: In OpenRCT2 for our Windows dependencies we've moved fully to vcpkg-provided prebuilts (thanks, it's proven really helpful in managing the libraries). We would like to have AppVeyor build the full set of our required libraries, but when naively doing vcpkg install zlib; vcpkg install libzip; vcpkg install openssl; …, the build hits the hard limit of 60 minutes available on the free account with AppVeyor. Upon investigation, it turns out most of the time is wasted spent on configuring cmake-based projects (see https://developercommunity.visualstudio.com/content/problem/162361/cmake-configuration-is-unbearably-slow.html and https://github.com/Microsoft/vcpkg/issues/625) and I am looking into various ways of speeding it up (e.g. https://github.com/Microsoft/vcpkg/pull/2332). Unfortunately, ninja only speeds up the build so much and is available for non-UWP targets. AppVeyor offers 2 CPUs to each job and when vcpkg is busy configuring a library, it is only using 1 core and the other simply idles. I thought building libraries in parallel would speed thing up, and indeed, I'm able to reduce the build times from over 60 minutes to around 30.

My initial test of changing:

vcpkg install zlib
vcpkg install libzip # depends on zlib, takes long time to build
vcpkg install openssl

to:

vcpkg install zlib
start "" vcpkg install libzip # this now spawns separate process
vcpkg install openssl

proved successful, so I looked into more ways how can I parallelise the builds.

I ended up with a build.ninja that I can execute and get a full set of libraries, but occasionally run into the issue described here.

My build.ninja file, crafted manually after inspecting full set of our dependencies:

triplet = x86-windows

pool cmake_download_deps_pool
  depth = 1

pool vcpkg_download_deps_pool
  depth = 1

rule cmake_dep
  command = cmake -DDEP=$out -P downloader.cmake
  description = Downloading cmake dependency $out
  pool = cmake_download_deps_pool

rule vcpkg_dep
  command = powershell scripts/fetchDependency.ps1 $out
  description = Downloading vcpkg dependency $out
  pool = vcpkg_download_deps_pool

rule vcpkg
  command = vcpkg install $out:$triplet
  description = Building $out

# fetch cmake dependencies
build JOM: cmake_dep
build PERL: cmake_dep
build NASM: cmake_dep

# fetch vcpkg dependencies
build cmake: vcpkg_dep
build git: vcpkg_dep
build nuget: vcpkg_dep

# libraries
build bzip2: vcpkg | cmake git
build openssl: vcpkg | cmake git JOM PERL NASM
build speexdsp: vcpkg | cmake git
build sdl2: vcpkg | cmake git
build zlib: vcpkg | cmake git
build jansson: vcpkg | cmake git
build discord-rpc: vcpkg | cmake git
build libzip: vcpkg | zlib cmake git
build libpng: vcpkg | zlib cmake git
build libssh2: vcpkg | openssl zlib cmake git
build curl: vcpkg | openssl zlib libssh2 cmake git PERL
build freetype: vcpkg | zlib bzip2 libpng cmake git

build all: phony | zlib freetype jansson libpng sdl2 speexdsp discord-rpc curl libzip openssl nuget

default all

Where downloader.cmake is just a simplified version of your ports.cmake

While doing this, I thought vcpkg could use another command to act as a ninja generator and provide such graph for given set of libraries. Perhaps this could be integrated into vcpkg itself, parallel building (behind a switch). Let me know if you think filing requests for these features would be worthwhile.

[ras0219-msft]

Parallel building in general would be a more complex feature, but I think there's a very low hanging fruit here: we could run the cmake configure for release and debug in parallel. This should (approximately) saturate the two CPUs.

What do you think?

[janisozaur]

That sounds like an improvement, as it is where the most time is spent, but I have another suggestion as well: building packages (configuration + build steps) could be done in parallel by vcpkg itself and then only the installation steps is what needs to be serialised.

The two approaches are not exclusive, but should probably be hidden behind options to explicitly enable them.

I'd like to take a look at this myself, if you don't mind.

[ghost]

I don't think there's really a need for parallel builds: most build systems parallelize the build pretty well. Configuring does take a significant chunk of underutilized time though; perhaps even beyond what's proposed, vcpkg could parallelize configuring packages that don't depend on each other.

For example, if A depends on B and C, then vcpkg could download B as it does now, and as soon as vcpkg_configure_cmake is called, vcpkg could start downloading C. Basically, this would mean that while the configuring stage is running, vcpkg can work on other things.

[janisozaur]

@xoviat the problem with your suggestion is that vcpkg_configure_cmake is in CMake code and without a more precise verbs (or hacks that would somehow notify vcpkg at which stage package plan is) there is nothing vcpkg can do. Usually (at least in cases I've seen) the building part is significantly shorter and significantly more parallel than the configuration part, so it actually makes a lot of sense to parallelise whole package plans.

In case of a package that relies on many calls to CHECK_FUNCTION_EXISTS and the like, the configuration takes multiple minutes, while building (compilation+linking) only a few seconds. This is the case I want to solve here.

[ghost]

Yes, but once the portfile calls that function, vcpkg is fully in control. As for implementation, the tool can put a lockfile somewhere and then configure and remove the file at the beginning of the function and put it back at the end. There are details that need to be worked out but it's doable.

[janisozaur]

the tool can put a lockfile somewhere

hacks that would somehow notify vcpkg at which stage package plan is

How does calling vcpkg_configure_cmake put vcpkg back in control? I'm not familiar with this code yet, so I'm likely missing something or maybe I misunderstand you – do you mean vcpkg, the compiled binary?

[ghost]

Specifically what I mean is that we can't manage race conditions over the entire portfile surface, but we can manage race conditions in the ~100 lines of that function. Specifically, we can queue up configure calls and then run them all at once in parallel.

[janisozaur]

If vcpkg is to place lockfiles, then the original issue about serialising installation step can be solved this way and it would be much simpler than having to handle all the cases of configure

[tobiaskohlbau]

I've hacked together a very ugly powershell script which is called from within vcpkg_configure_cmake and runs debug and release in parallel. I've tried to install some packages and put together a small table with the timing reported by vcpkg itself.

| | | | | |
|-------------|-------------|-------------|-------------|-------------|
| | OLD | NEW | OLD | NEW |
| | x64 | x64 | x86 | x86 |
| curl | 5.584 | 4.633 | 5.313 | 3.682 |
| discord-rpc | 0.794833333 | 0.705 | 0.712333333 | 0.74 |
| freetype | 0.5295 | 0.610333333 | 0.577666667 | 0.4485 |
| jansson | 2.052 | 1.44 | 1.894 | 1.25 |
| libpng | 0.3325 | 0.384 | 0.332166667 | 0.303833333 |
| libzip | 3.361 | 2.4 | 2.956 | 4.075 |
| openssl | 5.155 | 6.479 | 10.52 | 4.922 |
| sdl2 | 1.671 | 1.672 | 1.471 | 1.669 |
| speexdsp | 0.623666667 | 0.531833333 | 0.618 | 0.790333333 |
| zlib | 0.411 | 0.381666667 | 0.526666667 | 0.34 |
| libssh2 | 2.237 | 2.635 | 5.313 | 1.865 |
| bzip2 | 0.295166667 | 0.286 | 0.2445 | 0.257333333 |
| | | | | |
| SUM | 23.04666667 | 22.15783333 | 30.47833333 | 20.343 |

Timings are in minutes. I wouldn't assume my implementation is very expressive or right but maybe it leaves a hint if a official implementation is worth to work on.

I've used an external PS script as cmake is unable to run processes parallel. Note it's possible with execute_process but stdin from first command is piped into second command which isn't the desired behavior within vcpkg. I'm happy to hear other ideas how to implement parallel config of Debug/Release.

I'm open for comments on the attached scripts as I'm a newbie related to powershell.

Ugly Hack

set(BUILD_OPTIONS_RELEASE
        ${_csc_SOURCE_PATH} ${_csc_OPTIONS} ${_csc_OPTIONS_RELEASE}
        -G \"${GENERATOR}\"
        -DCMAKE_BUILD_TYPE=Release
        -DCMAKE_INSTALL_PREFIX=${CURRENT_PACKAGES_DIR}
    )
    set(BUILD_OPTIONS_DEBUG
        ${_csc_SOURCE_PATH} ${_csc_OPTIONS} ${_csc_OPTIONS_DEBUG}
        -G \"${GENERATOR}\"
        -DCMAKE_BUILD_TYPE=Debug
        -DCMAKE_INSTALL_PREFIX=${CURRENT_PACKAGES_DIR}/debug
    )
    vcpkg_execute_required_process(
        COMMAND powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "& { & '${VCPKG_ROOT_DIR}/scripts/executeProcess.ps1' -Verbose -executable '${CMAKE_COMMAND}' -relDirectory ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel -dbgDirectory ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg -relArguments '${BUILD_OPTIONS_RELEASE}' -dbgArguments '${BUILD_OPTIONS_DEBUG}' }" 2>&1
        WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
        LOGNAME config-${TARGET_TRIPLET}-rel)

[CmdletBinding()]
param(
    [Parameter(Mandatory=$true)][string]$executable,
                                [string]$relDirectory, 
                                [string]$dbgDirectory, 
                                [string]$relArguments = "--help",
                                [string]$dbgArguments = "--help"
)

function invoke([string]$executable, [string]$directory, [string]$arguments)
{
    Write-Verbose $arguments

    $arguments = $arguments -replace "  ", " "
    $arguments = $arguments -replace "= (?=[^-])", "="
    $arguments = $arguments -replace "((?:(?:\/\w+(?:[:-]\w+)* ))+)", "`"`$1`" "

    Write-Verbose $executable
    Write-Verbose $directory
    Write-Verbose $arguments

    $process = Start-Process -FilePath $executable -ArgumentList $arguments -NoNewWindow -PassThru -WorkingDirectory $directory

    return $process
}

$relProcess = invoke $executable $relDirectory $relArguments
$dbgProcess = invoke $executable $dbgDirectory $dbgArguments

Wait-Process -InputObject $relProcess, $dbgProcess

[ras0219-msft]

That's awesome! I'm probably a bit too excited over that openssl speedup in x86 😋

My other idea was to leverage Ninja here -- we could write out a very simple .ninja that would do the parallel tasks. The problem is that Ninja is only currently available for x64 machines as a precompiled binary. This affects our use of Ninja elsewhere in vcpkg too -- we can't ever _depend_ on it being available. As a larger task, we should really look into building Ninja locally ourselves, which will let us take a much stronger dependency on it (use it _always_ instead of just prefer it).

I think writing out a temporary file is probably a good idea regardless here, both for debugging and dealing with the escaping nightmare that this will bring along.

Also, I have a sneaking suspicion that just launching powershell is actually adding a huge amount of overhead for the smaller libraries in your table. When running powershell directly from powershell they do clever optimizations to make you not realize the true startup cost, but you can see it if you compare cmd /c cmd versus cmd /c powershell.

I'll see if I can take your first approach and rewrite it to avoid powershell, then see if it bumps the speeds up even further!

[MikeGitb]

The problem is that Ninja is only currently available for x64 machines as a precompiled binary.

A bit OT, but are there really users of vcpkg that run a 32 Bit version of Windows?

[ras0219-msft]

Yep! We've gotten several bug reports when we've broken it the past.

[tobiaskohlbau]

I think it's a good idea to leverage ninja instead of powershell., didn't thought about that idea.

If you're willing to open a branch early within development I'm happy to test and contribute to.

[cmpute]

So maybe this issue is already solved?

[janisozaur]

The problem reported originally is probably not sorted, but there were other solutions put in place to better utilise available resources. I have not retested it since reporting, but as the end goal of being able to compile packages on AppVeyor within the time limit, @ras0219-msft can feel free to close.

[NancyLi1013]

@janisozaur

Thanks for your update. I'm closing this issue for now. If you have other problems, please open a new issue. Thanks.