Vcpkg: Impossible to install successfully new ports after a custom port has been installed

[traversaro]

Describe the bug
With the current master branch (commit https://github.com/microsoft/vcpkg/commit/1be75a28bfce2084bb0b59ec92a6853f13d7be66) the use of custom ports at some point prevents to install any further non-custom port from vcpkg.

Environment

• OS: Windows
• Compiler: VS2019

To Reproduce
~
git clone https://github.com/microsoft/vcpkg C:/vcpkg
git clone https://github.com/robotology/robotology-vcpkg-binary-ports C:/robotology-vcpkg-binary-ports
C:/vcpkg/bootstrap-vcpkg.sh
C:/vcpkg/vcpkg.exe --overlay-ports=C:/robotology-vcpkg-binary-ports install --triplet x64-windows ipopt-binary
C:/vcpkg/vcpkg.exe install --triplet x64-windows asio
~
The last command will fail with the following error:
~
Computing installation plan...
Error: while loading ipopt-binary:x64-windows: Port definition not found.
Please run "vcpkg remove ipopt-binary:x64-windows" and re-attempt.
~

Expected behavior
The asio port should be installed correctly, even if the custom port ipopt-binary had been installed before.

Failure logs
See https://github.com/robotology/robotology-vcpkg-binary-ports/pull/5 for a CI script that is working fine in 2020.01 and 2019.12, and failing in master due to this bug.

[traversaro]

cc @pattacini

[traversaro]

At a first glance, this seems to be related to https://github.com/microsoft/vcpkg/pull/9895, fyi @ras0219-msft

[vicroms]

Looks like after the changes in #9895 you have to specify the --overlay-ports location on each install command 😟

[NancyLi1013]

Hi @traversaro
Can this work for you after add --overlay-ports?

[traversaro]

Hi @traversaro
Can this work for you after add --overlay-ports?

Hi @NancyLi1013, what you mean by "this"? Specifying --overlay-ports for every install command, as @vicroms suggested?
To be honest it seems quite a big drawback the need to explicitly mention all the --overlay-ports that you have used in the past for every execution, especially if you have a lot of them. However I don't pay for vcpkg, so I am not in a position to complain. : )

Jokes apart, probably this problem could mitigated if there was a way to specify overlay-ports not through the command line but with a way that can be done only once, for example with a config file added in the vcpkg directory, or with an enviroment variable.

[traversaro]

To provide an additional data point, I just come across another repo containing instructions are broken by this change: https://github.com/qis/toolchains#compiler .

[traversaro]

Hi @vicroms @NancyLi1013 , this issue is kind of preventing us to update our internal vcpkg version. If you could let us know which kind of solution or mitigation (see https://github.com/microsoft/vcpkg/issues/10119#issuecomment-588164699) it would make sense to implement for you, I would be happy to submit a PR.

[JackBoosY]

I remember this bug has been fixed, you can update and rebuild vcpkg to remove the wrong port using the remove command as prompted.

[traversaro]

Hi @JackBoosY, I am not sure what you mean, I created a new build in https://github.com/robotology/robotology-vcpkg-binary-ports/pull/9 and apparently the problem is still present in the latest commit on the master branch, while everything is fine in 2019.10 and 2020.1 . Just to clarify, the problem was exactly that before February 2020 it was possible to install new ports after a custom port has been installed without removing the custom port.

[traversaro]

I investigated a bit the issue.

The root cause is that in https://github.com/microsoft/vcpkg/blob/941d5464544eb5812b2cbd6e687c3ebc78ed2624/toolsrc/src/vcpkg/dependencies.cpp#L593 for every install action a complete graph of the installed ports and their source CONTROL file is constructed, and that is where it fails for ports that are installed via overlay-ports . Note that this information is then not used if the installed package is not related to the missing port. However, modifying this part to build a graph of just the ports connected to the port that is being installed seems to be a non-trivial modification.

Furthermore, more in general I think that if after https://github.com/microsoft/vcpkg/pull/9895 the assumption is that the available sources CONTROL reflects the configuration used when a port have been installed, I think that leaving to the user to specify via the command line every time the used overlay-ports as described in https://github.com/microsoft/vcpkg/blob/941d5464544eb5812b2cbd6e687c3ebc78ed2624/docs/specifications/ports-overlay.md seems to be error prone. In particular, the actual ports installed strongly depend on the order of the passed overlay-ports, and for a user it is quite difficult to get exactly the same order every time.

For these reasons, I think the best solution to be able to use the overlay-ports features after the modifications in https://github.com/microsoft/vcpkg/pull/9895 is to add the possibility to have a default-overlay-ports.txt file in the vcpkg root. If this files is not present, the functionality of vcpkg and the overlay-ports are the same. If instead the file default-overlay-ports.txt is present in the vcpkg root directory, it is used to be passed as an overlay-ports options automatically. In particular, if default-overlay-ports.txt is present, then every call to vcpkg:
~
./vcpkg.exe --opt1 arg1 --opt2 arg2 ... --optN argN verb argVerb
~
is actually equivalent to the call:
~
./vcpkg.exe --opt1 arg1 --opt2 arg2 ... --optN argN --overlay-ports ./default-overlay-ports.txt verb argVerb
~

In this way, adding an overlay port can be done permanently with a few commands (in Powershell or bash):
~
cd $VCPKG_ROOT
git clone https://github.com/robotology/robotology-vcpkg-binary-ports ./overlay-ports/robotology-vcpkg-binary-ports
echo "./overlay-ports/robotology-vcpkg-binary-ports" >> ./default-overlay-ports.txt
~

If this seems a reasonable solution I would be happy to provide a PR.

[traversaro]

For these reasons, I think the best solution to be able to use the overlay-ports features after the modifications in #9895 is to add the possibility to have a default-overlay-ports.txt file in the vcpkg root. If this files is not present, the functionality of vcpkg and the overlay-ports are the same. If instead the file default-overlay-ports.txt is present in the vcpkg root directory, it is used to be passed as an overlay-ports options automatically.

To validate this feature, I tried to implement it in https://github.com/traversaro/vcpkg/tree/fix/10119 . While it still needs to be completed (the relative paths in the files are now resolved as the relative paths passed to the overlay-ports command line parameters, instead of being resolved relative to the file location) but it seems to be working as intended.

[michaelquigley]

Could we, or should we open an issue to deal with implementing a mechanism to persist --overlay-ports for an installation? Having to specify --overlay-ports on every vcpkg invocation is fairly painful.

[traversaro]

Hi @michaelkonecny ! I proposed such a mechanism in the previous comments, but unfortunately it did not received any feedback. If you are interested in this, feel free to continue to work on it (I do not currently plan to work on that), I suspect that if we open a PR with a concrete proposal it would be much easier to get feedback rather then opening another issue with just abstract discussions.

[traversaro]

Note that I also saw mentioned in the Roadmap (https://github.com/microsoft/vcpkg/wiki/Roadmap) that upstream is working on a Feature called "Package federation" that should allow "Vcpkg will allow you to easily install packages from multiple sources". I do not have any further info on this, but it may be possible that this new feature will render --overlay-ports and this issue obsolete, and that is the main reason I stopped working on solving this issue.

[Neumann-A]

@traversaro what about the new environment variable VCPKG_OVERLAY_PORTS. (see #12640)

[traversaro]

@traversaro what about the new environment variable VCPKG_OVERLAY_PORTS. (see #12640)

Thanks, that is indeed a convenient feature. For some corner use cases (such as distributing a ready to use archive with vcpkg + some repo of custom ports) the file variant was more convenient, but the env variable can be used instead with some tricks. In any case, before closing this issue it could make sense to explicitly document that once you install a port from a overlay you need to always keep the overlay directory in VCPKG_OVERLAY_PORTS.

[traversaro]

If I understand correctly, this issue should be fixed by https://github.com/microsoft/vcpkg/pull/13231 .

[strega-nil]

Fixed in #13231 :D

[traversaro]

Thanks a lot to everyone that contributed to fixing this bug!