Vcpkg: Is there a way to specify *.exe in \vcpkg\downloads\tools\?

[Frank-Dz]

Recently, there is a problem bothering me.
One pkg needs to use one tool: msys2.exe to install something. And the setting in VCPKG is to download a zipped file of msys2, extract it and use it. Yet this method needs to apply a key from a key server for msys2. I tried to make it (with key updated and set proxy up) and wait for VCPKG to use it in \downloads\tools. However, when I run my command in Powershell, it seems like VCPKG re-extract the zipped file and re-install it (including applying for key and update.) After such an operation, all my setting is removed. Since the default key server in msys2 is bad (but to use such an extracted msys2 in VCPKG, we must enter a key.)

Thus, I decided just to download an installer install it in \vcpkg\downloads\tools\ and make VAPKG try to use this tool. Is it possible? How should we specify the setting?

In my problem, I guess VCPKG firstly check the downloaded zip file and check the key between msys2.exe and zipped msys2. When it finds the key is not compatible, it deletes my program which has been set up.

Best,
Frank

[Frank-Dz]

This is the details:
I wonder why it doesn't directly check the folder downloads\tools but to check the zipped uninstalled pkg first?

Log:
./vcpkg install icu:x64-windows
Computing installation plan...
The following packages will be built and installed:
icu[core]:x64-windows
Starting package 1/1: icu:x64-windows
Building package icu[core]:x64-windows...
-- Using cached C:/local/vcpkg/downloads/icu4c-65_1-src.tgz
-- Using source at C:/local/vcpkg/buildtrees/icu/src/c-65_1-src-702c9cff96
-- Acquiring MSYS2...
-- Using cached C:/local/vcpkg/downloads/msys2-base-x86_64-20181211.tar.xz
CMake Error: Problem with archive_write_header(): Can't unlink already-existing object
CMake Error: Current file: msys64/usr/bin/dirmngr.exe
CMake Error: Problem extracting tar: C:/local/vcpkg/downloads/msys2-base-x86_64-20181211.tar.xz
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
C:/local/vcpkg/downloads/tools/msys2/msys64/usr/bin/awk.exe: error while loading shared libraries: msys-readline7.dll: cannot open shared object file: No such file or directory
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
gpg: Generating pacman keyring master key...
gpg: key 33A44C00167212D5 marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/EEFEDB144146E4D28890DBA733A44C00167212D5.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
==> Appending keys from msys2.gpg...
==> Locally signing trusted keys in keyring...
-> Locally signing key D55E7A6D7CE9BA1587C0ACACF40D263ECA25678A...
-> Locally signing key 123D4D51A1793859C2BE916BBBE514E53E0D0813...
-> Locally signing key B91BCF3303284BF90CC043CA9F418C233E652008...
-> Locally signing key 9DD0D4217D75A33B896159E6DA7EF2ABAEEA755C...
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
==> Updating trust database...
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 4 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1 valid: 4 signed: 3 trust: 0-, 0q, 0n, 4m, 0f, 0u
gpg: depth: 2 valid: 3 signed: 0 trust: 3-, 0q, 0n, 0m, 0f, 0u

[ghost]

@Frank-Dz

I am no expert, but I would try.

• create a test instance of vcpkg - I have 4 instances named vs2017.prod vs2017.test vs2019.prod and vs2019.test, This way I can experiment with vcpkg without corrupting my good prod ones used for day-to-day programming 👍
• to put my version of msys2 where vcpkg looks for it, like you have already done (_vcpkg-root_\downloads\tools\msys2\msys64 or _vcpkg-root_\downloads\tools\msys2\msys32 if the OS is 32-bit).
• create a msys2-base-x86-64-20181211.tar.gz from the directory of your msys2 (or msys2-base-i686-20181211.tar.xz if the OS is 32-bit) and put in the _vcpkg-root_\downloads folder/directory
• get the SHA512 hash of the msys2-base-x86-64-20181211.tar.gz file created above (or msys2-base-i686-20181211.tar.xz if the OS is 32-bit)
• modify the _vcpkg_root_\scripts\cmake\vcpkg_acquire_msys.cmake line 59 with the SHA512 hash of msys2-base-x86-64-20181211.tar.gz file created above (or msys2-base-i686-20181211.tar.xz if the OS is 32-bit line 67)
• lastly may sure you have a file with a 0 (zero) in it named initialized-msys2_64.stamp (or initialized-msys2_32.stamp if the OS is 32-bit) in the _vcpkg-root_\downloads\tools\msys2 folder\directory
• hopefully when vcpkg runs and needs msys2, it will see a the tar.gz file exists in the downloads folder/directory, the SHA512 matches and then see it has already been untarred and ungzipped and exists.

cc @JackBoosY

• I too experience problems with the MSYS2 host used in vcpkg_acquire_msys.cmake file, from about 12am to 4am eastern (GMT -5)
• I researched and it seems the host server is in Toronto, Canada, eastern time zone (GMT -5)
• Most of the users having a problem are in China (GMT +8)
• Basically the server may be undergoing maintenance from when it is mid day in China???
• I research further and there appears to be several mirrors, maybe a change to vcpkg can be made if there is a timeout on the primary server http://repo.msys2.org/msys it could keep trying the mirrors before creating a download error:
  
  
  
  • Primary: http://repo.msys2.org/msys
  
  
  • Mirror1: https://sourceforge.net/projects/msys2/files/REPOS/MSYS2
  
  
  • Mirror2: https://www2.futureware.at/~nickoe/msys2-mirror/msys
  
  
  • Mirror3: https://mirror.yandex.ru/mirrors/msys2/msys
  
  

It looks like the issues https://github.com/msys2/MSYS2-packages/pull/1659 https://github.com/msys2/MSYS2-packages/pull/1658 on the https://github.com/msys2/MSYS2-packages project site

Best of luck and have a wonderful weekend 👍

[Frank-Dz]

WOW!!!!
Thank you very much! It works for me!!!
Thank you very much for providing me with such detailed instructions. Your work is great!
====================log======================
PS C:\local\vcpkg2> ./vcpkg install icu:x64-windows
Computing installation plan...
The following packages will be built and installed:
icu[core]:x64-windows
Starting package 1/1: icu:x64-windows
Building package icu[core]:x64-windows...
-- Using cached C:/local/vcpkg2/downloads/icu4c-65_1-src.tgz
-- Extracting source C:/local/vcpkg2/downloads/icu4c-65_1-src.tgz
-- Applying patch C:/local/vcpkg2/ports/icu/disable-escapestr-tool.patch
-- Applying patch C:/local/vcpkg2/ports/icu/remove-MD-from-configure.patch
-- Applying patch C:/local/vcpkg2/ports/icu/fix_parallel_build_on_windows.patch
-- Applying patch C:/local/vcpkg2/ports/icu/fix-extra.patch
-- Using source at C:/local/vcpkg2/buildtrees/icu/src/c-65_1-src-702c9cff96
-- Acquiring MSYS Packages...
-- Acquiring MSYS Packages... OK
-- Configuring x64-windows-rel

Best,
Frank

[ghost]

@Frank-Dz

Glad you got it to work as you needed it 🥇

You are more than welcome, I just try to help the best I can in my spare time.

Note: I would save the changed vcpkg_root\scripts\cmake\vcpkg_acquire_msys.cmake, the tar.gz file of your msys2 installation and the instructions, in case there is a change to vcpkg_acquire_msys.cmake file in the future or you need to update your msys2 installation you can recreate it so vcpkg will work with your version :)

[Frank-Dz]

Yep! Many thanks for you kind help and guidance! Plus, after reading your Bio. I was deeply moved! I express my greatest respect and thanks to you.

I am not sure if you are still working on the design and development of VCPKG. I do hope that tools like VCPKG will work well for more people on the Windows platform.
In fact, many users from China and other countries can not use VCPKG smoothly due to Great Wall restrictions (firewall). For those unfamiliar with some basic configuration, this will be frustrating and not conducive to the promotion of VCPKG.
If these can be further improved, I believe it will benefit the spread of this great package manager!

Again, thanks for your kind help and guidance!

Best,
Frank

[Frank-Dz]

@timautry

Hi~ But I still wonder why VCPKG checks the downloaded .tar.xz or .tar.gz files firstly?
If it works in such a way, does that mean I can not delete these .tar.xz or .tar.gz files to save my
disk space?

Thanks!

Best,
Frank

[ghost]

I am not sure why the "items in the tools" folder/directory are that way, most likely checking it has the correct tools to work with, but it can be changed as done with msys2.
The "old" downloads files can be deleted, like a new version is downloaded - my downloads folder is about 7.5 gb and I have not cleaned it up in several months.
A big space saver is the buildtrees folder, you can delete anything in there as it is just used as a work area. The packages folder can be deleted also, "if" you are not going to export the port packages to another vcpkg instance, either locally or another machine.

[JackBoosY]

@Frank-Dz
vcpkg will first check the download file according to the url in portfile.cmake (Yes: use the downloaded file in downloads, no: download source), and determine whether the port has been updated and corrected according to the version number (Yes: use the downloaded file to restart Extract, no: use the decompressed source in buildtrees directly), then build the port.

I also thought about adding a synchronous repo to the Chinese environment, but most of the download tasks of vcpkg are in other upstream, what we really need is a source like 163 or aliyun.

[Frank-Dz]

I am not sure why the "items in the tools" folder/directory are that way, most likely checking it has the correct tools to work with, but it can be changed as done with msys2.
The "old" downloads files can be deleted, like a new version is downloaded - my downloads folder is about 7.5 gb and I have not cleaned it up in several months.
A big space saver is the buildtrees folder, you can delete anything in there as it is just used as a work area. The packages folder can be deleted also, "if" you are not going to export the port packages to another vcpkg instance, either locally or another machine.

Thanks! My 'buildtrees' is more than 40GB... I just clear it. And I feel so good!
Thank you for your help!

Best,
Frank

[Frank-Dz]

vcpkg will first check the download file according to the url in portfile.cmake (Yes: use the downloaded file in downloads, no: download source), and determine whether the port has been updated and corrected according to the version number (Yes: use the downloaded file to restart Extract, no: use the decompressed source in buildtrees directly), then build the port.

That's great! Thank you! It is really good news for the community in China to learn and use VCPKG.

Best,
Frank