Vcpkg: Unix tools: move away from MSYS2 (if possible)

Created on 29 Jul 2020  路  41Comments  路  Source: microsoft/vcpkg

Is your feature request related to a problem? Please describe.
There are several projects like win-bash, UnxUtils and GnuWin32 that provides most of the tools needed for bash/autotools-based build systems. If we can move to one of these projects, we can avoid the slow build time caused by MSYS2

picture longnguyen2004
👍3

Most helpful comment

For pieces that still require Msys, we can use a technique like in https://github.com/microsoft/vcpkg/pull/12626; download the packages directly and manually unpack them together instead of using pacman. Most (all?) of our observed msys instability comes from the package manager and silent updates, not from the actual binaries.

Obviously other solutions like busybox are fine too (and may be less complicated if the msys version has lots of dependencies!), but I think we can pick-and-choose on a per-tool basis what's best.

picture ras0219 on 10 Aug 2020
👍4

All 41 comments

The tools in these projects were very old, over 10 years not updated.

linquize picture linquize on 29 Jul 2020

VCPKG could probably build all those tools itself which I am strongly in favor of. Some of those tools are also simply scripts. Depending on MSYS will always occasionally break VCPKG.

Neumann-A picture Neumann-A on 29 Jul 2020
👍1

The tools in these projects were very old, over 10 years not updated.

We don't really need up to date tools for these, unless the build system depends on some new features

longnguyen2004 picture longnguyen2004 on 29 Jul 2020

And AFAIK win-bash is still active (last update around 2019) so I think it's fine

longnguyen2004 picture longnguyen2004 on 29 Jul 2020

And AFAIK win-bash is still active (last update around 2019) so I think it's fine

Where is it ?

linquize picture linquize on 29 Jul 2020

Whoops I misread something, sorry for that xD
But still moving away from MSYS2 is crucial for performance and compatibility with 32 bit systems (not that I use 32 bit Windows myself, but it's nice to have)

longnguyen2004 picture longnguyen2004 on 29 Jul 2020

@longnguyen2004
busybox ?

...
    set(TOOLSUBPATH busybox64)
    set(URLS "https://frippery.org/files/busybox/busybox-w64-FRP-3532-g01a256149.exe")
    set(_vfa_RENAME "busybox.exe")
    set(ARCHIVE "busybox-w64-FRP-3532-g01a256149.exe")
    set(NOEXTRACT ON)
    set(HASH ba5143ebc54c499923e7289336e236d7b833f153ac7b6fe633433dbfc3bfc453825671e6f39de5dcf9279b2942f3040cd1110947fc5943798d093e2b38e19b11)
    set(STAMP "initialized-busybox64.stamp")
  else()
    set(TOOLSUBPATH busybox86)
    set(URLS "https://frippery.org/files/busybox/busybox-w32-FRP-3532-g01a256149.exe")
    set(_vfa_RENAME "busybox.exe")
    set(ARCHIVE "busybox-w32-FRP-3532-g01a256149.exe")
    set(NOEXTRACT ON)
    set(HASH 74786326c07c1cf2b11440cbd7caf947c2a32ebcc2b5bb362301d12327a2108182f57e98c217487db75bf6f0e3a4577291933e025b9b170e37848ec0b51a134c)
    set(STAMP "initialized-busybox86.stamp")
...


    vcpkg_execute_required_process(
      ALLOW_IN_DOWNLOAD_MODE
      COMMAND ${BUSYBOX_SETUP} --install ${PATH_TO_ROOT}
      WORKING_DIRECTORY ${TOOLPATH}
      LOGNAME busybox-setup-${TARGET_TRIPLET}
    )

Currently defined functions:
        [, [[, ar, arch, ash, awk, base64, basename, bash, bunzip2, busybox, bzcat, bzip2, cal,
        cat, chattr, chmod, cksum, clear, cmp, comm, cp, cpio, cut, date, dc, dd, df, diff,
        dirname, dos2unix, dpkg, dpkg-deb, du, echo, ed, egrep, env, expand, expr, factor, false,
        fgrep, find, fold, fsync, ftpget, ftpput, getopt, grep, groups, gunzip, gzip, hd, head,
        hexdump, httpd, iconv, id, inotifyd, ipcalc, kill, killall, less, link, ln, logname, ls,
        lsattr, lzcat, lzma, lzop, lzopcat, man, md5sum, mkdir, mktemp, mv, nc, nl, od, paste,
        patch, pgrep, pidof, pipe_progress, pkill, printenv, printf, ps, pwd, readlink, realpath,
        reset, rev, rm, rmdir, rpm, rpm2cpio, sed, seq, sh, sha1sum, sha256sum, sha3sum, sha512sum,
        shred, shuf, sleep, sort, split, ssl_client, stat, strings, su, sum, tac, tail, tar, tee,
        test, time, timeout, touch, tr, true, truncate, ts, ttysize, uname, uncompress, unexpand,
        uniq, unix2dos, unlink, unlzma, unlzop, unxz, unzip, usleep, uudecode, uuencode, vi, watch,
        wc, wget, which, whoami, whois, xargs, xxd, xz, xzcat, yes, zcat
Voskrese picture Voskrese on 31 Jul 2020
👍2

Seems good enough for the job, just need GNU make (Chocolatey has 4.3). @ras0219-msft what do you think?

longnguyen2004 picture longnguyen2004 on 1 Aug 2020

Perl would be a problem though, I haven't known yet if the openssl build system can use native Perl or not...

longnguyen2004 picture longnguyen2004 on 1 Aug 2020

msys2 download time is relatively long.

  1. vcpkg download msys2.
  2. msys2's pacman update packages to latest.

I do not understand that building such a small library zlib requires msys2.

linquize picture linquize on 2 Aug 2020

I thought zlib uses CMake, no? That's probably the pkg-config part, we could use a standalone pkg-config for that.
The real problem is with Perl-based build systems, such as openssl, which will die on native Perl due to backward-vs-forward slash. Patching that out results in the configure script writing unescaped backslashes into the code.
Also, Busybox is not a true Unix environment, and also have other oddities (such as mapping / to the current drive, which makes any scripts using /tmp, /usr,... to either fails, or clutters the drive root with folders)

longnguyen2004 picture longnguyen2004 on 2 Aug 2020

zlib portfile.cmake has vcpkg_fixup_pkgconfig() that calls vcpkg_acquire_msys(pkg-config)

Can we download standalone pkg-config for windows, instead of the whole msys2?

linquize picture linquize on 2 Aug 2020

For vcpkg_fixup_pkgconfig(),
Downloading msys2 base files http://repo.msys2.org/distrib/x86_64/msys2-base-x86_64-20190524.tar.xz +
pkg-config https://repo.msys2.org/msys/x86_64/pkg-config-0.29.2-1-x86_64.pkg.tar.xz
is enough.
Extract only /usr/bin/pkg-config.exe from pkg-config-0.29.2-1-x86_64.pkg.tar.xz is enough.

No need to run pacman that spends too much time updating all the packages before installing pkg-config.

linquize picture linquize on 2 Aug 2020

@linquize the idea is not new see #12626

Neumann-A picture Neumann-A on 2 Aug 2020

About using BusyBox as a substitute for common Unix tools:
Seems like autotools-based build systems don't like it, it complains about missing awk, sed, grep,... so I end up having to do AWK="awk" SED="sed" GREP="grep" ...
Also, native make has some differences compared to MSYS2 make. It defaults to using COMSPEC for the shell, which breaks pretty much all autotools Makefiles. Adding SHELL=sh to the make invocation might work

longnguyen2004 picture longnguyen2004 on 2 Aug 2020

@Neumann-A

why not, an auxiliary utility, if it really helps in the assembly, then obviously can use it?
substitute MSYS I wouldn't recommend using only as an add-on (busybox)

@longnguyen2004
cygwin ? )))
https://github.com/Voskrese/vcpkg-template/blob/master/scripts/cmake/vcpkg_acquire_cygwin.cmake

Voskrese picture Voskrese on 2 Aug 2020

@Voskrese nope that's pretty much same as msys2 xD I need fast GNU make for openssl

longnguyen2004 picture longnguyen2004 on 2 Aug 2020

Alright, current state of my BusyBox test:

| Build systems | Workaround |
| --- | --- |
| autotools-based | Need to specify PATH_SEPARATOR=";", since BusyBox doesn't convert path separators for us. Also needs AWK="awk" SED="sed" .... Resulting Makefiles need to be run with SHELL=sh.exe |
| Simple Perl-based (libvpx) | Needs some patching to the configure script, due to diff --version not returning 0 on BusyBox, otherwise builds fine |
| Complex Perl-based (openssl) | Needs to remove forward slash check, but then it writes unescaped backslashes into the source file (I'm using ActiveState, Strawberry might be different) |

And that's only for common tools. Other ports might require tools such as m4, which needs MSYS2 anyway.

longnguyen2004 picture longnguyen2004 on 2 Aug 2020

~I've just tested ICU today and it doesn't like BusyBox. Seems like we'll have to stick with MSYS2 for now~ Nevermind, just need some patching and it's working now

longnguyen2004 picture longnguyen2004 on 8 Aug 2020
👍1

For pieces that still require Msys, we can use a technique like in https://github.com/microsoft/vcpkg/pull/12626; download the packages directly and manually unpack them together instead of using pacman. Most (all?) of our observed msys instability comes from the package manager and silent updates, not from the actual binaries.

Obviously other solutions like busybox are fine too (and may be less complicated if the msys version has lots of dependencies!), but I think we can pick-and-choose on a per-tool basis what's best.

ras0219 picture ras0219 on 10 Aug 2020
👍4

One thing to note: The current busybox binary on the site doesn't have install, so if someone wants to test it out, you should grab the source and build it yourself

longnguyen2004 picture longnguyen2004 on 11 Aug 2020

What ports are still using sh-based build systems in vcpkg? I wanna do a test on all of them to determine if it's viable to switch to busybox

longnguyen2004 picture longnguyen2004 on 11 Aug 2020

@longnguyen2004 You can find using the keyword vcpkg_configure_make, vcpkg_acquire_msys and vcpkg_find_acquire_program(YASM).

JackBoosY picture JackBoosY on 11 Aug 2020
👍1

PR #9966

Neumann-A picture Neumann-A on 11 Aug 2020

Why related to #9966?

linquize picture linquize on 11 Aug 2020

I'll update this comment as I continue my testing
~All autotools-based build systems require replacing /bin/sh with sh, as busybox doesn't recognize the former~ A recent commit allowed /bin/sh to work. Also we need to explicitly pass tool names (awk, sed, grep,...) and targets (as it can't recognize the target)
All libtool-based build systems require these 2 patches to configure and ltmain.sh/libtool
All tests are done with x64-mingw-dynamic triplet
Additional tools that might be needed:
m4, libregex (for m4)
Configure command: ./configure PATH_SEPARATOR=";" AWK="busybox awk" SED="busybox sed" GREP="busybox grep" DIFF="busybox diff" PATCH="busybox patch" AR="ar.exe" CC="cc" CXX="c++" STRIP="strip" RANLIB="ranlib" RC="windres" WINDRES="windres" --prefix=... --build=... --host=...
Make command: PATH_SEPARATOR=";" AWK="busybox awk" SED="busybox sed" GREP="busybox grep" DIFF="busybox diff" PATCH="busybox patch" AR="ar.exe" CC="cc" CXX="c++" STRIP="strip" RANLIB="ranlib" RC="windres" WINDRES="windres" make SHELL="busybox sh" -j4 (some libs run configure during make)

vcpkg_configure_make

| Libraries | Build system type | Result | Additional notes |
|------------|---------------------|--------------------------|----------------------------------------------------|
| libcrafter | autotools | UNTESTED | Requires autoconf |
| libb2 | autotools + libtool | PASS | |
| libmagic | autotools + libtool | PASS | |
| libosip2 | autotools + libtool | PASS | |
| libwandio | autotools + libtool | SKIP | Doesn't support Windows |
| farmhash | autotools + libtool | PASS | |
| libudns | autotools | SKIP | Doesn't support Windows |
| unixodbc | autotools + libtool | SKIP | Doesn't support Windows |
| openmpi | autotools | SKIP | Doesn't support Windows |
| x264 | Custom | PASS | |
| hwloc | autotools | SKIP | Custom CMakeList.txt available |
| healpix | Custom (healpix) + autotools + libtool (libsharp) | FAIL | No way to pass target to libsharp |
| libmicrohttpd | autotools + libtool | ~FAIL~ PASS | Works after applying my patch |
| gperftools | autotools + libtool | SKIP | Custom CMakeList.txt available |
| libmesh | autotools + libtool | PASS | Requires m4 for netcdf. Compile fails, but configure works |
| ocilib | autotools + libtool | PASS | |
| sdl1 | autotools + libtool | PASS | |
| libusb | autotools + libtool | PASS | |
| freexl | autotools + libtool | PASS | |
| tcl | autotools | PASS | |
| nettle | autotools | PASS | Requires m4 |
| pbc | autotools + libtool | PASS | |
| gmp | autotools + libtool | PASS | |
| libxslt | autotools | SKIP | MSVC + mingw makefile available |
| mpg123 | autotools + libtool | SKIP | CMakeList.txt available, should we switch? |
| mpir | autotools + libtool | PASS | Mark as PASS since gmp also passes |
| libpq | autotools | UNTESTED | Not tested yet due to size constraints |

vcpkg_acquire_msys

| Libraries | Build system type | Result | Additional notes |
|-|-|-|-|
| kf5i18n | CMake | SKIP | Only used for gettext |
| ffnvcodec | None | SKIP | Only used for copying headers |
| openssl-unix | Custom | FAIL | Requires MSYS2 (Unix) Perl |
| libvpx | Custom | PASS | patch configure: diff --version -> command -v diff |
| tensorflow-cc | Bazel | UNTESTED | Not tested yet due to Bazel dependency |
| ffmpeg | Custom | PASS | |
| icu | autotools | PASS | patch pkgdata.cpp: bash -c \"%s\" -> \"%s\") |

longnguyen2004 picture longnguyen2004 on 13 Aug 2020

The PR is related because it adds a whole bunch of ports using autotools.

Neumann-A picture Neumann-A on 13 Aug 2020

Phew, going through that list is quite tiring. Nevertheless, I've made it.
All my results are above.

longnguyen2004 picture longnguyen2004 on 20 Aug 2020

AR="ar.exe" CC="cc" CXX="c++" STRIP="strip" RANLIB="ranlib" RC="windres" WINDRES="windres

That is not a native windows build environment. You need to change it to use cl, link, lib and rc

Neumann-A picture Neumann-A on 20 Aug 2020

AR="ar.exe" CC="cc" CXX="c++" STRIP="strip" RANLIB="ranlib" RC="windres" WINDRES="windres

That is not a native windows build environment. You need to change it to use cl, link, lib and rc

I'm gonna pick a few random ones and test with cl. I expect most of them to have similar results.
But to be fair, most of them already have VS projects, so I don't think it'll make a difference.

longnguyen2004 picture longnguyen2004 on 20 Aug 2020

I do not understand the motivation here. Can someone help?

You are moving from a well tested, constantly updated Linux-like distro to some custom builds. This seems like a big effort for little gain because of some perceived slowness of pacman updates? You could use the nightly installs (so less gets installed), or else install exactly what you need. You can cache all the packages needed too if download speed is an issue. We use zstd now so our packages unpack very very quickly indeed and they compress very well too.

mingwandroid picture mingwandroid on 21 Aug 2020
👍2

You are moving from a well tested, constantly updated Linux-like distro to some custom builds.

The problem is that the 'constantly updated' part results in breaking vcpkg seemingly every week or so. pacman doesn't seem to like being driven in an automated way like we do, and when it prompts for input or otherwise becomes unhappy we waste ~92 hours of compute time.

I believe @ras0219 was looking at downloading the packages from msys directly rather than relying on pacman as an alternate fix.

BillyONeal picture BillyONeal on 21 Aug 2020
👍1

that is #13019

Neumann-A picture Neumann-A on 21 Aug 2020

Pacman can be fully automated. Check the help. You can also point it at your own repository if you are unhappy with the stability of msys2 and curate the package updates manually.

mingwandroid picture mingwandroid on 21 Aug 2020
👍1

@mingwandroid Currently Microsoft does not host its own repositories for the tools, URLs always point to official download locations.

linquize picture linquize on 22 Aug 2020

an alternative is always needed, but not on a permanent basis. msys2 and pacman work properly.

Voskrese picture Voskrese on 22 Aug 2020

The other problem is the size. Although this is somewhat better now due to the use of zstd, but as of today, BusyBox is still way lighter than MSYS2 installer (~500KiB vs ~90MiB) and also works pretty well with the libraries I've tested (see above).

longnguyen2004 picture longnguyen2004 on 22 Aug 2020

BusyBox a stripped-down version of the main utilities and commands

BusyBox sed --help

Usage: sed [-i[SFX]] [-nrE] [-f FILE]... [-e CMD]... [FILE]...
or: sed [-i[SFX]] [-nrE] CMD [FILE]...

    -e CMD  Add CMD to sed commands to be executed
    -f FILE Add FILE contents to sed commands to be executed
    -i[SFX] Edit files in-place (otherwise sends to stdout)
            Optionally back files up, appending SFX
    -n      Suppress automatic printing of pattern space
    -r,-E   Use extended regex syntax
    -b      Keep CR/LF (Windows-only)

If no -e or -f, the first non-option argument is the sed command string.
Remaining arguments are input files (stdin if none).

sed --help

Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...

-n, --quiet, --silent
suppress automatic printing of pattern space
-e script, --expression=script
add the script to the commands to be executed
-f script-file, --file=script-file
add the contents of script-file to the commands to be executed
--help display this help and exit
--text switch to text mode
-V, --version output version information and exit

and this will be seen in errors during compilation

Voskrese picture Voskrese on 22 Aug 2020

Here's my results with MSVC:
Configure command: PATH_SEPARATOR=";" AWK="busybox awk" SED="busybox sed" GREP="busybox grep" DIFF="busybox diff" PATCH="busybox patch" AR="lib.exe -verbose" CC="cl.exe -nologo" CXX="cl.exe -nologo" STRIP=":" RANLIB=":" RC="rc.exe" WINDRES="rc.exe" NM="dumpbin.exe -symbols -headers" DLLTOOL="link.exe -verbose -dll" ./configure --prefix=... --build=x86_64-pc-mingw32 --host=x86_64-pc-mingw32
Make command: PATH_SEPARATOR=";" AWK="busybox awk" SED="busybox sed" GREP="busybox grep" DIFF="busybox diff" PATCH="busybox patch" AR="lib.exe -verbose" CC="cl.exe -nologo" CXX="cl.exe -nologo" STRIP=":" RANLIB=":" RC="rc.exe" WINDRES="rc.exe" NM="dumpbin.exe -symbols -headers" DLLTOOL="link.exe -verbose -dll" SHELL="busybox sh" make -j4 (some libs run configure during make)

vcpkg_configure_make

| Libraries | Build system type | Result | Additional notes |
|------------|---------------------|--------------------------|----------------------------------------------------|
| libcrafter | autotools | SKIP | Doesn't support Windows |
| libb2 | autotools + libtool | PASS | portfile said it's not supported, but my build succeed? |
| libmagic | autotools + libtool | SKIP | Doesn't support Windows? (requires libgnurx) |
| libosip2 | autotools + libtool | SKIP | Doesn't support MSVC |
| libwandio | autotools + libtool | SKIP | Doesn't support Windows |
| farmhash | autotools + libtool | PASS | Compile errors about __builtin_expect and __builtin_unreachable, but configure works|
| libudns | autotools | SKIP | Doesn't support Windows |
| unixodbc | autotools + libtool | SKIP | Doesn't support Windows |
| openmpi | autotools | SKIP | Doesn't support Windows |
| x264 | Custom | FAIL | configure treats MS as GNU for some reason (should we switch to ShiftMedia?) |
| hwloc | autotools | SKIP | Custom CMakeList.txt available |
| healpix | Custom (healpix) + autotools + libtool (libsharp) | FAIL | No way to pass target to libsharp |
| libmicrohttpd | autotools + libtool | SKIP | VS projects available |
| gperftools | autotools + libtool | SKIP | Custom CMakeList.txt available |
| libmesh | autotools + libtool | SKIP | Doesn't support MSVC |
| ocilib | autotools + libtool | SKIP | VS projects available |
| sdl1 | autotools + libtool | SKIP | VS projects available |
| libusb | autotools + libtool | SKIP | VS projects available |
| freexl | autotools + libtool | SKIP | MSVC makefiles available |
| tcl | autotools | SKIP | MSVC makefiles available |
| nettle | autotools | SKIP | VS projects available |
| pbc | autotools + libtool | SKIP | VS projects available |
| gmp | autotools + libtool | SKIP | Use mpir instead |
| libxslt | autotools | SKIP | MSVC + mingw makefiles available |
| mpg123 | autotools + libtool | SKIP | CMakeList.txt available, should we switch? |
| mpir | autotools + libtool | SKIP | VS projects available |
| libpq | autotools | UNTESTED | Not tested yet due to size constraints |

vcpkg_acquire_msys

| Libraries | Build system type | Result | Additional notes |
|-|-|-|-|
| kf5i18n | CMake | SKIP | Only used for gettext |
| ffnvcodec | None | SKIP | Only used for copying headers |
| libvpx | Custom | FAIL | Generator shell scripts incompatible with BusyBox (ShiftMedia?) |
| tensorflow-cc | Bazel | UNTESTED | Not tested yet due to Bazel dependency |
| ffmpeg | Custom | PASS | Add LN_S=cp -pR |

longnguyen2004 picture longnguyen2004 on 24 Aug 2020

I know #13019 isn't exactly what you asked for here but I think we should merge something like that, and then if you still think that's too slow we can have a continued discussion. Unless we are able to completely eliminate the msys dependency from all ports I don't think it would be a good idea to not have a single source of truth for these unix-y tools.

BillyONeal picture BillyONeal on 27 Aug 2020

Sure, after all MSYS2 is a complete Unix environment, while BusyBox is simply a collection of common Unix tools. The main problem for me is that MSYS2 make is SLOW (~30min to build openssl compared to 5-6min in WSL2), if we can find out a way to use native make then it'd be great.

longnguyen2004 picture longnguyen2004 on 28 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ThinkalVB picture ThinkalVB  路  3Comments
LilyWangL picture LilyWangL  路  3Comments
tzbo picture tzbo  路  3Comments
invy picture invy  路  3Comments
ghost picture ghost  路  3Comments