Vcpkg: [libmagic] Clean build failure on Windows

[emptyVoid]

Host Environment

• OS: Windows 10 64-bit
• Compiler: not applicable

To Reproduce
Steps to reproduce the behavior:

git clean -fXd
.\bootstrap-vcpkg.bat
.\vcpkg.exe install libmagic

Failure logs

CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:72 (message):
    Command failed: D:/Projects/GitHub/vcpkg/downloads/tools/msys2/msys64/usr/bin/bash.exe --noprofile --norc --debug -c "autoreconf -vfi"
    Working Directory: D:/Projects/GitHub/vcpkg/buildtrees/libmagic/src/9852e25398-4363addce8/
    Error code: 127
    See logs for more information:
      D:\Projects\GitHub\vcpkg\buildtrees\libmagic\autoconf-x64-windows-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_make.cmake:349 (vcpkg_execute_required_process)
  ports/libmagic/portfile.cmake:9 (vcpkg_configure_make)
  scripts/ports.cmake:76 (include)

autoconf-x64-windows-err.log

Additional context
I believe it's a defect in vcpkg_configure_make, since the necessity of autoconf gets decided too late for it to be acquired.

Here's where decision to use autoconf is made:
https://github.com/microsoft/vcpkg/blob/27c6c9420277c533cc4333bdd880f9862012cb72/scripts/cmake/vcpkg_configure_make.cmake#L322-L332
And here's where it gets acquired:
https://github.com/microsoft/vcpkg/blob/27c6c9420277c533cc4333bdd880f9862012cb72/scripts/cmake/vcpkg_configure_make.cmake#L154-L161

@Neumann-A, could you, please, take a look?

[Voskrese]

vcpkg install libmagic

Computing installation plan...
The following packages will be built and installed:
    libmagic[core]:x86-windows
Starting package 1/1: libmagic:x86-windows
Building package libmagic[core]:x86-windows...
-- The Fortran compiler identification is unknown
-- Downloading https://github.com/file/file/archive/a0d5b0e4e9f97d74a9911e95cedd579852e25398.tar.gz...
-- Extracting source E:/tools/vcpkg/downloads/file-file-a0d5b0e4e9f97d74a9911e95cedd579852e25398.tar.gz
-- Using source at E:/tools/vcpkg/buildtrees/libmagic/src/9852e25398-4363addce8
-- Acquiring MSYS Packages...
-- Acquiring MSYS Packages... OK
-- Generating configure for x86-windows
-- Finished generating configure for x86-windows
-- Configuring x86-windows-dbg

CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:72 (message):
    Command failed: E:/tools/vcpkg/downloads/tools/msys2/msys64/usr/bin/bash.exe --noprofile --norc --debug -c " CC='E:/tools/vcpkg/downloads/tools/msys2/msys64/usr/share/automake-1.16/compile cl.exe -nologo' CXX='E:/tools/vcpkg/downloads/tools/msys2/msys64/usr/share/automake-1.16/compile cl.exe -nologo' LD='link.exe -verbose' AR='E:/tools/vcpkg/downloads/tools/msys2/msys64/usr/share/automake-1.16/ar-lib lib.exe -verbose' RANLIB=':' CCAS=':' NM='dumpbin.exe -symbols -headers -all' DLLTOOL='link.exe -verbose -dll' ./../src/9852e25398-4363addce8/configure --build=x86_64-pc-mingw32 --target=i686-pc-mingw32 --host=i686-pc-mingw32 --disable-silent-rules --verbose --enable-shared --disable-static --prefix=/E/tools/vcpkg/installed/x86-windows/debug --bindir='${prefix}'/../tools/libmagic/debug/bin --sbindir='${prefix}'/../tools/libmagic/debug/sbin --includedir='${prefix}'/../include --datarootdir='${prefix}'/share/libmagic"
    Working Directory: E:/tools/vcpkg/buildtrees/libmagic/x86-windows-dbg
    Error code: 1
    See logs for more information:
      E:\tools\vcpkg\buildtrees\libmagic\config-x86-windows-dbg-out.log
      E:\tools\vcpkg\buildtrees\libmagic\config-x86-windows-dbg-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_make.cmake:499 (vcpkg_execute_required_process)
  ports/libmagic/portfile.cmake:9 (vcpkg_configure_make)
  scripts/ports.cmake:76 (include)

another problem
deps: libgnurx

configure: error: libgnurx is required to build file(1) with MinGW

[JackBoosY]

@Neumann-A Could you please take a look?

Thanks.

[emptyVoid]

Same issue with libb2.

[emptyVoid]

And with libwandio.

[Neumann-A]

@JackBoosY: This one cannot be build with cl until we get a C regex library with visual studio.

Reason:
D:\qt2\buildtrees\libmagic\src\9852e25398-4363addce8\src\file.h(69): fatal error C1083: Cannot open include file: 'regex.h': No such file or directory

[Voskrese]

@Neumann-A
https://github.com/nscaife/file-windows

file(libmagic) + libgnurx-2.5

https://github.com/nscaife/file-windows/blob/master/build.sh

#!/bin/bash
mkdir dist/
cd libgnurx-2.5
make
sudo cp regex.h /usr/x86_64-w64-mingw32/include/
sudo cp libregex.a /usr/x86_64-w64-mingw32/lib/
sudo cp libgnurx.dll.a /usr/x86_64-w64-mingw32/lib/
cp COPYING.LIB ../dist/COPYING.libgnurx
cp libgnurx-0.dll ../dist/
cd ../file/
autoreconf -f -i
./configure --disable-silent-rules --enable-fsect-man5
make -j4
cp magic/magic.mgc ../dist/
make clean
./configure --disable-silent-rules --enable-fsect-man5 --host=x86_64-w64-mingw32
make -j4
cp src/.libs/libmagic-1.dll ../dist/
cp src/.libs/file.exe ../dist/
cp COPYING ../dist/COPYING.file

[Voskrese]

tested, not work in msvc libmagic

mingw (deps: libtre + libsystre)
msvc(cmake) (deps: dirent + getopt + pcre2) -> https://github.com/julian-r/file-windows

[JackBoosY]

Since #11810 merged. does this issue resolved?

[emptyVoid]

No, it's a vcpkg_configure_make issue and a port bug. Should be partially fixed by #11836.

[JackBoosY]

Should be fixed by PR #11836.

[emptyVoid]

@JackBoosY, no the issue is still there:

PS D:\Projects\GitHub\vcpkg> .\vcpkg.exe install libmagic
Computing installation plan...
A suitable version of cmake was not found (required v3.17.2). Downloading portable cmake v3.17.2...
Downloading cmake...
  https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2-win32-x86.zip -> D:\Projects\GitHub\vcpkg\downloads\cmake-3.17.2-win32-x86.zip
Extracting cmake...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
  https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> D:\Projects\GitHub\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
Extracting 7zip...
A suitable version of nuget was not found (required v5.5.1). Downloading portable nuget v5.5.1...
Downloading nuget...
  https://dist.nuget.org/win-x86-commandline/v5.5.1/nuget.exe -> D:\Projects\GitHub\vcpkg\downloads\22ea847d-nuget.exe
The following packages will be built and installed:
    libmagic[core]:x64-windows
Detecting compiler hash for triplet x64-windows...
A suitable version of powershell-core was not found (required v6.2.1). Downloading portable powershell-core v6.2.1...
Downloading powershell-core...
  https://github.com/PowerShell/PowerShell/releases/download/v6.2.1/PowerShell-6.2.1-win-x86.zip -> D:\Projects\GitHub\vcpkg\downloads\PowerShell-6.2.1-win-x86.zip
Extracting powershell-core...
Starting package 1/1: libmagic:x64-windows
Building package libmagic[core]:x64-windows...
Could not locate cached archive: C:\Users\emptyVoid\AppData\Local\vcpkg/archives\a6\a61992b75be04ea1a06acc8161b560477d8f28c0.zip
-- Downloading https://github.com/file/file/archive/a0d5b0e4e9f97d74a9911e95cedd579852e25398.tar.gz...
-- Extracting source D:/Projects/GitHub/vcpkg/downloads/file-file-a0d5b0e4e9f97d74a9911e95cedd579852e25398.tar.gz
-- Using source at D:/Projects/GitHub/vcpkg/buildtrees/libmagic/src/9852e25398-4363addce8.clean
-- Acquiring MSYS2...
-- Downloading https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20190524.tar.xz/download...
-- Downloading http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz...
-- Downloading http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig...
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
gpg: Generating pacman keyring master key...
gpg: key D0AE18980C5C8225 marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/5210D2BA8E5635D450D21D6ED0AE18980C5C8225.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
==> Appending keys from msys2.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key D55E7A6D7CE9BA1587C0ACACF40D263ECA25678A...
  -> Locally signing key 123D4D51A1793859C2BE916BBBE514E53E0D0813...
  -> Locally signing key B91BCF3303284BF90CC043CA9F418C233E652008...
  -> Locally signing key 9DD0D4217D75A33B896159E6DA7EF2ABAEEA755C...
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   4  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   4  signed:   3  trust: 0-, 0q, 0n, 4m, 0f, 0u
gpg: depth: 2  valid:   3  signed:   0  trust: 3-, 0q, 0n, 0m, 0f, 0u
==> Checking D:/Projects/GitHub/vcpkg/downloads/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig...
gpg: assuming signed data in 'D:/Projects/GitHub/vcpkg/downloads/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz'
gpg: Signature made Mon Jun 29 08:36:14 2020 RTZST
gpg:                using DSA key AD351C50AE085775EB59333B5F92EFC1A47D45A1
gpg: Good signature from "Alexey Pavlov (Alexpux) <[email protected]>" [full]
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) msys2-keyring-r21.b39fb11-1

Total Installed Size:  0.05 MiB
Net Upgrade Size:      0.03 MiB

:: Proceed with installation? [Y/n]
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
upgrading msys2-keyring...
==> Appending keys from msys2.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key 6E8FEAFF9644F54EED90EEA0790AE56A1D3CFDDC...
  -> Locally signing key D55E7A6D7CE9BA1587C0ACACF40D263ECA25678A...
  -> Locally signing key 123D4D51A1793859C2BE916BBBE514E53E0D0813...
  -> Locally signing key B91BCF3303284BF90CC043CA9F418C233E652008...
  -> Locally signing key 9DD0D4217D75A33B896159E6DA7EF2ABAEEA755C...
  -> Locally signing key 69985C5EB351011C78DF7F6D755B8182ACD22879...
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
==> Disabling revoked keys in keyring...
  -> Disabling key B19514FB53EB3668471B296E794DCF97F93FC717...
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   6  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   6  signed:   6  trust: 0-, 0q, 0n, 6m, 0f, 0u
gpg: depth: 2  valid:   6  signed:   0  trust: 6-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2021-06-19
:: Synchronizing package databases...
downloading mingw32.db...
downloading mingw32.db.sig...
downloading mingw64.db...
downloading mingw64.db.sig...
downloading msys.db...
downloading msys.db.sig...
resolving dependencies...
looking for conflicting packages...

Packages (3) libzstd-1.4.5-2  zstd-1.4.5-2  pacman-5.2.2-1

Total Download Size:    11.31 MiB
Total Installed Size:   34.03 MiB
Net Upgrade Size:      -12.57 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
downloading libzstd-1.4.5-2-x86_64.pkg.tar.xz...
downloading zstd-1.4.5-2-x86_64.pkg.tar.xz...
downloading pacman-5.2.2-1-x86_64.pkg.tar.xz...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
installing libzstd...
installing zstd...
upgrading pacman...
checking dependencies...

Packages (2) rebase-4.4.4-1  dash-0.5.10.2-1

Total Removed Size:  1.12 MiB

:: Do you want to remove these packages? [Y/n]
:: Processing package changes...
removing rebase...
removing dash...
:: Synchronizing package databases...
 mingw32 is up to date
 mingw64 is up to date
 msys is up to date
:: Starting core system upgrade...
warning: resolving dependencies...
terminate other MSYS2 programs before proceeding
looking for conflicting packages...

Packages (5) bash-4.4.023-2  filesystem-2020.02-7  mintty-1~3.3.0-1  msys2-runtime-3.1.6-3  pacman-mirrors-20200329-1

Total Download Size:    5.27 MiB
Total Installed Size:  24.38 MiB
Net Upgrade Size:       1.26 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
downloading msys2-runtime-3.1.6-3-x86_64.pkg.tar.xz...
downloading bash-4.4.023-2-x86_64.pkg.tar.xz...
downloading filesystem-2020.02-7-x86_64.pkg.tar.xz...
downloading mintty-1.3.3.0-1-x86_64.pkg.tar.xz...
downloading pacman-mirrors-20200329-1-any.pkg.tar.xz...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
upgrading msys2-runtime...
upgrading bash...
upgrading filesystem...
upgrading mintty...
upgrading pacman-mirrors...
:: To complete this update all MSYS2 processes including this terminal will be closed. Confirm to proceed [Y/n] warning: terminating MSYS2 processes failed

-- Acquiring MSYS2... OK
-- Acquiring MSYS Packages from D:/Projects/GitHub/vcpkg/downloads/tools/msys2-20200812...
-- Acquiring MSYS Packages... OK
-- Downloading https://repo.msys2.org/mingw/i686/mingw-w64-i686-pkg-config-0.29.2-1-any.pkg.tar.xz...
-- Downloading https://repo.msys2.org/mingw/i686/mingw-w64-i686-libwinpthread-git-8.0.0.5906.c9a21571-1-any.pkg.tar.zst...
-- Generating configure for x64-windows
-- Finished generating configure for x64-windows
-- Configuring x64-windows-dbg
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:72 (message):
    Command failed: D:/Projects/GitHub/vcpkg/downloads/tools/msys2-20200812/msys64/usr/bin/bash.exe --noprofile --norc --debug -c "V=1 CPP='compile cl.exe -nologo -E' CC='compile cl.exe -nologo' CXX='compile cl.exe -nologo' AR='ar-lib lib.exe -verbose' LD='link.exe -verbose' RANLIB=':' CCAS=':' STRIP=':' NM='dumpbin.exe -symbols -headers' DLLTOOL='link.exe -verbose -dll' ./../src/9852e25398-4363addce8.clean/configure --build=x86_64-pc-mingw32 --disable-silent-rules --verbose --enable-shared --disable-static --prefix=/D/Projects/GitHub/vcpkg/installed/x64-windows/debug --bindir='${prefix}'/../tools/libmagic/debug/bin --sbindir='${prefix}'/../tools/libmagic/debug/sbin --includedir='${prefix}'/../include --datarootdir='${prefix}'/share/libmagic"
    Working Directory: D:/Projects/GitHub/vcpkg/buildtrees/libmagic/x64-windows-dbg
    Error code: 1
    See logs for more information:
      D:\Projects\GitHub\vcpkg\buildtrees\libmagic\config-x64-windows-dbg-out.log
      D:\Projects\GitHub\vcpkg\buildtrees\libmagic\config-x64-windows-dbg-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_make.cmake:566 (vcpkg_execute_required_process)
  ports/libmagic/portfile.cmake:9 (vcpkg_configure_make)
  scripts/ports.cmake:79 (include)


Error: Building package libmagic:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `.\vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: libmagic:x64-windows
  Vcpkg version: 2020.06.15-nohash

Additionally, attach any relevant sections from the log files above.

config-x64-windows-dbg-err.log
config-x64-windows-dbg-out.log

[sfhacker]

@emptyVoid Have you been able to resolve this issue?

We have just given it a try and this is what we got:

We are not familiar with this project (libmagic) so we do not know how to continue!

[emptyVoid]

Neither do I, sorry. I was just trying to use this port to test vcpkg_acquire_msys.

[sfhacker]

@emptyVoid Who has asked you to use this library (libmagic)? What do you need it for?

[emptyVoid]

I was testing the changes I had made to vcpkg_acquire_msys on all the ports which directly or otherwise used it. That's all.

[sfhacker]

I came to know that libmagic is obsolete and no longer maintained!
Is it wise to continue using it?
Which ports, directly or otherwise, use libmagic?

[emptyVoid]

You could probably ask the author of the port request in #7895.

[sfhacker]

7895 has been closed for some time now .

So the question is: @emptyVoid Why/What do you need libmagic project for? Which other ports? Please, be specific. You are the one who have raised this issue!

[emptyVoid]

Of course #7895 got closed as soon as the request was satisfied.
And could you, please, re-read my previous posts -- I've already told you twice how I stumbled upon this build issue.

[stephanecharette]

I'm currently blocked on this issue while porting some Linux software to Windows. If a simple workaround is known, I could definitely use it.

[sfhacker]

Are we STILL having issues with this small and simple port (libmagic)?

15/09/2020 05:58 AM 223296 libmagic.a

@stephanecharette How familiar are you with building projects from source? A newbie?
How much experience do you have with cmake? There are plenty of online tutorials you can look at!
zydis tag 3.1.0 may be incomplete. However, we should not make a mountain out of a molehill!
It is somehow embarrassing to read that you are stuck!

To my mind, there are two options here (depending on your particular needs):

a) Generate a *-config.cmake file for the particular tag no. you need
b) Create .pc file so pkg-config can find zydis project installed on your PC

If you lack the skills to perform any of those options or do not understand what you need to do, let me know. I'll provide a workaround for you.

Have fun.

[stephanecharette]

@sfhacker I don't know how else to say this to you...but get lost. You are not helping.

Unless you can actually contribute by telling people the exact vcpkg.exe command you ran to install libmagic, you claiming that people are an embarrassment just shows that you are out here to troll.

[JackBoosY]

After adding

if(CMAKE_HOST_WIN32)
    if (VCPKG_TARGET_ARCHITECTURE STREQUAL "x86")
        vcpkg_acquire_msys(MSYS_ROOT DIRECT_PACKAGES
            "https://repo.msys2.org/mingw/i686/mingw-w64-i686-libgnurx-2.5.1-2-any.pkg.tar.zst"
            e5a9b1101d703e29cc5aab1dd79e8ca11ae80c1565dbd66e4ad7e0b896db8b49c274e6a29b0936bf4f88e9ba528ba2a3269344aba055292b5ec64b7b8e65212a
        )
    else()
        vcpkg_acquire_msys(MSYS_ROOT DIRECT_PACKAGES
            "https://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-libgnurx-2.5.1-2-any.pkg.tar.zst"
            687026d14b931df86fb9459577d65ed0d5afd4d03aa2d80ff297bf0c5205313e05e70a033958f29fcda53b3cb6591232adad9a809c49a6305e9d6611bbd42c3d
        )
    endif()
endif()

to _VCPKG_ROOT/ports/libmagic/portfile.cmake_, this issue was not resolved.

The compiler serch for gnurx.lib instead of libgnurx.a.

[sfhacker]

This issue seems particularly hard to troubleshoot and fix!

[JackBoosY]

@sfhacker Yes, I don’t know much about cross-platform compilation, maybe you can help?