Vcpkg: boost-python build failure

[jepessen]

Host Environment

• OS: Kubuntu 19.10 64 bit
• Compiler: gcc 9.2.1

I've updated vcpkg to following hash

299e8b4ac6008c057edb503eb218f0217e9c8f99

I had already boost installed and I used them before.

When I run the vcpkg upgrade --no-dry-run command I've an error on boost-python.

To Reproduce
Steps to reproduce the behaviour if boost are not installed.
./vcpkg install boost

Steps to reproduce the behaviour if boost are installed:
./vcpkg upgrade --no-dry-run

Failure logs

vcpkg/buildtrees/boost-python/install-x64-linux-dbg-out.log:

[1/2] cd /home/login/Projects/vcpkg/buildtrees/boost-python/src/ost-1.72.0-c6a2496f74/build && /home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build/b2 --user-config=/home/login/Projects/
vcpkg/buildtrees/boost-python/x64-linux-dbg/user-config.jam --stagedir=/home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/stage --build-dir=/home/login/Projects/vcpkg/buildtrees/boost-
python/x64-linux-dbg runtime-link=static link=static address-model=64 target-os=linux toolset=gcc -sZLIB_BINARY=z -sZLIB_LIBPATH="/home/login/Projects/vcpkg/installed/x64-linux/debug/lib/libz.a" -sB
ZIP2_BINARY=bz2d -sBZIP2_LIBPATH="/home/login/Projects/vcpkg/installed/x64-linux/debug/lib/libbz2d.a" variant=debug --layout=system --with-atomic --with-random --with-date_time --with-filesystem --w
ith-system --with-thread --with-chrono -sZLIB_INCLUDE="/home/login/Projects/vcpkg/installed/x64-linux/include" -sBZIP2_INCLUDE="/home/login/Projects/vcpkg/installed/x64-linux/include" -sICU_PATH="/h
ome/login/Projects/vcpkg/installed/x64-linux" -j1 -sBOOST_ROOT=/home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build -sBOOST_BUILD_PATH=/home/login/Projects/vcpkg/installed/x64-linux/tool
s/boost-build --debug-configuration --ignore-site-config --hash -q architecture=x86 threading=multi threadapi=pthread debug-symbols=on stage
FAILED: CMakeFiles/boost 
cd /home/login/Projects/vcpkg/buildtrees/boost-python/src/ost-1.72.0-c6a2496f74/build && /home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build/b2 --user-config=/home/login/Projects/vcpkg/
buildtrees/boost-python/x64-linux-dbg/user-config.jam --stagedir=/home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/stage --build-dir=/home/login/Projects/vcpkg/buildtrees/boost-python
/x64-linux-dbg runtime-link=static link=static address-model=64 target-os=linux toolset=gcc -sZLIB_BINARY=z -sZLIB_LIBPATH="/home/login/Projects/vcpkg/installed/x64-linux/debug/lib/libz.a" -sBZIP2_B
INARY=bz2d -sBZIP2_LIBPATH="/home/login/Projects/vcpkg/installed/x64-linux/debug/lib/libbz2d.a" variant=debug --layout=system --with-atomic --with-random --with-date_time --with-filesystem --with-sy
stem --with-thread --with-chrono -sZLIB_INCLUDE="/home/login/Projects/vcpkg/installed/x64-linux/include" -sBZIP2_INCLUDE="/home/login/Projects/vcpkg/installed/x64-
linux/include" -sICU_PATH="/home/login/Projects/vcpkg/installed/x64-linux" -j1 -sBOOST_ROOT=/home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build -sBOOST_BUILD_PATH=/home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build --debug-configuration --ignore-site-config --hash -q architecture=x86 threading=multi threadapi=pthread debug-symbols=on stage
notice: found boost-build.jam at /home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build/boost-build.jam
notice: loading Boost.Build from /home/login/Projects/vcpkg/installed/x64-linux/tools/boost-build/src/kernel
notice: Site configuration files will be ignored due to the
notice: --ignore-site-config command-line option.
notice: Loading explicitly specified user configuration file:
    /home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/user-config.jam
notice: Searching '/home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg' for user-config configuration file 'user-config.jam'.
notice: Loading user-config configuration file 'user-config.jam' from '/home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg'.
notice: will use '/usr/bin/c++' for gcc, condition <toolset>gcc-5.4.1
notice: using gcc libraries :: <toolset>gcc-5.4.1 :: /usr/bin /usr/lib /usr/lib32 /usr/lib64
notice: using gcc archiver :: <toolset>gcc-5.4.1 :: /usr/bin/ar
notice: using gcc ranlib :: <toolset>gcc-5.4.1 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from /home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/user-config.jam:21
notice: using rc compiler :: <toolset>gcc-5.4.1 :: /usr/bin/as
notice: [python-cfg] Configuring python...
notice: [python-cfg] Checking interpreter command "python"...
notice: [python-cfg] running command 'python -c "from sys import *; print('version=%d.%d\nplatform=%s\nprefix=%s\nexec_prefix=%s\nexecutable=%s' % (version_info[0],version_info[1],platform,prefix,exec_prefix,executable))" 2>&1'
notice: [python-cfg] ...requested configuration matched!
notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python"
notice: [python-cfg]   include path: "/usr/include/python2.7"
notice: [python-cfg]   library path: "/usr/lib/python2.7/config" "/usr/lib"
notice: [python-cfg] Checking for NumPy...
notice: [python-cfg] running command 'python -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())"'
notice: [python-cfg] NumPy enabled
...patience...
...patience...
...found 3113 targets...
...updating 47 targets...
gcc.compile.c++ /home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/boost/build/001ce85aa818da7cc790075190598904/list.o
In file included from ../include/boost/python/detail/prefix.hpp:13,
                 from ../include/boost/python/list.hpp:8,
                 from ../src/list.cpp:5:
../include/boost/python/detail/wrap_python.hpp:57:11: fatal error: pyconfig.h: No such file or directory
   57 | # include <pyconfig.h>
      |           ^~~~~~~~~~~~
compilation terminated.

    "/usr/bin/c++"   -fPIC -g -m64 -pthread -O0 -fno-inline -Wall -g  -DBOOST_ALL_NO_LIB=1 -DBOOST_PYTHON_SOURCE -DBOOST_PYTHON_STATIC_LIB  -I"../include" -I"/home/login/Projects/vcpkg/installed/x64-linux/include" -I"/usr/include/python2.7" -c -o "/home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/boost/build/001ce85aa818da7cc790075190598904/list.o" "../src/list.cpp"

...failed gcc.compile.c++ /home/login/Projects/vcpkg/buildtrees/boost-python/x64-linux-dbg/boost/build/001ce85aa818da7cc790075190598904/list.o...
...failed updating 1 target...
...updated 5 targets...
ninja: build stopped: subcommand failed.

[cenit]

same here, on a wsl-based ubuntu 18.04

[seanfdnn]

Same here.

[JackBoosY]

Reason should be here:
_VCPKG_PATH/ports/boost-python/CONTROL_:

Build-Depends: ..., python3 (!osx&!linux)

[yurybura]

@jepessen Could you try to build with #9780?

[PhoebeHui]

Building boost-python on Linux looks never pass before, It's not a regression issue, also adding the python3 dependency to boost-python doesn't solve this issue.

It seems that building boost-python on windows is different with Linux(or OSX).
On windows, It invoke '.\installed\x64-linux\tools\boost-build\src\tools\python.jam', and 'No working Python interpreter found', so It used configured python on user-config.jam, the python version, includes and libraries path get when build boost-python.

if "boost-python" = "boost-python"
{
    using python : 3.7 : : "F:/VCPKG/clean/vcpkg/installed/x64-windows/include/python3.7" : "F:/VCPKG/clean/vcpkg/installed/x64-windows/lib" ;
    using python : 3.7 : : "F:/VCPKG/clean/vcpkg/installed/x64-windows/include/python3.7" : "F:/VCPKG/clean/vcpkg/installed/x64-windows/debug/lib" : <python-debugging>on ;
}

notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python"
notice: [python-cfg]   include path: "F:/VCPKG/clean/vcpkg/installed/x64-windows/include/python3.7"
notice: [python-cfg]   library path: "F:/VCPKG/clean/vcpkg/installed/x64-windows/lib"
notice: [python-cfg]   DLL search path: "<empty>"

On Linux, it invoke installed\x64-linux\tools\boost-build\src\tools\python.jam", it try to search python2 that system installed

notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python"
notice: [python-cfg]   include path: "/usr/include/python2.7"
notice: [python-cfg]   library path: "/usr/lib/python2.7/config" "/usr/lib"

[cenit]

In any case it's strange it passed CI during upgrade. I want boost installed on Linux and now it's broken.

[cenit]

boost-python came in as an undesired broken dependency

[PhoebeHui]

boost-python passed on CI testing due to 'boost-python:x64-linux=ignore' added to https://github.com/microsoft/vcpkg/blob/master/scripts/ci.baseline.txt#L164

[cenit]

Issue still open 20 days after, and it is valid also on Mac.
For any user of vcpkg not on Windows, installing boost is very troublesome...

[cenit]

Still no solution in sight.

./vcpkg install boost

is broken on any non-windows system... @ras0219-msft

[yurybura]

@cenit I've fixed it in #9780, but review is delayed without any comment.

[cenit]

@yurybura let's hope for the best

[yamcenutzer]

./vcpkg install boost seems to work on Ubuntu 19.10 (gcc 9.2.1) but not on Centos7 (using gcc8 or gcc7)

[yamcenutzer]

vcpkg on centos is anyway broken, since you can't bootstrap...

[JackBoosY]

Close this issue via #9780 merged.

[JackBoosY]

@yamcenutzer Please see #9927 #9955.

[yamcenutzer]

if I allow bootstrap using a patch from 9955 i still can't install boost properly, because now it insists on intalling python3 which fails, although python3 already is installed by the system.

Linux binary incompatibilty across releases ??

[yamcenutzer]

Even after enabling bootstrap e.g. by using the patch in #9955:
still can't install boost on centos7 due to failing python3 install
What's the point?

Give up Centos7 or revert back to state of 12-2019, where everything worked.

[cenit]

Exactly my point in the comment of the PR #9780 itself. Unfortunately it seems that now "boost-python" has been fixed, but in reality it is still depending on a broken python3, which makes impossible to do a "./vcpkg install boost" on linux successfully
@ras0219-msft @JackBoosY

[yamcenutzer]

BTW, On Centos 8 gcc is 8.2.1 whereas on cent07 with devtoolset-8 gcc is 8.3.1-3
On Centos 8 you can bootstrap without applying any patches (noexept seems to work), however,
building python3 still fails.

I only got boost to install completely (incl python...wahtever) on ubuntu 19.10 with gcc 9.2...

[PhoebeHui]

@cenit, @yamcenutzer, thanks for bring this up.

However, I tried to build boost-python and boost on ubuntu 18.04 with latest clean vcpkg source, all built successfully for me.

I don't have centos environment, we have exist https://github.com/microsoft/vcpkg/issues/9955 for tracking the problem.

./vcpkg install boost-python
Building package python3[core]:x64-linux... done
Installing package python3[core]:x64-linux...
Installing package python3[core]:x64-linux... done
Elapsed time for package python3:x64-linux: 6.367 min
Starting package 83/83: boost-python:x64-linux
Building package boost-python[core]:x64-linux...
-- Downloading https://github.com/boostorg/python/archive/boost-1.72.0.tar.gz...
-- Extracting source /home/phoebe/vcpkg/downloads/boostorg-python-boost-1.72.0.t ar.gz
-- Using source at /home/phoebe/vcpkg/buildtrees/boost-python/src/ost-1.72.0-c6a 2496f74
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Packaging headers
-- Packaging headers done
-- Using cached /home/phoebe/vcpkg/downloads/boost_LICENSE_1_0.txt
-- Installing: /home/phoebe/vcpkg/packages/boost-python_x64-linux/share/boost-py thon/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package boost-python[core]:x64-linux... done
Installing package boost-python[core]:x64-linux...
Installing package boost-python[core]:x64-linux... done
Elapsed time for package boost-python:x64-linux: 1.465 min

./vcpkg install boost
Building package boost[core]:x64-linux...
-- Performing post-build validation
-- Performing post-build validation done
Building package boost[core]:x64-linux... done
Installing package boost[core]:x64-linux...
Installing package boost[core]:x64-linux... done
Elapsed time for package boost:x64-linux: 164.3 ms

[cenit]

I cannot build it (python3) in ubuntu 18.04 nor on macOS

[PhoebeHui]

@cenit, do you use wsl?

[PhoebeHui]

I also tried to build python3 with wsl, it built successfully for me.

@cenit, could you creat a new issue for the failures?