Conan-center-index: [package] boost/1.72.0: dependency 'b2' not found when cross compiling

[mheyse]

Summary

When cross compiling boost the conan build system does not find the build_requires dependency b2. b2 does get built with the correct compiler and works but the boost recipe can not access the dependency - specifically, the key b2 does not exist in self.deps_cpp_info and the compilation process aborts with this error message:

ERROR: boost/1.72.0: Error in build() method, line 408
    full_command = "%s %s" % (self._b2_exe, b2_flags)
while calling '_b2_exe', line 340
    return os.path.join(self.deps_cpp_info["b2"].rootpath, "bin", b2_exe)
    KeyError: 'b2'

This has been tested with two different cross toolchains for different target systems.

Package and Environment Details

• Package Name/Version: boost/1.72.0
• Operating System+version: Linux Ubuntu 19.10
• 'build' context compiler: gcc 9.2.1
• 'host' context compilers: x86_64-w64-mingw32-gcc 9.2-win32 and x86_64-apple-darwin19-clang 9.0.0-2 (from https://github.com/tpoechtrager/osxcross)
• Conan version: conan 1.24.0
• Python version: Python 3.7.5

Conan profiles

Configuration for profile default (the build profile):

[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=gcc
compiler.version=9.2
compiler.libcxx=libstdc++
build_type=Release
[options]
[build_requires]
[env]

Configuration for profile x86_64-w64-mingw32 (one of the tested host profiles):

[settings]
os=Windows
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=gcc
compiler.version=9.2
compiler.libcxx=libstdc++
build_type=Release
[options]
[build_requires]
[env]
CC=x86_64-w64-mingw32-gcc
CXX=x86_64-w64-mingw32-g++

Configuration for profile x86_64-apple-darwin19 (one of the tested host profiles):

[settings]
os=Macos
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=clang
compiler.version=9
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]
CC=x86_64-apple-darwin19-clang
CXX=x86_64-apple-darwin19-clang++
SDKROOT=/opt/osxcross/SDK/MacOSX10.15.sdk

Steps to reproduce

• Install conan-1.24.0 and above profiles.

• Create a simple conanfile.txt in an otherwise empty directory:

[generators]
cmake

[requires]
boost/1.72.0

• Build and install boost with the following command:

conan install . --profile:host x86_64-w64-mingw32 --profile:build default --build

Logs

boost-build.log
conan_trace.log
conanfile.txt

[mheyse]

The attached boost-build.log was missing the stderr output, here a complete version.
boost-build.log

[uilianries]

Hi @mheyse indeed we can't access "b2", but using the old approach should work:

conan install . -pr x86_64-w64-mingw32 --build

Let's see what the x-building sorcerer @jgsogo can tell to us.

@jgsogo How can we access "b2" path from the build profile? Could you help us?

[jgsogo]

Hi! It is great to know that someone is experimenting with the new cross building model, receiving feedback about this kind of features in real scenarios is very important for us 🙌

Using the new approach with the host and build profiles, it changes a little bit how things are propagated to consumers. Important bits here (we are interested in the else branch):

if not using_build_profile:  # Do not touch anything
    conan_file.deps_user_info[n.ref.name] = n.conanfile.user_info
    conan_file.deps_cpp_info.update(n.conanfile._conan_dep_cpp_info, n.ref.name)
    conan_file.deps_env_info.update(n.conanfile.env_info, n.ref.name)
else:
    if n in transitive or n in br_host:
        conan_file.deps_cpp_info.update(n.conanfile._conan_dep_cpp_info, n.ref.name)
    else:
        env_info = EnvInfo()
        env_info._values_ = n.conanfile.env_info._values_.copy()
        # Add cpp_info.bin_paths/lib_paths to env_info (it is needed for runtime)
        env_info.DYLD_LIBRARY_PATH.extend(n.conanfile._conan_dep_cpp_info.lib_paths)
        env_info.DYLD_LIBRARY_PATH.extend(n.conanfile._conan_dep_cpp_info.framework_paths)
        env_info.LD_LIBRARY_PATH.extend(n.conanfile._conan_dep_cpp_info.lib_paths)
        env_info.PATH.extend(n.conanfile._conan_dep_cpp_info.bin_paths)
        conan_file.deps_env_info.update(env_info, n.ref.name)

Only requires and build_requires from the host context are propagated to the deps_cpp_info object (it contains compiler flags, libraries to link,... ) and information from the build_requires (build context) is available only as environment, so you can run these applications, but they are not supposed to be linked.

---

Back to ConanCenter: right now we don't have a dedicated effort to migrate all the recipes in ConanCenter to the new model because we need feedback about the feature, maybe we have broken something and we don't know yet... although I'm starting to feel more comfortable about it, it's been out for a month and a half now and there aren't issues related to it 😃

The boost recipe in the new model should have just a

self.run("b2 ....", run_environment=True)

and the executable should be in the path. Right now, we can add some if/else to the recipe, or maybe use the tools.which to know if b2 is available in the PATH.

---

Notes.-

• We cannot know if b2 is coming from build_requires or it is already installed in the system. Is it important? Can we do it somehow?
• tools.which(), should it take a run_environment argument?

[mheyse]

conan install . -pr x86_64-w64-mingw32 --build

I had tried this, it fails when building b2. Note: I've never used conan before, so I dont' really understand it so it might be entirely possible that I'm doing something wrong here, but if I use the conanfile.txt from above and the -pr x86_64-w64-mingw32 option the build fails with

b2/4.2.0: Calling build()
A C++11 capable compiler is required for building the B2 engine.
Toolset 'cxx' does not appear to support C++11.

I assume this means that b2 is being built with the cross compiler, not the build compiler but that's just a guess - I don't know how to log the actual commands executed by build.sh.

Log attached.

build-boost.log

[uilianries]

We cannot know if b2 is coming from build_requires or it is already installed in the system. Is it important?

Yes, it's important. We discussed about avoiding system binaries, because we can't their version/behavior. It would be nice a built-in property to validate if we are running with dual profile or cross-building with the old approach.

[SSE4]

@jgsogo we may still need to get the path to the tool, e.g. to specify on command line or to write to some config file used on build system input. thus, run_environment is not always a sufficient. how to achieve that with new model?
example:

./configure AS=`here goes path to my YASM executable`

[jgsogo]

I'm moving this conversation about the paths to slack, let's keep this issue for the original topic.

---

Using

conan install . -pr x86_64-w64-mingw32 --build

will tell Conan to build everything using the profile in x86_64-w64-mingw32. As it is one of the host profiles, it doesn't make sense to build b2 using that compiler as it won't be able to run in the build_machine.

---

With the old approach, people need to run Conan twice: first time to build the build_requirements and another one to build the actual packages for host:

1. You would use the profile for the _build_machine_: conan install b2/version@ --profile=<your build profile>. Now you will have binaries for b2 that can run in your build machine. You can use a virtualrunenv generator to activate an environment (add the PATHs) and check they work
2. Now you can build your application using the b2 you have just built and it is declared as build_requires in the boost recipe:

conan install boost/version@ --profile=x86_64-w64-mingw32

boost will take some settings from that profile: os, arch, compiler, while b2 is declared to use os_build and arch_build.

This is the old graph approach.

[mheyse]

This works, thank you for the clarification!

So for my part unless you still want to continue the discussion about the new approach here this issue can be closed.

Thanks!

[jgsogo]

Great! Thanks a lot for your issue, it originated a really interesting conversation (and things we need to improve in Conan).

And kudos to the people working on recipes, they are the ones making sense of Conan. 🎉