Conan: [bug] package_info() env variables not applied when pr:h matches pr:b

[ericriff]

Environment Details (include every applicable attribute)

• Operating System+version: Ubuntu 18.04
• Compiler+version: gcc8
• Conan version: 1.33.1
• Python version: 3.6

Steps to reproduce (Include if Applicable)

-- This may be a bug or me using conan the wrong way --

We had a CI pipeline that crosscompiled a bunch of libraries using the standard, single profile, approach.
Recently we came across its limitations so we changed the workflow to use 2 profiles, the host one and the build one.

Since this is automated, we specify the build profile on a variable and then we list, for each package, the host profile to use.
Something like:

buildProfile: x86_64
jobs:
  zlib:
    1.2.11:
      x86_64:
      arm:
      aarch64:
      mips:
  apriltag:
    1.2.11:
      x86_64:
      arm:
      aarch64:
      mips:

That file gets parsed so we always use x86_64 as the build profile, and the host profile is the result of iterating over the list under the version of a given package.
When we build a package for x86_64, the host and build profile are the same, and I guess that is OK? But some packages fail to build, because the env variables defined in package_info are not visible to the test_package.

As an example, this happens for b2.
This works as expected:

conan create recipes/b2/standard b2/4.2.0@eric/stable -pr x86_64

But this fails

conan create recipes/b2/standard b2/4.2.0@eric/stable -pr:h x86_64 -pr:b x86_64
...
...
b2/4.2.0@eric/stable (test package): Running test()
ERROR: b2/4.2.0@eric/stable (test package): Error in test() method, line 15
        os.environ['...'].replace("\\", "/")+"\" ;"
        KeyError: 'BOOST_BUILD_PATH'

But BOOST_BUILD_PATH is being defined in b2's conanfile

    def package_info(self):
        self.cpp_info.bindirs = ["bin"]
        self.env_info.path = [os.path.join(
            self.package_folder, "bin")] + self.env_info.path
        self.env_info.BOOST_BUILD_PATH = os.path.join(
            self.package_folder, "bin", "b2_src", "src", "kernel")

Is this a bug or I'm not allowed to have matching build and host profiles? I know it doesn't make much sense, but from the automation point of view is super easy for us to list the build profile once (since it is always the same) and then just change the host accordingly.

Thanks!

[solvingj]

I believe there's a check in the code which amounts to "isCrossCompiling" and if that resolves to true, then a number of behaviors are different because the assumption is that the host and build systems will never be the same, and thus many standard behaviors are invalid. For example, assuming the env var BOOST_BUILD_PATH would be the same in a real cross-compile scenario would be an unsafe assumption, so my guess is that environment variables aren't propagated from build to host context.

I'm going to guess that we always set isCrossCompiling to true whenever a user provides separate host and build profiles.

If so, there might be an easy fix here. When resolving this boolean of isCrossCompiling, we might be able to compare host and build profiles for quality. If they are the same, then maybe we could return false.

If this is the situation, then there's a philosophical debate here that some might want to get into. Something like "should we support this case or force users NOT to use -pr:h/-pr:b unless they're cross-compiling. But, lets figure out if my other guesses are right first.

[solvingj]

Here's the code which is used in the codebase to determine if it's a cross_build scenarios (including comparing pr:h and pr:b):

https://github.com/conan-io/conan/blob/develop/conans/client/tools/oss.py#L454

I will need some assistance decrypting this method implementation to determine if I'm even on the right track.

[ericriff]

Thanks for looking into this!
I did some quick testing and, in the particular case of b2, tools.cross_building(self.settings) always resolves to False (even if I actually try to crosscompile, using -pr:b x86_64 -pr:h arm or similar). I think that is because on this recipe we have
settings = "os_build", "arch_build", which makes sense since that is a recipe for a build tool.
So I added the classic settings to the recipe settings = "os", "arch", "compiler", "build_type" which makes tools.cross_building(self.settings) evaluate to True when it should, but the behavior is the same.

[SSE4]

When we build a package for x86_64, the host and build profile are the same, and I guess that is OK?

yes, this is definitely okay. keep in mind, using two profiles is always a different mode, even the profiles are identical.

conan create recipes/b2/standard b2/4.2.0@eric/stable -pr x86_64

this is single profile mode

tools.cross_building(self.settings) always resolves to False

tools.cross_building may incorrectly return False if recipe doesn't define os/arch in the settings attribute (or doesn't define settings attribute at all)
in this case, even if you pass -s arch=x86 in command line, tools.cross_building sees arch=None in your recipe unless you declare something like settings = ("os", "arch")

settings = "os_build", "arch_build"

this is an old model, and two profile approach was designed to go away from os_build/arch_build and their limitations. if you're using two profiles, consider switching from os_build/arch_build to just os/arch in your recipes.

but the behavior is the same.

the problem is with test_package and how it works.
test package automatically inserts magic graph node, and it always a requires relations.
in case of two profiles mode, requires do not propagate their env_info into consumers, that's done only for build_requires. so that's simply test_package limitation, which wasn't designed to test tools or build requires.
more information could be found here: https://github.com/conan-io/conan/issues/7132
so in general, test package was designed to test just library packages, not executables.

[ericriff]

So this is the key

in case of two profiles mode, requires do not propagate their env_info into consumers, that's done only for build_requires

So when using two profiles, how are you supposed to consume a package that provides a library and also some executables (i.e. protobuf)? Should I list it as a requirement AND a build_requirement?

[solvingj]

Very good question. I wonder what the behavior of the graph resolution would be in that case. @memsharded ?

[SSE4]

So this is the key

in case of two profiles mode, requires do not propagate their env_info into consumers, that's done only for build_requires

So when using two profiles, how are you supposed to consume a package that provides a library and also some executables (i.e. protobuf)? Should I list it as a requirement AND a build_requirement?

in this case, you want two nodes, and both requires and build_requires for the same package.
imagine you're cross-compiling to the ARM from x86.
in such case, you want to link with ARM libraries (libprotobuf), but launch x86 protoc executables.
if the same package provides libprotobuf and protoc, it's unavoidable to have two nodes of the same package, but in different contexts.
otherwise, it will either won't link or won't run for you.
for protoc, you want build_requires and context switch to happen, so you get x86 binary
for libprotobuf, you want regular requires and you do not want context switch, so you get ARM binary

[ericriff]

Thanks for clarifying that.
Then about building a tool like b2 using two profiles. That is something that can't be done using current conan behavior, right?
I'm OK with closing this thread if that is the conclusion.

[SSE4]

Thanks for clarifying that.
Then about building a tool like b2 using two profiles. That is something that can't be done using current conan behavior, right?
I'm OK with closing this thread if that is the conclusion.

the b2 itself builds, and it can be successfully consumed via build_requires (in two profiles mode).
the problem with test_package which fails.
the entire test_package feature was designed long before we had any support for the cross-building, build_requires and two profiles. it was designed simply for testing regular library packages only, and can handle only that purpose.

however, in conan, there are many different relations between nodes in graphs:

• requirements (default and most wide-spread)
• private requirements
• build requirements
• python requirements
• consuming generators

test package only handles the first one (requirements). you probably should either completely skip test package for your case, or add build_requires node on your own (I am not sure it will work).

but my understanding is test_package is just wrong tool for build_requires - it's like a using saw for nails instead of a hammer :)

[ericriff]

Just for the sake of testing, I added this to the test_package conanfile

def build_requirements(self):
    self.build_requires("b2/{}@{}/{}".format(self.requires["b2"].ref.version, self.requires["b2"].ref.user, self.requires["b2"].ref.channel))

And it passes now, with one profile and with two :)