Conan: [bug] Dependencies' build_policy overwrites command line parameter

[FalkorX]

We recently encountered an issue with our CI system, when incorrectly configured dependency packages were rebuilt in a CI step, where we didn't allow it via --build=never.

Environment Details

• Operating System+version: Windows 10
• Conan version: 1.22.0 and 1.23.0
• Python version: 3.6.4

Steps to reproduce

a/conanfile.py

from conans import ConanFile

class TestA(ConanFile):
  name = "TestA"
  version = "1.0"
  requires = "TestB/1.0@test/test"

  def build(self):
    self.output.info("Build TestA")

b/conanfile.py

from conans import ConanFile

class TestB(ConanFile):
  name = "TestB"
  version = "1.0"
  build_policy = "missing" # !!!

  def build(self):
    self.output.info("Build TestB")

1. Create packages TestA and TestB
   conan create b test/test conan create a test/test
2. Remove package TestB and reexport the conanfile
   conan remove TestB conan export b test/test
3. Install TestA without build
   conan install TestA/1.0@test/test --build=never

Expected behavior:

Install command should fail, since TestB is not installed and --build=never is specified.

Actual behavior:

Install command succeeds and TestB is rebuilt (see log).

Remark: The same bug occurs when setting build_policy=always in b/conanfile.py. You can then skip step 2 and specify --build=missing in step 3, then TestB will be rebuilt although it shouldn't.

Logs

Log for step 3:

PS E:\Workspace\conan_bug> conan install TestA/1.0@test/test --build=never
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=15
os=Windows
os_build=Windows
[options]
[build_requires]
[env]

TestB/1.0@test/test: Building package from source as defined by build_policy='missing'
Installing package: TestA/1.0@test/test
Requirements
    TestA/1.0@test/test from local cache - Cache
    TestB/1.0@test/test from local cache - Cache
Packages
    TestA/1.0@test/test:5c07eea89709163d6f80759ede846381d3745442 - Cache
    TestB/1.0@test/test:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Build

Installing (downloading, building) binaries...
TestB/1.0@test/test: Configuring sources in C:\Users\Oliver\.conan\data\TestB\1.0\test\test\source
TestB/1.0@test/test: Copying sources to build folder
TestB/1.0@test/test: Building your package in C:\Users\Oliver\.conan\data\TestB\1.0\test\test\build\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
TestB/1.0@test/test: Generator txt created conanbuildinfo.txt
TestB/1.0@test/test: Calling build()
TestB/1.0@test/test: Build TestB
TestB/1.0@test/test: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' built
TestB/1.0@test/test: Build folder C:\Users\Oliver\.conan\data\TestB\1.0\test\test\build\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
TestB/1.0@test/test: Generated conaninfo.txt
TestB/1.0@test/test: Generated conanbuildinfo.txt
TestB/1.0@test/test: Generating the package
TestB/1.0@test/test: Package folder C:\Users\Oliver\.conan\data\TestB\1.0\test\test\package\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
TestB/1.0@test/test: Calling package()
TestB/1.0@test/test: WARN: This conanfile has no package step
TestB/1.0@test/test package(): WARN: No files in this package!
TestB/1.0@test/test: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' created
TestB/1.0@test/test: Created package revision 4d0ed6b951018d8af295b25c8922813c
TestA/1.0@test/test: Already installed!

[memsharded]

Hi @FalkorX

Thanks for your detailed report.

This behavior is by design, not a bug. The build_policy on recipes was a mechanism requested by users to allow some recipes to build from sources irrespective of the --build argument of the command line. The default behavior of conan install is already never to build packages. This is documented in the docs, and in the command line help (see last lines):

-b [BUILD], --build [BUILD]
                            Optional, use it to choose if you want to build from
                            sources: --build Build all from sources, do not use
                            binary packages. --build=never Never build, use binary
                            packages or fail if a binary package is not found.
                            --build=missing Build from code if a binary package is
                            not found. --build=cascade Will build from code all
                            the nodes with some dependency being built (for any
                            reason). Can be used together with any other build
                            policy. Useful to make sure that any new change
                            introduced in a dependency is incorporated by building
                            again the package. --build=outdated Build from code if
                            the binary is not built with the current recipe or
                            when missing binary package. --build=[pattern] Build
                            always these packages from source, but never build the
                            others. Allows multiple --build parameters. 'pattern'
                            is a fnmatch file pattern of a package reference.
                            Default behavior: If you don't specify anything, it
                            will be similar to '--build=never', but package
                            recipes can override it with their 'build_policy'
                            attribute in the conanfile.py.

[KerstinKeller]

Hi, are you intending to keep this behavior for Conan 2.0?

From a design point of view, I find it weird, that a recipe can control a build policy, and then that it even has precedence over manually specified values.

I feel that that is a property that should come from the outside, when putting all libraries together. It's like when a CMakeLists.txt file defines its own C++ standard to use, even though this should be an input from the outside, depending on the compiler.

I would be interested to hear where users really need this feature. From my point of view it would be more natural to have it, e.g. in a profile, though I am not sure if build_policy is allowed there.

[memsharded]

Hi, are you intending to keep this behavior for Conan 2.0?

We are considering removing these policies for Conan 2.0, or at least redefine their behavior.

The use cases are for example: a driver, that needs to be built from sources always in the target machine. Only the recipe is uploaded, but it is a bit annoying to always fail when users do install them (maybe even transitively, they are not aware of this driver). Having it in the profile is still on the consumer side, they will still fail, realize they need to build that, add it to their profiles, so a bit inconvenient too. But might be more explicit, and then more aligned with general conan design.

Lets annotate this for Conan 2.0 review.

[FalkorX]

Hi @memsharded

Thanks for your quick answer.

I read this part in the docs and still didn't fully understand it. The part

--build=never Never build, use binary
packages or fail if a binary package is not found.

makes it seem like --build=never should fail if _any_ of the packages has to be built, irrespective of their build_policy.

And the part

Default behavior: If you don't specify anything, it
will be similar to '--build=never', but package
recipes can override it with their 'build_policy'
attribute in the conanfile.py.

makes it seem like the build_policy only affects builds if the --build parameter is not set. It sounds like the command line parameter always takes precedence over the conanfile's build_policy.

Maybe we can change the wording of this section in the docs to reflect that behavior a bit better?

[memsharded]

Maybe we can change the wording of this section in the docs to reflect that behavior a bit better?

Yes, certainly this deserves some better wording in the command line help and docs, you are right. This would be an easy contribution, would you like to submit a fix to the command line help yourself, as first step? (docs can be updated later)

[FalkorX]

Yes, certainly this deserves some better wording in the command line help and docs, you are right. This would be an easy contribution, would you like to submit a fix to the command line help yourself, as first step? (docs can be updated later)

Yes, I can do that. I have never contributed before though, do I just stick to the rules in the contributing guide? Or is there anything else I should pay attention to?

[memsharded]

Yes, this one is very low risk, so those rules should be more than enough. I don't think tests are necessary for this either. Thanks!

[memsharded]

Reopening this, so we discuss about the build_policy in recipe in 2.0