Conan: Issues with build_requirements

[theodelrieu]

Hello,

During a discussion we had yesterday with @lasote, he advised me to open a Github issue about what we talked about.

I'll try to explain the problems I see first, then some leads that I think Conan could follow in order to solve them.

Build requirements

There are two main categories of build_requirements:

1. Build tools
2. Toolchains

I will focus on the latter.

Toolchains

In Conan, toolchain packages usually define a bunch of options and flags that will change how packages are built.

For example, android-toolchain specify how to cross-build for Android, depending on settings (e.g. os.api_level) and options (e.g. use_system_python).
We could imagine adding an option link_static_libcxx to toggle static linking against standard C++ libraries.

The first issue arise when building the same package twice, with a different build_requirement option:

$ conan create . theo/test -o android-toolchain:link_static_libcxx=True
$ conan create . theo/test -o android-toolchain:link_static_libcxx=False

Surprisingly, both packages will have the same package ID, therefore a single package is stored.

Package ID

As we saw before, build requirements have no effect on their consumer's package ID.

This holds true even if the binary package changes (e.g. by using an optimization option).

This behavior is very error-prone. I personally had my share of nasty surprises when updating toolchain recipes when cross-building.

The main question here is:

• How could Conan know which build_requirement option should modify consumer's package ids?

Maybe having a self.deps_info attribute?
I know that @memsharded is working actively on the internal dependency graph calculation, I'm sure he will have way better insights than me about it.

What's the big deal?

One has no idea how a package was exactly built.

Here's what's available in the conaninfo.txt:

• Settings
• Package and runtime deps options
• Recipe hash
• Environment variables

I don't believe it's enough, but more on that later.

Private use-case

Well, for my company, it's not that much of an issue. For a simple reason: we keep our packages in-house.

But what would happen if we wanted to release them?

Of course, we could release these packages alongside the toolchain package with our own user/channel. Then, every package with user tanker would be built with android-toolchain/X@tanker/stable.

Not optimal, but alright for the moment.

Conan-center, bincrafters, and other public repositories

Apparently, there are no binary package available on conan-center, which were built with a build_requirement toolchain.

But you can imagine the pain that package id collision would cause on a public repository. Right now, if one want to download Android packages, they have to build everything from sources.

Custom flags

I've talked about toolchains, but the problem is not limited to them.

I think we should think about custom user options (or settings maybe? I'm not sure yet), rather than toolchain packages.

The latter could well be implemented by using those custom user options/settings.

There are so much similarities with an existing project that I will start the comparison from now on...

Gentoo USE flags

Quick disclaimer: I'm not a Gentoo user, I've tried it two days so my understanding is limited.

Here is the Gentoo wiki page about USE flags, I won't describe them in this issue.

I think there is a lot of similarities between Conan and Gentoo's portage.

Do you think having a similar behavior is a good idea for Conan? Getting in touch with Gentoo maintainers/packagers could bring lots of info about all this.

I'll stop writing this initial post, I'm looking forward to discussing this huge topic :)

[maikelvdh]

I am exactly in the need of similar traceability of the toolchain information on the published packages, which I believe is a very important challenge to be solved specially if you want to support decently cross compilation. Is there already any further exploration done how to solve this other then the suggested Gentoo USE flags?

[KristianJerpetjon]

As a gentoo user..
I totally agree on the flags.
I would like to see a package expose it's flags sort of the way you now do settings but with flags ="-fpic","-threads" etc however.. if not set my the profile unlike settings the flag is just ignored.
however different flags yield different hashes. This makes potential stripped down or altered versions of the package managable.. i would also suggest these flags to come from the profile or to be added on command line during conan create

[Minimonium]

Could such flag dictionary settings generate toolchains for build systems too? So one could integrate the whole build process into the conanfile.py.
As I mentioned in the Slack I have a feeling that link_static_libcxx option is a bit backwards, since I feel it should be an option of the standard lib requirement. Not sure about it.

[theodelrieu]

So one could integrate the whole build process into the conanfile.py.

I'm not sure what you mean by that, could you post a small example?

I have a feeling that link_static_libcxx option is a bit backwards

Android is a special case, since you have both versions of the libcxx in the NDK. Otherwise, I'd agree if you had a libc++ package, it should define static/shared as an option.

[Minimonium]

@theodelrieu
I mean to generate toolchains for appropriate build helpers for any dependency one want to build automatically from Conan settings (flag dictionary). Force rewrite flags and such. Possibly can decypher existing toolchain files to the Conan format too via additional helpers.
So the dev process would be to have all abi breaking (toolchain) information in the settings which would propagate to any build helper any dependency is using. And the developer would write down conan build without worrying much.

[lasote]

Reading this issue again I have some thoughts:

• The "requirements" model of conan is not complete, This is a good example of a build_require that SHOULD affect the package ID of the packages depending on it. Currently, this is not possible. This is not the only flaw we have detected, it looks like the requirements model should allow you to configure how your requirements behave:

  • Affects the ID or the dependent packages?
  • Only for building? (do not retrieve unless the dependent packages need to be built)
  • Propagates the info further from to the direct requirer?
  • Allowed to have different versions of the same build_require in the same graph? (conflicts?)

In this case, your android toolchain:
- Should affect the ID of the dependent packages.
- Should be only for building
- Should propagate only the info to the direct dependent packages.
- Should conflict if two versions of the toolchain are specified? Probably yes.

Another different example: the cmake_installer package:
- Should not affect the ID of the dependent package. mostly not, but it depends, a different CMake could apply different flags.
- Should be only for building
- Should propagate only the info to the direct dependent packages.
- Should conflict if two versions of cmake are specified? Probably not.

So apparently the "require" should be more flexible, instead of fixed concepts like "requires", "build_requires" or "private" maybe they should have properties.

• The "cross build model" model of conan is not complete either, is in the roadmap to be improved before Conan 2.0
• I don't see how the Gentoo USE flags could help here. The conan model is different and based on different things, how would you apply that concept to Conan?

[theodelrieu]

Should not affect the ID of the dependent package. mostly not, but it depends, a different CMake could apply different flags.

If CMake introduces a new policy (e.g. CMP0042), who is responsible to take the new behavior into account and reflect it via a Conan option, flag, or something alike?

I don't see how the Gentoo USE flags could help here.

In the case mentioned just above, the cmake_installer might set a global flag rpath on Darwin platforms, which dependent packages could override locally if they want to.

I have no idea if such a thing is desirable, it looks very user unfriendly... Again, I'm not used to that USE flags system, so my opinion does not weigh much.

The "requirements" model of conan is not complete

Correct me if I'm wrong, but I think that requirements' versions do not modify the dependent's package id as well. Same thing when overriding a requirement.

[Minimonium]

Correct me if I'm wrong, but I think that requirements' versions do not modify the dependent's package id as well. Same thing when overriding a requirement.

Isn't it decided by https://docs.conan.io/en/latest/creating_packages/define_abi_compatibility.html#versioning-schema ?

[theodelrieu]

I was not even aware those existed, thanks!

[solvingj]

@lasote is there any update on a plan for this? I don't mean to rush, I just am curious if you've figured out a feasible approach. It does seem like a pretty big challenge to me.

[lasote]

Roadmap for Conan 2.0, but will be challenging

[grisumbras]

I think, the list of properties of a dependency can be reduced to 2:

• only used for building consumer;
• a change in dependency's configuration may affect consumers in a visible way;

The former is requirements/build_requirements separation. The latter is private/public dependency. This covers the first 3 properties.

The fourth (whether to conflict when several different variants of the package are required) seems to me to be more relevant to cross-compilation. For example, if a build system's variant should not affect the built package in a visible way, then there's not much reason to allow multiple variants in a single environment. If several variants of a package may lead to errors (for example, ODR violations), then there's even a reason to forbid multiple variants of it in a single environment. But when 2 variants of a package have different host/target platforms, then they cannot actually be used simultaneously, so they should not conflict. And finally, some projects (GCC, ICU) require a native variant of itself to be cross-compiled (or to create cross-compiling variant). So, maybe the solution here is to allow several variants of a package if all of them use different platform combinations?