Conan: Strategy for addressing common USER and CHANNEL challenges

[solvingj]

This is a suggestion of a concept to solve a group of related scaling challenges I've hit in both OSS and enterprise use cases.

At my company, we have a requirement to compile all third party libs from source. The adherence to this is so stringent, that we remove Conan Center from our remotes in our build containers. Of course, we still want to make use of the Bincrafters and Conan Community recipes for building the software.

So, for example, openssl and zlib.

I use "Bitbucket Import" to bring in conan-openssl and conan-zlib to our source control.

I add a Jenkinsfile, and replace the build.py with our own custom. Occasionally, we'll patch a source.
When Jenkins does our builds, the packages are create with OUR conan username:

Example: zlib/1.2.11@mycompany/stable

Jenkins then uploads this package to our repository.

Now we do the same for OpenSSL. However, of course it still references zlib/1.2.11@conan/stable.

So, we have two choices. Patch the conan-openssl recipe, or override it in the downstream package which consumes openssl.

Unfortunately, both of these are really rough for us. Patching is no good, because we're going to be using about 40 of the bincrafters/conan recipes. We would have to build and then patch our way up the dependency tree, one-by-one. As for overriding the dependencies in the consuming recipes, we have roughly 20-25 "final product" recipes that go into our installers. Separate executables. Each of these recipes would have to set the overrides on every single transitive dependency it uses. It almost defeats the purpose of having a transitive dependency tree to begin with.

There is a similar problem with managing references to channel ( testing vs stable). This one comes up very frequently when users are trying to fix or upgrade some intermediate dependency, so they push their changes to the testing channel package. Now, they just want to run a build or locally execute a stable channel of some consuming package against the testing channel of this one dependency. There are two possible solutions here. First, maintain testing channel on all your packages, and in the conanfile.py on all your testing branches, reference the testing channel of all your dependencies. This is not fun because you can't really safely go on merging testing into stable the normal way without overwriting all your references. Second, parameterize all references in all recipes with user and channel variables, which you can set at runtime with environment variables or something. This is again undesirable for a few reasons... it's a lot of clever engineering, it basically forces all-dependencies-one-channel. Despite the drawbacks, I know there are enterprise people who do this today just to have a testing branch strategy that kind of works.

The alternative mechanism I came up which addresses both reference segments in a way that is usable during both local Conan executions, AND continuous integration builds is shown below:

conan create . some/thing -u "zlib:mycompany" -u "openssl:mycompany" -c "openssl:testing"

or, using the wildcard syntax, if the whole dependency tree is uniform, as it likely is in most enterprise codebases:

conan create . some/thing -u "*:mycompany" -c "*:testing"

And then in Conan Profiles, i'm not sure what section to put it in, or what to call a new section for these things if that is what is called for:

[user_overrides]
zlib:mycompany
openssl:mycompany
[channel_overrides]
openssl:testing

I think the concept is self-explanatory because it mimics the control of options and settings on a per-package basis at build time. In theory, this could be implemented without any breaking changes because it's a brand-new feature. As usual, with any "idea" for a new feature, there could be unforeseen implications which make this completely infeasible. Thus, feedback welcome.

• [X] I've read the CONTRIBUTING guide.
• [X] I've specified the Conan version, operating system version and any tool that can be relevant.
• [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[theodelrieu]

I'm in favor of a solution like this, the only thing I'm afraid of is the confusion caused by the override itself.

This is an issue I already have with the current override feature though, because what you see if not necessarily what you get, i.e. you have to look at the final conaninfo.txt to get the real information.

Let's say you want to override one of your dependencies' requirement on Boost/1.66.0@user/stable like this:

# top-level conanfile.py

def requirements(self):
    self.requires("Boost/1.67.0@user/stable", override=True)

Then you put the following in your profile:

[user_overrides]
Boost:bincrafters

[channel_overrides]
Boost:testing

That's what I call confusing... I do not have a solution in mind right now, it might require some thoughts on the override feature in the first place.

[keithrob91]

I am also in favour of a feature like this. Personally though, I don't like the profile part of the solution, but I do like the command line arguments bit!

Then you put the following in your profile:

Doesn't the @solvingj solution mean that you no longer need the override in the top-level recipe? Just the command line or profile settings are required?

[theodelrieu]

Doesn't the @solvingj solution mean that you no longer need the override in the top-level recipe?

Indeed, I think it makes sense that less things are done in the recipe. Just like for cross-building, you do not want to handle that in every recipe out there, but rather using something like a build requirement.

This could be a part of a bigger discussion on how to write recipes reusable in different contexts. i.e. writing recipes in a quite generic way to allow external packages/profiles provide the correct flags for specific use-cases.

For instance, I would like to never see cross-building related code in a recipe, it would make my life easier :smile:

[solvingj]

@keithrob91 : preferences are fine. It's simply natural for profiles to support all the same package-related parameters that the CLI does. Also, regarding overrides, indeed, the goal is to make override-in-recipe no-longer required, but provide an alternative mechanism to achieve the same.

Profile/Recipe/CLI... each of the various places you can define variables are useful in different scenarios. I'm glad we have multiple.

@theodelrieu : you certainly have more cross-building experience than I, so it's interesting to hear your insight about how recipes with cross-building logic make your life more difficult. Are you encountering OSS recipes with this logic? Or are these ones others have made in your company?

[solvingj]

@theodelrieu also a good point about multiple places to override. This reminds me to request a "Bill of Materials" hook which pretty-prints something similar to conaninfo.txt, which would be very useful for CI logging.

[keithrob91]

My problem with some of this stuff being set in a profile is that it is hidden and not always obvious what settings are being used. This is normally a problem when something isn't working as expected and you're trying to find out why.

But if you like them, fine!

[solvingj]

It's not that i like them, it's that they're far more practical in guaranteeing consistent settings and options across teams and CI builds together. However, to your point, the latest version of Conan seems to print out a nice "reconciled" list of settings and options by default. Not sure if you've seen this, or if it's comprehensive.

[theodelrieu]

@solvingj

Are you encountering OSS recipes with this logic?

Mostly Autotools-based libraries, the worst being OpenSSL with no surprise.

I'd like to see recipes only dealing with specific stuff (e.g. if building for Android, add a specific file to compile), but leave more general things (e.g. sysroot and flag handling) to someone else.

It usually goes smoothly with CMake based libraries, although I have to use tools.patch from time to time...

[uilianries]

You touched on an old problem I had in my previous job. I liked your way to think and also I thought about taking advantage of the solution adding one option to override an entire dependency:

conan create . some/thing --override "zlib/1.2.11@mycompany/testing"

[solvingj]

Your strategy has the advantage of allowing the caller to override the version number, which mine does not. Pretty significant. Yours might also support wildcard: *@mycompany/testing. I think I would accept either syntax.

I think the general realization here is that user and channel fields of dependencies are new and useful abstractions that Conan seems to have invented, however several of us are finding that we want to control over these fields for the entire dependency tree as the caller of the Conan command, which is the feature that Conan currently does not provide.

Perhaps @lasote or @memsharded can comment as to whether or not they agree or disagrees with the general principle.

[lasote]

So, let's try to summarize, we need a mechanism to override user/channel in a dependency tree:

• We don't want to change the recipes, they will continue having "fixed" requirements to the transitive deps.
• We don't want to modify the conanfile.py from our project to override dependencies. Also, if the "project" is a future conan package, introducing in the recipe the overrides would violate the first point.
• We want to do it from a profile, so we can change the dependency tree (testing/stable) applying a different one.
• We want wildcards, to change globally N deps: *:user/channel
• We want to override only the channel or only the user but also both.
• We may want a way to introduce regular dependency overrides from a profile. (not currently possible)

Let's think the feature looks like this (not even a real proposition, but useful to disscuss possible flaws):

myprofile.txt

[overrides]
zlib/1.2.8@[user]/[channel] # Regular override (not related with this feature request) of the zlib library to use the 1.2.8 with the same user and channel
zlib/[version]@my_user/my_channel # Will override any zlib version to use my_user/my_channel
*/[version]@my_user/[channel] # Will override all the packages to use `my_user`

Let's start the discussion!

[keithrob91]

@lasote Not sure if it is implicit in what you said above, but I would also want this sort of capability available from the command line, not just from a profile.

[solvingj]

I just had a thought, that another interesting reason one might want to add a "regular dependency" from a profile is to "quasi-override" uses of system_requires. For example, a recipe might do a system_requires for libusb when on Linux. In time, a Conan package for libusb might appear on Conan Center, or on an internal Artifactory server with custom patches for the organization. While not technically an "override", if the libusb Conan package is "injected" into the dependency graph from a profile, it could be applied to any number of recipes that have system_requires. This would seem to provides nice migration path away from System Requires where possible, as more and more "system-like" dependencies get Conan recipes/packages.

[lasote]

I think that is a different use case/issue. It could work in the other way, if a recipe has a requirement for a Conan package libusb, you could override it with one "proxy" package that only appends the cpp_info.libs = ["usb"] to make it work against a system package.

Otherwise, the issue would be very different, you would be injecting dependencies to a specific node, I know that the system requirements could be "the plug" that allows that kind of override but indeed it would be a very different issue.

Maybe the KIS way to achieve this is to recommend creating always a proxy package when using the system requires. The recommendation would be:

1) Create a recipe for the system requirements
2) In the recipe, call the system_requirements to install the needed stuff
3) declare the needed libs in the cppinfo

That way you could always override them.

[solvingj]

Oh, that's a very interesting idea.

[monsdar]

The situation @solvingj described initially is ours as well. Though we're altering the recipes to not download from the internet but use an internal server for that. So not only do we change the dependencies, but we also overwrite the source() method.

A feature that makes this easier would greatly help. I like the proposed solutions.

[lasote]

The source() method overwriting is also part of this discussion here: https://github.com/conan-io/conan/issues/4734
We don't have yet any idea to mitigate that.