Conan: Override dependency versions from command line

[rweickelt]

It would be great if we could override requirements versions of a Conan recipe from command line during conan install or conan create.

Use case: Distributed CI build environment that builds package A from source and wants to build dependent packages B and C in order to test changes in A. Packages are shared via an internal artifactory server.

Right now we are

• putting the commit hash or branch name into the version when building A, for instance A/1.13.0+COMMITHASH@ci/testing
• then replace requirement lines in dependent conan recipes with sed on-the-fly before building those packages
• cleaning up the ci userspace on artifactory from time to time

It would be less error-prone if we could omit the sed step.

[uilianries]

Hi @rweickelt!

You update your profile to include your specific requirement:

https://github.com/conan-io/conan/issues/4785#issuecomment-474961686

[boussaffawalid]

+1 This will be very useful to test dependency changes in ci without editing the conanfile.py

[Minimonium]

@uilianries I'm not sure I follow. Comment in your link explains the way to override settings.
But here is the way to extract overrides of requirements out of the recipe. I'd also consider a profile field for that.

[uilianries]

@Minimonium I see. I thought using profile to override the requirement, but it's only possible for build_requirements.

I would say it could be a feature request, since I saw the same question on Slack more than once.

@lasote Do you think is it possible accepting this feature request?

[lasote]

I think the correct way to achieve this is using package revisions and not managing dev versions in the reference. Probably managing different Artifactory repositories for the development cycle too. We are working hard right now to define a correct workflow for CI but it will take some time.

[Minimonium]

@lasote How would package revisions help when the reference is a different one?

Having a different Artifactory repository also assumes that we have the same reference (but from a different source), while for many situations you usually have separate references (user/testing) that you want to plug into your recipe tree for local development (or CI to check stuff before merging).

You would maybe use something with editable packages, but again, it'd use the same reference (or you need to edit the recipe), which is not the most correct workflow.

Ideally, recipes should not be changed often and expose functionality in such a way to allow users to inject it, meaning cleaner recipes.

[boussaffawalid]

My current solution to achieve this was to use a custom requires method.
basically you can override the version of a package XXX using the env var CUSTOM_CONAN_XXX_VERSION

def my_requires(self, package, private=False, override=False):
    package_name = package.split('/')[0]
    package_override_env_var = 'CUSTOM_CONAN_{}_VERSION'.format(package_name.upper())
    package_override = os.environ.get(package_override_env_var)
    if package_override is not None:
        self.output.warn('Environment var {} is set!! Overriding {} by {}'.format(package_override_env_var, package, package_override))
        package = package_override
        if not re.match(r"^(\w+)/(\S+)@(\w+)/(\S+)$", package_override):
            self.output.error("malformed package name {}".format(package_override))
            sys.exit(1)

    self.requires(package, private=private, override=override)

[lasote]

@lasote How would package revisions help when the reference is a different one?

Not changing the reference. If the reference has changed because of "temporary versioning" reasons.
Changing "user/channel" is not the recommended practice. It is better to promote packages between repositories.

[fulara]

It would also be beneficial in my use case.

It is a complicated world we are living in.

We have a packages that we know are compiled with different libraries, we know that beforehand.
And we also store prebuilt binaries inside our artifactory.

This would allow us to prebuild these dependency with a matrix of dependencies beforehand. rather than relying on --build missing.

Its actually kinda lacking that conan has the capability to do so, (when you override some dependency in the package upstream ) but doesnt expose it to the users.

i would imagine something like this would work for me:
conan create --requires=boost/1.71.0 .
to override the conanfiles requires.

We have implemented somewhat similar solution similar solution to what boussaffawalid above wrote.

[boussaffawalid]

The solution I proposed above works but it has one side effect.
I have to always define the env variable CUSTOM_CONAN_XXX_VERSION before consuming my package, otherwise the override wont work.

Do you recommend anyway to save the state of the override ? maybe the package itself can define the env variable CUSTOM_CONAN_XXX_VERSION

[KerstinKeller]

Is there any news on this feature? I'd really appreciate it. Often packages can work with [>=] references, but then the CI will prebuild the package for a given set of supported dependency versions.

[KerstinKeller]

So I tried to add just an option to my package for the requirement version, and that seems to work well:

class MyTest(ConanFile):
  options = {"protobuf_version": "ANY"}
  default_options = {"protobuf_version": "3.5.0"} 

  def requirements(self):
    self.requires.add("protobuf/{}@user/stable".format(self.options.protobuf_version))

I guess you can still add manual range checking, and throw some kind of exception (InvalidConfiguration?).

While it works (though it is quite verbose, especially when adding version range checks etc), it would be great if Conan provided something for it out of the box.

[KerstinKeller]

Obviously, the real problem with the approach mentioned above is that when version resolving is actually taking place (e.g. MyTests is again consumed by someone else), then it's not part of the rest of the version resolving process by Conan.

And maybe to elaborate why I want this feature:
E.g. when I specify a requirement range (self.requires.add("somelib/[>=x.y.z]@user/stable"), maybe I'd like to verify in my CI, that it actually does build with this range of versions (e.g. oldest matching, newest matching, or even all available).

[fulara]

Hello @KerstinKeller you can read here more about the solution that I've taken:
https://github.com/conan-io/conan/issues/6707#issuecomment-601443704
in short - we use semver heavily in our environment - and I feel that Lockfiles which is probably conan choice forward will never give us such freedom, and I am getting a total freedom thanks to that solution
Option may not be desired in your case - because you lose the versioning_schema ability actually,
but in most cases people probably would rather full_package_mode read here Just be careful - because you may get surprised ( conans default is semver_direct_mode )

[steinerthomas]

Hi,
in my CI job I'm also pre-building some packages with different toolchains, versions and options and upload them to Artifactory for our consumer projects (we can't use --build missing neither).

Are there any news on this issue?
Will there be a possibility to override versions for transitive dependencies in the future?
e.g. conan create --requires=sqlite3/3.30.1 . sqlitecpp/2.5.0@ (like @fulara mentionend above)

[tvogel]

If I could vote for this, I would.

[KerstinKeller]

I would also really like to see this issue addressed. We want to do matrix builds for releases which are not distributed via conan, and we have not found a good solution so far.

I would also be totally fine just to create a lockfile and then use the lockfile for the actual package creation.