Conan: Support for version ranges and other expressions

[memsharded]

This is a feature to summarize and discuss implementation details and UX for the so called "Installing latest version or supportint 1.X": https://github.com/conan-io/conan/issues/15.

Ongoing work can be followed in: https://github.com/conan-io/conan/pull/640

It is also related to the issue: https://github.com/conan-io/conan/issues/548 and related work by @Prinkesh

Lets outline this feature and the proposed approach:

• It will allow package recipes and consumers to specify a "version expression" instead of an exact version. The syntax proposed for it would be MyPackage/[expression]@user/channel, where the [] are literal.
• User/channel are not to be evaluated or changed. They are intended for very different use cases, and an installation of a package from another (lets say, testing) channel or user, doesn't seem desirable at all.
• Typical syntax could be, e.g.: MyPackage/[>1.2, <1.8]@user/channel, or something like that. Still not defined, but this concerns me very little. Of course, expressions like ~3.0.0 as suggested by @DEGoodmanWilson can also be supported.
• By default, it will evaluate against the local conan cache, i.e. installed packages, so if it finds matching installed packages for the given expressions, it will use them. If they are not found, it will check in the remotes, in order, like a normal conan install
• That means, that for a given install with all the packages in the local cache, re-running conan install will not make any connection or query to any remote to try to resolve/update such version expressions
• If a check in the remotes for newer versions is desired, that can be done with the conan install --update argument. Such argument will ignore the locally installed packages and directly query the remotes, in order to retrieve the best matching version to the given expression.
• The usual ABI compatibility rules still apply for a given package, and must be defined as such in the package conan_info() method. That means, that if a package A has a dependency to B/[b-version-ranges]@user/channel, and using different versions of B inside A (like if B is a header only being embedded in a shared A library), generates a different binary for A, then the required specification will still be necessary in conan_info(). This is not a simple task, but it is exactly the same as right now with versions overrides, only the package creator can know about the package binaries compatibility regarding its dependencies.
• It will work locally and incrementally for each package, from downstream, starting from the user project, moving up in the dependency graph. When a package recipe is retrieved while building the graph, its dependencies will be evaluated using the version expressions found in its requires, and will move up in the graph. Further version ranges up in the graph might satisfy such requirement or fail, but they will not re-define the version. A global SAT solver for the full graph would be extremely complicated to implement given the complexity of the dependencies, as they can, and often are, conditional on the specific settings like OS. Keeping the current policy, which is that the downstream packages and lately the final consumer is the one defining the versions, makes the most sense, due to the nature and typical flows with C and C++ projects.

Help wanted: It would be extremely useful if someone could contribute with part of the logic. I have already started with the scaffolding and global approach, but the evaluation of the version ranges and expressions would be a completely isolated functionality, easy to implement regarding conan, but will require a good effort.

Looking forward feedback, thanks!

[memsharded]

Regarding some of the feedback by @monsdar and #510, not sure about whether the desired approach has to be specified by the creator of the package or the consumer. Using something like a use_semver flag in package recipes (defined by the package creator) has utility at most for showing information to the users, but all possible functionality related to it should be defined by the consumers. I don't care too much if some library uses semver, but I want to specify that I will consume it as semver, or maybe not, just use a given range of minors or patches.

[DEGoodmanWilson]

I have some feedback—I apologize if I am late to the party, and this has been hashed closely already.

Regarding these lines:

• By default, it will evaluate against the local conan cache, i.e. installed packages, so if it finds matching installed packages for the given expressions, it will use them. If they are not found, it will check in the remotes, in order, like a normal conan install
• That means, that for a given install with all the packages in the local cache, re-running conan install will not make any connection or query to any remote to try to resolve/update such version expressions

Depending on the nature or meaning of the expression, this may not be desired behavior. For example, in the Ruby world, if I "twiddle-waka" a minor version (_e.g._ ~>1.1), my expectation is that new deploys will pull down any new patch-level changes that I might not be aware of. That is, the ~> operater _implies_ that updates should be applied of available. The reasoning is that there may be, _e.g._ important security updates that I don't know about, and hence don't know to update.

So I would encourage thinking about rolling the update logic into the expression language.

This has the added benefit of being able to express on a per-package basis which packages should be auto-updated—a global CLI option would entail that _all_ possible updates be applied (which, granted, sometimes you want, but not always).

Thoughts? I would be happy to start on some of the version expression implementation, but I think we should have a little bit of a consensus on the syntax.

[lasote]

What I can remember of ruby and other package managers like bower & npm the behaviour is similar to the described by @memsharded and the update option. Your require's version's get "freeze" until you make an action (like remove a lock file or perform an update). In a fresh installation (like testing or a deployment) the requires will be upgraded but two consecutive invocations to install command won`t.

[DEGoodmanWilson]

You know, you are probably right. Most deploys that I have been involved with do not push the gemfile.lock so that fresh deploys always get the most recent version that matches the requirements (and then re-deploys don't). I'd been pretty blind to that because of the way deploys happened where I worked in the past ensure that each deploy was a fresh one (a brand new Docker container each time), so I had forgotten. Thanks for pointing that out.

[memsharded]

Yes, that was the idea. In a new installation of conan or just after a conan remove "*", it will freshly install things, evaluating everything and getting the latest matching versions from remotes.

The subtle difference is about the lock file. We are not proposing a lock file that should be or not committed to the repo. If a project wants to ensure to use a certain set of exact dependencies, the way to go would be to make them explicit in a conanfile (.txt or .py).

[DEGoodmanWilson]

So, one question I have as I delve into this a little on my own is: Obviously, we need to support a more general versioning scheme than SemVer, but a) _how_ general? Some examples of crazy version numbers out in the wild would be helpful (where "crazy" means "at odds with other versioning schemes" _e.g._ 1.2.3.4 or 1.2b are both at odds with SemVer) and b) Do we want to include SemVer as a subset to support future package authors who want to use SemVer as intended? And if so, does that conflict with any versioning schemes in use? In particular the semantics of the -foo and +bar suffices, do these risk trouble?

[lasote]

Yes, we need to define the "version" format we will support. I suppose any not correctly defined version according our definition will be discarded to be selected. Do you think we should support more than the semver format?

From semver.org:

A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92.

Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85.

Maybe this is flexible enough. What do you think?

[DEGoodmanWilson]

I suppose that is what I am asking :D Can we shoehorn the kind of versioning schemes actually in use in the world of C into Semver? I suspect the answer is probably: Yes. But, I wanted to be sure.

That said, while we can certainly shoehorn the various versioning schemes out there into the _format_ of Semver, there remains a distinct open question about the ability to preserve the _meaning_ of Semver. That seems pretty likely to be something we'll just need to throw out for most existing software projects.

[DEGoodmanWilson]

If we want to go with a syntax compatible with SemVer, FWIW, we should consider just using this module directly: https://github.com/podhmo/python-semver

[memsharded]

Yes, good tip, I was implementing my own (extending conan Version), but I will try that one, thanks!

[memsharded]

Updated my initial proposal with such package, works mostly fine, I just had to use "," instead of space to delimit conditions on ranges, but besides that it work. Updated my PR accordingly: https://github.com/conan-io/conan/pull/640.

Tests mostly passing, at least for consuming, if someone wants to have a look, use my feature/deps_ranges branch.

The syntax would be requiring Pkg/[>1.0, <1.6 || 1.8]@user/channel

I still have to add some tests to check for correct upstream propagation, in fact I'd say that some new evaluations should be done.

[memsharded]

Integration tests passing, ready to test!

PR updated. Manual testing very welcome, running from source my branch: https://github.com/memsharded/conan/tree/feature/deps_ranges

There might be some minor issue with the Semver library, or at least I find annoying that it doesnt find version "1.2", with an expression like "<=1.2". It might find it using 3 numbers, though.

Will require installing requirements pip install -r conans/requirements.txt, new dep added.

[DEGoodmanWilson]

There are a _lot_ of libraries out there that use an x.y scheme. It might be worth forking the module to add support for those cases…

[memsharded]

Yes, you are probably right. In any case, this is a minor and localized issue, I suggest testing the overall approach, transitive dependencies, usability... then we might try to fix the library if necessary.

[memsharded]

Hi @DEGoodmanWilson , we are about to merge it to develop, to be released as in initial proposal in next 0.16.

The "annoying" behavior (at least for me) is localized in this test: https://github.com/conan-io/conan/pull/640/files#diff-136c17ae49ba530f666509bc28575f65R37

In case someone can have a look, try to fix from upstream?

[memsharded]

Released in 0.16.

The remaining small issue about python node-semver with <=X.Y might be checked. In any case, there is an easy workaround, it can always be specified something like <X.Y+1.

Marking this as fixed and closed (released), but feedback very welcome.

[osechet]

With this new feature, is there a way to freeze the dependencies of a project, for example before a release, as shrinkwrap does it for an npm project?

[memsharded]

Not explicitely/automatically. But IMHO, I would certainly not use version ranges for real development, especially in large projects. And from what I know, very much the same is done in many C and C++ dev companies. We are not NPM, we do not pull hundreds of deps, and we want to know which deps we are pulling. We do not depend on Boost, any version from 1.55 to 1.60, we decide to use one, typically a careful decided one, like 1.59, and stick with it for the whole project. I might use it in early stages of a project, while trying different versions, but as soon as I have something stable, I would change my version ranges to fixed dependencies.

IMHO shrinkwrap is a patch to the common use of using version ranges everywhere, all the time, which is typical of dynamic languages.

Besides that, there is something to be considered. Dependencies could be frozen per configuration. Please remember that projects have different dependencies in different OSs, for example. You can check projects like Boost to realize this. So there is no way to generate a simple single lock file to freeze dependencies. Any suggestions?

Another factor to take into account is that you don't need to define version ranges to be able to evolve a project, as dependencies can be overriden downstream. You can have package A -> B.1, and B.1 -> C.1. If a defines a dependency to C.2 (can be "override" type), then B -> C.2

Said that, I think the effect could be achieved in different ways:

• If you are using conanfile.txt, just create a new one conanfile_release.txt, with the fixed dependencies (could be obtained from conaninfo.txt, or from conan install output).

• If the project is being built with a conanfile.py, maybe just define an "option", to switch between version-ranges requirements or the fixed requirements.

If the dependencies are not first level, to achieve exactly the same order of propagation of options, it might be necessary to define them as "override", which can be only done in conanfile.py.