Conan: Backward compatible package version resolution

[progician]

Hi,

So I was exploring a bit deeper into ABI compatibility of packages when I've found a problem which might not be a bug in as such, but leads to unexpected behaviour.

So, as per SemVer, the minor version change signifies a backward compatible change. That is to say, as long as the semantic version past the required one, it should be ABI compatible. Now, the default ABI compatibility is resolved by taking only the major version number into consideration. But that's not the entire story.

Let me demonstrate:
LibA 1.1 and 1.0 - shared library.
LibB 1.0 - shared library, depending on LibA 1.1
App - application, depends on LibB 1.0 and LibA 1.0

When you install the dependencies for App, what you get is a LibB 1.0.0 and a LibA 1.0.0 dependency. Even though, LibB/1.0 requires LibA/1.1 when you look at the resolution, it will be just LibA/1.0.0 which will lead on some platforms only to runtime failure. Conan will not notify me about this.

Now I know, if only I would use version ranges, this problem would be solved. Say if App would require LibA/[^1.0] then it would pick resolve the package dependency to LibA/1.1. However, this puts the burden on the client package, which looks backward to me. So soon I would find myself writing version ranges everywhere, however, it isn't really what I want. If LibB next version decides to version range its dependencies too (say, for the very same reason), then I find myself with a moving target builds.

Also, it could be just solve to make sure that I always pick the largest version number in the client package, right? But... I'm not necessary aware by default what dependencies it requires. It means that the burden is on the client package's dev to figure this out.

I propose that at least conan should warn about the discrepancy in requirements. It should be an error condition to say the least, to require a lower version number in a downstream package between minor changes. They ain't really compatible anyway, not really. It's missing the additional functionality another package requires. And, perhaps as an automated resolution to these problems, conan should have a configuration setting of some kind that allows it to pick the version by precedent, as that's the only resolution that makes any sense. Otherwise the developer must made aware of this and remedy it by matching the highest requirement.

[memsharded]

Yes, this is the implemented behavior in Conan.

Dependency overriding is a common mechanism in package/dependency managers. If the final consumer says that they want to use LibA/1.0, that should have priority, over the LibA/1.1 definition of LibB.

Dependency overriding shouldn't raise an error every time an upstream dependency is overridden. It is not a burden, if App is using LibA, they should be able to define which version of LibA they are using and that should be followed.

A different story is the binary compatibility, and it was probably not the best decision to base default ABI compatibility in semver, but unfortunately that default cannot be changed now without breaking. As it is implemented, the default package_id() using the form of the dependencies as LibA/1.Y.Z in the package ID hash, it means exactly that:

• The binary of LibB will be binary compatible with any of the LibA/1.0, LibA/1.1, LibA/1.2, etc.
• Then an application can perfectly decide to use LibA/1.0, even if the LibB was compiled and linked against LibA/1.1, because in theory that means that such versions can be used interchangeably, and it is not an error.
• This might be ok in many cases, like pure C, or if the logic an changes are totally restricted to the binary and not propagated in the headers or to the linkage in any way.

The good news is that it is only the default behavior, and can be easily changed. You can define in your LibB package_id():

def package_id(self):
     self.info.requires.full_version_mode() # or full_package_mode() if you want more

This will force a build of the LibB package using now the exact version of LibA. If LibB won't build against LibA/1.0, then the conflict can only be resolved if App uses LibA/1.1 too.

More details can be found in: https://docs.conan.io/en/latest/creating_packages/define_abi_compatibility.html

We are working in the idea of a new package_id() model that will use the exact binaries by default, and maybe we build a opt-in global configuration for that new mode.

[progician]

* Then an application can perfectly decide to use LibA/1.0, even if the LibB was compiled and linked against LibA/1.1, because in theory that means that such versions can be used interchangeably, and it is not an error.
* This might be ok in many cases, like pure C, or if the logic an changes are totally restricted to the binary and not propagated in the headers or to the linkage in any way.

I disagree. This interpretation only true for the patch number. In the case of SemVer at least. Or then, what's the point of the third digit? For backward compatibility, the point exactly that the highest number should be taken, as it is backward compatible with all the preceding changes within that major version.

Even in C I can add a new function to a shared library that will then will be overriden and thus at runtime I might get into a situation where a symbol cannot be resolved at it will use the wrong version of the library.

I know that I could possibly restrict the ABI compatibility only to patch number, but that would be quite some restriction, because if I switch the numbers around and say App -> LibA/1.1, LibB/1.0 and LibB/1.0 -> LibA/1.0 then it actually true the default 1.Y.Z compatibility. So the solution to backward compatibility is to say 1.Y.Z if you take max Y.

I'm not saying it must be default behaviour, though that would actually comply to SemVer which is supposed to be the basis of the versioning system in conan. As I said, if I override the version for in the downstream side, I will not get any warning until runtime that something is amiss. Which is really suboptimal, isn't it?

[memsharded]

I disagree. This interpretation only true for the patch number. In the case of SemVer at least. Or then, what's the point of the third digit? For backward compatibility, the point exactly that the highest number should be taken, as it is _backward_ compatible with all the _preceding_ changes within that major version.

Yes, it is true that the logic is wrongly based on MAJOR version when you make incompatible API changes => MINOR version changes do not break API compatibility. I wish the default behavior was not that one.

Still, the override of LibA from App shouldn't be an error by default, even if it is downgrading the previous requirement defined in LibB, as it is totally possible that LibB is not using the latest functionality added in LibA/1.0, and might be still compatible with LibA/1.0, even if it was developed and tested against LibA/1.1. Is your suggestion that this override should display a warning in case of downgrades? Maybe that could be done.

We will consider if something could be done in the binary package_id computation but sounds extraordinarily difficult, and maybe it is not worth, considering that the majority of the feedback request "exact" binary IDs, that is, if anything changes upstream, then strictly require a new binary. Thanks for the feedback!

[progician]

It would be a good start to show a warning, but even better if I could turn that warning into failure (optionally, based on conan.conf for example).

Still, I would prefer even more a model of dependency resolution where it can resolve to the 'latest' of all the required compatible versions, even if the downstream. It does not sound too bad for me. I will look a bit into this in the source, but I haven't hacked conan myself just yet.

A configurable global mode of version compatibility mode would be nice either way though, so sign me up for that. In my company we currently trying to figure out how to do our compatibility best. Some people would prefer the stricter full version thing, which I consider as a throwback to our old monolithic builds. My problem with that is that we end up rebuilding a whole lot of things for no good reason. Like we have Boost in every single package we have pretty much, so any upgrade in one will cascade through our entire dependency DAG, a lots of packages I assure you. Not only just rebuild, but alter, as the recipes must changed to pick up the later version of Boost. Yet Boost is one of the mostly stable ABI/API, so there will be almost no change in anything at all. The current status quo suits me fine TBH, as if a package breaks its contract with regards to SemVer, we can make it stricter with full_version_mode(). Though it could be annoying to add it to every single dependent recipe.

I don't see too much tooling available to navigate a large DAG to replace version numbers ATM, a problem which stricter versioning would surely require. Any suggestions? I mean, something better than sed...

EDIT: Come to think of it, without package revisions, basically with full_version_mode() the implication that I need to adjust the conanfile.py of all the dependencies the change every time a single transitive dependency changes, it also will mean that in order to have reproducible builds, I have to adjust the version number of the affected dependent packages, which then again means that it will cascade into a large number of edits across the board and then rebuild. Does not sound particularly good way to handle this problem.

[memsharded]

Ok, lets try to propose it. The idea would be:

• A configurable, global, opt-in "override mode" (like in conan.conf)
• Modes could be None (current default), error_downgrade, error_all, warning_downgrade (just an idea, not a proposal).
• The moment a dependency is overridden, that mode would be used to act accordingly.

[Minimonium]

Can we add this into the graph model project too? :slightly_smiling_face: