Conan: Is defining ABI compatibility in the producer instead of the consumer possible/desired?

[niosHD]

As far as I understood, by default, conan assumes that dependencies follow semantic versioning and that I can override the default in the package_id method of the consuming recipe. (see https://docs.conan.io/en/latest/creating_packages/define_abi_compatibility.html#versioning-schema)

However, when a package does not follow semver, does this mean that every consuming recipe has to override the versioning scheme or is there a possibility to define this directly within the package?

For reference, I had the following discussion with Hipony (Thank you!) on Slack who encouraged me to open this issue:

Hipony [11:12 AM]

@niosHD that's an interesting question, docs are confusing here and I can't really tell how to do it from the producer. They describe how consumer can describe compatible packages, but it would make much more sense to describe ABI rules in the recipe so each consumer would automatically get version scheme for each package, by rules defined by package' maintainers. And fixing package_id is not an option since it's an end-user feature. Confusing indeed! :thinking_face:
Basically it requires for an end user to have consuming rules, because you can't really control it from the producer side

niosHD [11:33 AM]

@Hipony Right, I would also prefer to express the ABI guarantees as part of the actual package and was hoping that I just missed this possibility, hence the question.

Hipony [11:37 AM]

It's a bit tricky, I feel you that it have much more sense to define ABI compatibility from the producer part, but then it would require to inspect dependency recipes from the consumer recipe and they can contradict with each other (one thing to be ABI compatible with semver and the other with different users/channels).
and keep in mind that dep recipes could be changed at any time with their respective compatibility values
tricky!

niosHD [11:48 AM]

Semantically, specifying the default ABI compatibility in the producer with possibility to override it in the consumer seems to be what I want. However, I agree that implementing this may be tricky. I assume that the full dependency tree has to be evaluated in order to decide if a binary package can be used. I don't know if this is also required in the current model but if not than this would probably be unacceptable in terms of overhead. Let's see what others think that are deeper in the conan internals.

Hipony [11:51 AM]

I think this is a important use case, although it would be a pretty big change in internals.

[memsharded]

Indeed it is an interesting idea to explore, I am not saying it is not. How packages could define a versioning scheme themselves, and how that could be used by consumers regarding ABI.

However, I think the ABI compatibility regarding dependencies, it cannot be fully defined in the producer. E.g.:

• you have a package PkgA that produces a static library
• Then you have a PkgB that depends on PkgA
• If PkgB is a shared library, you probably want to use a package_id() in PkgB that defines full_package_mode(), so every change in PkgA generates a new binary of PkgB
• On the other hand, if PkgB is a static library, it is likely that it can keep the default "semver" mode, as long as the API doesn't break, it should work for most cases
• If PkgB is a header library, for sure, it doesn't need to generate different packages for different versions of PkgA

In short: PkgA cannot say, in a general way: for every change X in my version number, force my consumers to create a new binary

We have some ideas to start modeling high-level relations in the dependency graph, like header-libs, static-libs, shared-libs, and better models of private, transitive, etc.
But that would be at least conan 2.0. Without that high level model, a producer only source of ABI compatibility is not enough.

[niosHD]

Thank you James (and Diego on Slack) for the explanations, they clarify why the consumer is at the moment in charge of overriding the ABI compatibility. However, even without switching to a high level model with conan 2.0, I think that a first step towards defining ABI compatibility in the producer might be worthwhile for the current conan.

In particular, the reason why I started to look into the ABI compatibility settings is that one of my builds broke because conan picked up an old package instead of rebuilding it against the new (overridden) dependency. At first, I thought that it might happened due to a bug but then realized that it is actually the ABI compatibility feature which assumes that semver is used by default. Unfortunately, this particular library dependency does not use semver and provides absolutely no ABI compatibility guarantees between versions.

Currently, it seems that the only way to fix this particular problem is to modify all consumers which is not very nice and error prone. What I imagine instead is something like a version_policy (akin to the build_policy) property in the package which allows me to setup the default ABI compatibility for all consumers. Of course, overriding the policy in the consumer should still be possible. Finally, by keeping semver the default value for version_policy, adding the property would even be a backward compatible change.

[jgsogo]

Read #4640 for further arguments.

[michaelmaguire]

Allow packages to publish their suggestion for the default case

I agree with the decision that a consumer needs the full flexibility to define how it views changes to the producer, but I think today by default, with no changes or package_id() in a consumer, the assumption is that any package it consumes respects semver.

As I've learned the hard way, this is actually a pretty strong assumption.

I think it might be very useful if a package could publish the default consumers should assume, which could be overriden (or not) by the consumer's package_id() method as today.

[fulara]

I am also inclined to say that packages themselves should provide ability to the end user about whats their 'minimal' compatibility.

I think boost is a good example here.
Boost does not conform to semver, it uses form of minor_versioning per your name convention.
I think it would be beneficial to be able to set that boost declares itself as maximum minor_versioning compatible.

Right now I am forced to go to every package that uses boost and say that boost has a 'major_versioning' otherwise i can very easily end up in undefined behavior - where one package depends on boost X and one on boost Y.

You could argue that i could change globally the setting. unfortunatelly I cnanot do that, as this is a requirement of some of application in our system - we do use semver there.

[memsharded]

'major_versioning' otherwise i can very easily end up in undefined behavior - where one package depends on boost X and one on boost Y.

No, this is not possible. Version conflics are checked, unless you use some not-recommended way of using boost as build-requires or private, it is not possible to depend on different versions of the same package.

The major, minor, patch things that are involved in things like full_package_mode() define whether the consumer of a certain package should generate a new binary based on the version changes of that package. That is impossible to be defined in the consumed package, by definition. I will put an example, now that you cite boost:

• Creating a package that contains a shared library that links with a static boost, and doesn't expose the boost headers in the public api of the package. For every version of boost, no matter if major, minor, patch, my consuming package probably will need to generate a new binary, otherwise it will silently skip the latest bug fixes in the app release.
• Creating a package that contains a static library that only uses some boost headers and not even depends on the implementation of the library itself, only through the public package headers for some templates. For this case, it is not necessary to generate a new binary of the package for every minor or every patch. Actually it is not necessary to generate a binary even if the major of boost changes. It might happen that it doesn't work, because the API breaks, but still that is not an ABI issue, but an API one. There is nothing you can do building that package to manage that.

Then, by definition of how linking works in C and C++, a package cannot define by itself how it affects the consumers, but it is on the relation, how the consumers use the package what defines whenever it is necessary to build a new binary or not. For Conan 2.0 we will try to improve the default definition of package_id() to reflect and automate these things, but still this relation is what will be used, not what a package might declare.

[fulara]

Sorry, I had few inconsistencies in my post.
Apologies in my previous post i meant 'built against' rather than depended upon. with packages like boost I would be very afraid to link against different version of boost then built against.

we are using boost in a normal way, just put it in requires no build-requires nor private.

let me share you with todays example i had:
For the sake of example consider that we are only using shared library - no static builds - including boost.
I had 2 projects.
1) library that depends on boost.
2) binary that depends on both boost and library.

New version of boost is out - and we are using here the semver_direct_mode, however boost clearly does not conform to that convention, it has right now version 1.72.0.

Since we depend on boost in binary, we just changed the dependency in binary against newer boost.
Now, what we would expect to happen is that conan install would say that library was not build with a set of dependencies, because since we have changed boost version in the binary it overrode the dependency of the library with the newer version, and we had never built a library that depends on boost 1.72.0. However what we got instead was a succesful install result.

Therefore we would end up with: library that was built against previous version of boost but links against new version.
Binary that is both built & linked against new version of boost

Therefore we are left with two choices.
1) Everywhere where we depend on boost we will have to say that boost uses minor_mode - the current aproach.
2) Whenever boost change, go through every project that depends on boost (and any other library) and rebuild it - which makes every library change a herculean task.

[himikof]

I think that the Boost libraries are a great example of the limitations of the current (semver-mode by default, overridable only per-consumer or globally) model.

The situation with ABI stability in the modern C++ world is very complicated, and there are many misconceptions and misunderstandings out there. For example, the following is, unfortunately, wrong in some (most?) common cases:

Creating a package that contains a static library that only uses some boost headers and not even depends on the implementation of the library itself, only through the public package headers for some templates. For this case, it is not necessary to generate a new binary of the package for every minor or every patch. Actually it is not necessary to generate a binary even if the major of boost changes.

The most important rule for C++ ABI stability is One Definition Rule: in short, almost any entity with a global name should have an exactly-equivalent definition in all cpp files linked together, else the _behavior of the whole program is undefined_ (which is usually quite bad). The language-level definitions knows nothing of the platform specific things like shared libraries, but I've tried to summarize the platform specifics:

1. The scope of required uniqueness when using static libraries is always extended to the user of the library.
2. All shared libraries in a process on platforms with global symbol visibility have a single scope by default. This is the default case on Linux (and most other ELF systems).
3. A shared library on Windows (or when using hidden symbol visibility on Linux) has its own scope for the hidden symbols. There are cases where a library must mark symbols that are not part of its interface directly (exception classes, for example) as non-hidden on Linux, making them part of the global scope.
4. A header-only library instantiates its symbols in the user scope, so usually it has the same uniqueness scope.

So, the Boost library collection:

1. Does not follow semver in any way, has a single version number for a collection of independent libraries.
2. Most (almost all) of the code is header-only, even in libraries with some compiled parts.
3. Most libraries keep only loose API and no ABI compatibility at all. The API and implementation can be affected by many preprocessor definition options and things like availability of features in a given consuming compiler.
4. Non-header-only parts can be compiled and used both as static and shared libraries. Prior to version 1.69.0 most Boost libraries did not (and some could not) use hidden visibility, but since version 1.69.0 the hidden symbol visibility is the default for shared libraries. Of course, symbols in the library interfaces (and there are lots of them!) are still non-hidden.
5. Many boost packaging conventions encode the full Boost version in the name (and the soname!) of the shared library. Dynamic linkers will happily load two libboost_thread.1.X.0 libraries (they are seen as completely different libraries) in one executable, usually leading to runtime havoc.

So, I'm afraid, by default, unless extra careful measures or platform restrictions (Windows-only, for example) are in place, all the code in a single executable must come from a exactly the same single Boost version. And all Boost-depending code should always be rebuilt when upgrading Boost, unless the Boost dependency is completely isolated to some some subgraph, and the said subgraph is isolated at least at the level of shared library with hidden visibility, leaking no Boost symbols outside.
This is also the general reason why Linux distributions caring about ABI stability never upgrade the provided Boost version in a single release lifetime.

So, if I understand it correctly, the only safe way to depend on Boost in Conan is by using full_package_mode in all depending recipes, and even propagating the full_package_mode upwards until stopped by a shared-library ABI barrier leaking no Boost symbols in the global scope.
Or, alternatively, the full_package_mode propagation can be stopped on a shared-library boundary even leaking the symbols, if there is a guarantee that no other part of the dependency graph uses Boost in any way.

Unfortunately, C++ libraries like Qt that have an explicit ABI stability policy are the exception in the modern C++ ecosystem, not the norm. The default rules almost work for them (except the unexpected downgrade problem), but lead to silent failures in other cases.

[danpetry]

Having read through these issues, I feel that the following could solve the problem while also retaining consumer control of the package_mode:

Right now it is implemented something like this by default: "all my dependencies uses SemVer and then the consumer can override some of them".

The same default behavior could be achieved in a different way: "I will ask all my dependencies about the ABI compatibility model they are using (and by default every library responds with a SemVer mode)", the user can declare a different compatibility model for his library, and of course the consumer can override it.
(@jgsogo)

Or even more passive and easy-to-implement: each package could add a public abi_compatibility_model property which consuming packages could check, if they want, and use this information to judge what package_mode to use for each of its dependencies. Default behaviour could be kept as it is.
@memsharded is this something that could be considered? (Or will this discussion be made somewhat redundant by changes in Conan 2.0 and so we should just wait for that...?)