Conan: Install **latest** version of a given package. Allow depend on 1.X version

[memsharded]

DRAFT
From my point of view this has to be carefully analysed. I will write down here my thoughts, details and design considerations as a starting point for discussion about this issue.

[skizzay]

One idea based on Chef uses its cookbook management is to allow automatic upgrades of patches. The assumption is that patches are backwards compatible and that the API hasn't changed. Of course, for C/C++, this would also need to extend to ABI changes, particularly for shared objects/DLLs.

For example, if I had MyPackage version 1.2.3 released. Someone could reference MyPackage/1.2.X@skizzay/stable. Conan would see that 1.2.3 is available and would provide that particular version. At some point in the future I make a bug fix and release 1.2.8. The next time the dependency is resolved, conan would resolve to 1.2.8 since that is the highest patch of the version specified. Later, I realize that my interface needed to be refactored some and release 1.3.0. The dependency resolution would still use 1.2.8 since 1.3.0 is not a compatible version with the component being built.

[dragly]

Thanks for the reply on Twitter pointing to this issue.

In Python's pip, you may specify a range of versions, such as

pip install "package>=0.2,<0.3" 

where pip will try to find the best match for your package. This can be an alternative to a 1.X scheme.

@skizzay Perhaps Conan should require that a package is built against the version of the package that is the currently resolved version? Say package A depends on version 1.2.X of package B. If A is built against B/1.2.1, then B gets updated to B/1.2.2, I would prefer it that Conan would requires A to be rebuilt, this time against B/1.2.2. In other words, the identifier for a built package should be determined by the versions of its dependencies in addition to the OS, architecture, compiler, etc. Anything else would be really hard to maintain because a lot of smaller libraries don't document whether they break binary compatibility or not between versions.

@memsharded What is Conan's plans for supporting installation of packages on client computers? Currently I have only used Conan for installation of packages when developing. Is the idea that Conan should be used for installation of final software as well? In that case, it's really important that updates of packages don't break binary compatibility. If not, have you thought of a way to export all libraries that the executable depends on for easy deployment of the application?

In general, I'd say it's best to stick to fixed versions numbers until a strong and rigorous backend for dependency resolution has been identified or implemented. (I suppose a lot of dependency manager libraries already exist, so it might not be necessary to reimplement the wheel for this).

[thiagocrepaldi]

Another versioning suggestion, similar to @dragly's is the Debian's one: https://www.debian.org/doc/debian-policy/ch-relationships.html#s-binarydeps
It allows version ranges. A "version exclusion" might help in some corner cases too

[skizzay]

@dragly I agree with that suggestion. I'm currently experiencing versioning hell at work. Currently, if I make a bug fix to a fundamental library with no ABI changes (i.e. a patch as I explained above), then I need to manually rebuild all of the libraries depending on this lower level library. This is quite cumbersome and time consuming. If conan was able to rebuild these intermediate packages because it detected an upgraded library, then that would certainly make my life a lot easier.

[memsharded]

I think that conan might be able to do this in its current form, at least for common cases. If no ABI changes, I guess that that implies no API changes too?

• Lets suppose that all libraries in your dependency tree are static. You would have in your conanfile defined your [requires], that could be just a couple of libraries, lets say for example Boost/1.60.0 and Poco/1.7.2. They transitively depend on OpenSSL/1.0.2g and zlib/1.2.8. Now, imagine we fix a bug in zlib, we could have 2 options:
  
  
  
  • Create a new package version, as zlib/1.2.8.a, or maybe use a channel zlib/1.2.8@user/stablev2. The bug is fixed and the new package is uploaded. To consume the updated zlib, also by Poco and Boost, it is not necessary to modify them, we can just declare in our [requires] Boost/1.60.0, Poco/1.7.2 and zlib/1.2.8.a. This will override the upstream and Poco and Boost will also use it. Typically, if there are no API changes, it is not necessary to rebuild Poco and Boost, recall they are statically linked, which means that the libraries themselves are not linked, and it is just when assembling the executable where the linkage occurs. Conan default binary package ID already accounts for it, as Poco and Boost packages hashes (SHA1), to compute the ID are done with something like: "zlib/1.y" and "OpenSSL/1.Y". Following semantic versioning, this means that the Boost and Poco packages do not need re-compilation if just the minor changes.
  
  
  • Re build the package, with exactly the same version, and re-upload it. Client can check the updated packages (conan info --update), and get them (conan install --update). Poco and Boost packages do not require re-build either in this case.
  
  
• All libraries of packages in the dependency tree are dynamic/shared. Then, the same applies, and updating an upstream packages do not require building the downstream dependents, if there are no API changes. Both of the above approaches can be used.
• There is a third case, when the ABI actually changes, or when an upstream package change will be actually embedded in the downstream package library. This would be the case for example for header only libraries being internally used (wrapped) by a static library. Every minor change in the header only has to be build into the downstream library, and then, it requires a re-build. The same if a shared library depends on a static one, the linking step of the shared lib embeds the static library binary code (in case of static, typically requires to be built with fPIC) in the shared lib, and then it requires re-build too. In this case, it would be necessary:
  
  
  
  • To publish bug fixes as new versions/patches/channels... but not override the existing one.
  
  
  • In the consuming packages, implement the conan_info() method and modify the info.requires field to contain the full version, so the full version of the upstream package generates a new binary ID for it, and thus forcing re-compilation. Something like:
  
  

python def conan_info(self): boost_require = self.info.requires["Boost"] boost_require.version = boost_require.full_version

• define the upstream new version in your project conanfile
• install with conan install --build=missing. The downstream packages that depends in that way on the new overriden version will have a package binary ID different from the previous one, and then, will be re-built. Other packages, that are not affected by the change, will remain the same.

This conan_info functionality might be a bit undocumented, and probably could be improved, but it does the job. It can be also used to narrow or expand how settings as compiler versions affect the binary packages.

So, I would say that such use case could be already feasible with conan. Please have a look and tell me, looking forward to receiving your feedback, thanks a lot!

[skizzay]

While this does seem technically possible, I've never seen anyone actually just overwrite an existing version. I've always seen that the offending binary is patched and versioned accordingly. This is necessary for reproducing bugs in production or running a playback of data to generate a report of some historical values.
One practice that I've seen that I liked is that the version of the library is written to a source file as a hard-coded string. This would be generated during the build process and deleted during cleanup. The idea behind this practice is that anyone can run "strings myapp | grep version_fingerprint" and see what library versions went into a particular build.

[memsharded]

Actually, there are users that do it! That is a difficult part of developing dev-tools, trying to accommodate most work-flows and use cases. Some people prefer just to overwrite their packages than to manage more versions. Mostly development or testing versions, more typical in not very large or not strictly managed processes (like startups or research, for example).

There are also issues with packaging. Your library code has not changed at all, you create a package, but some days after you realize that you forgot to package() some lib, or maybe the .dll for the shared case. Then, the library version doesn't change at all, and the build process does not change either. Actually, it is not necessary to re-build, just calling conan package could do it. When fixing the package, probably the old one is not necessary, as it will not be useful to test any regression, no changes in code, so that could be why some people prefer just to discard it, overwriting it.

I like the idea of generating or adding to a file a hard-coded string, though we cannot force users to embed it in their libraries binaries, and in any case, for just package changes (without changing the source code or the build), it is not the best option. Currently, we are embedding the timestamp in the manifests, for the the package recipe and binaries. The manifests are created at "export" time for the recipe, when a package recipe is exported, and at build time when a package binary is created. We are using it to detect the updates (newer-older version), but it could be used as build-id too. Or we could also introduce an auto-incremental build-id integer that would probably be more intuitive. Then, we could print it in conan info so you would know exactly which build of each package version you are using.

[Prinkesh]

How about using symlinks ?
Users can have the ability to create links to any versions of a library
Like,
conan link package/0.x package/0.1 demo/test

will create a link (package/0.x@demo/test) to package/0.1@demo/test , so if a new release is made for the package , user will just update the link to point to the respective version , so 0.x can now be used by consumers
this command can use symlinks to create the link , so instead of just traversing normal directories , symlinks can be traversed too while generating list of packages of file store

[memsharded]

The problem is that the approach should be portable, and symlinks are missing in Win (well, sort of, but they are a at least, complicated to make them work). Conan usage among different platforms is even, so Windows has to be taken into account.

Just a question, when you say the user will update the link, do you refer to the user that created the package? or the consumer of the package? If it is the consumer of the package, the concept of the symlink would be just changing the requirement in the conanfile.txt, not sure what the contribution of the "link" command would be. So, in that case, it should be automatic, letting the user specify 0.X, and make conan resolving that into latest 0.X version when installed.

[Prinkesh]

@memsharded I was referring to user that created the package and in windows _mklink_ can be used to create symlink .

[marktsuchida]

It would be nice if there were some syntactic sugar to do the equivalent of the conan_info() example given by @memsharded. Perhaps something like

def requirements(self):
    self.requires("Foo/1.2.3@user/stable", versioning="exact")

where the default for the versioning kwarg could be "semver".

Background: I work on a project with dozens of third-party dependencies, some of which are written by people who haven't heard of SemVer. I would also like to track package revisions explicitly in our project (e.g. Foo/1.2.3-1 and Foo/1.2.3-2 would come from the same upstream but might differ in applied patches or required settings or some obscure compiler flag) -- so that older versions of the package consumer can be rebuilt using the exact same code. Writing the conan_info() code for every dependency in every consumer would be hard to maintain.

At the moment our single package consumer project is not itself used as a Conan package (Conan is only used to install its dependencies), so none of the above matters (do I understand that correctly?) -- but it would be nice to be able to manage transitive dependencies in the future, since Conan might finally provide a practical way for us to modularize our build.

By the way, thanks for this wonderful project. It came out just when we needed something exactly like this!

[KindDragon]

symlinks are missing in Win (well, sort of, but they are a at least, complicated to make them work)

You can use hardlinks instead on the same harddrive

[monsdar]

Is this issue still being discussed? I would love to see some sort of Foobar/1.*.*@user/testing being implemented, where all the * would be replaced by the highest available version. I see that dependencies are already displayed as 1.Y.Z when I do a very verbose search.

Perhaps there could be some meta-information added to the package that gives information what exact version has been used to build the package? This then could be used to rebuild the package if needed.

The usecase is automatically generated packages by CI. Though it is working to just build against something like Foobar/HEAD@user/testing it would obviously be better to specify a (range of) version.

[memsharded]

Yes, still being discussed, it is just that it is a quite complicated feature, involving also the server, didn't have time because other more urgent and requested issues. Sorry for that.

[monsdar]

Reiterating my previous comment:
A package could have a dependency like Foobar/1.*.*@user/testing. If it gets built the information about what has been used is stored within the package. So there a string like Foobar/1.0.1@user/testing in there, which allows to reproduce the exact build.

This would allow for the following:

• Packages have a distinct version
• Packages could depend on a version range
• Packages store information about what they've been built with. Allowing to reproduce the exact same build.

There could even be a feature that allows rebuilding with the same dependencies or with the newest versions that are found. Like conan install Bar/1.2.3@user/testing --build missing --keepVersions or --useNewestVersions. I'm not sure what makes for the better default.

Another issue with that is the formatting of version. Currently this is just a string, but it would be impossible to see which versions are newer that way. Perhaps this feature would only be applied if the semantic versioning is used. It could also allow to have a 4th field for additional text-based info like 1.0.0.a8ec08d for the exact revision hash.

And yeah, you're right: This is a very complicated feature which could mess up a lot when done wrong. Once users start to adapt something like 1.*.* it is hard to revert that feature.

[marktsuchida]

To me it would seem ideal if the package, rather than its consumers, could specify whether its version is to be interpreted as SemVer or as a mere (unordered) string.

Having a simple baseline behavior ("don't interpret version numbers") will go a long way toward dealing with various atypical situations, even if it is not the most convenient/automagical behavior in the typical case.

When the version is known to be SemVer (and only then), it will make sense for the consumer and/or command line to specify the policy for selecting a concrete version.

(Of course there could also be support for known version formats other than SemVer.)

[monsdar]

I created #510 to create something like @marktsuchida proposed. This would be a good addition to prepare bigger versioning features like the discussed ones.

Are there any other issues we could extract from this one to have some more digestible chunks (in terms of discussion and work)?

[memsharded]

we are (finally!) drafting a proposal for this feature, will update here and request feedback and maybe even help. cc/ @DEGoodmanWilson

[memsharded]

Opened a new issue to summarize a proposal and follow conversation about implementation and UX: https://github.com/conan-io/conan/issues/641

[memsharded]

Version ranges are supported from conan 0.16, which also cover this feature.