Conan: Add ability to force using system-installed dependency instead of Conan package

[GamePad64]

Hello, I am writing a Qt-based library and want to create a Conan package.
But, I want to give the end-user a choice:

• install Qt with Conan, as a usual Conan dependency

OR

• make Conan assume, that Qt is already installed by the platform-dependent package manager (apt, rpm, homebrew, ... whatever), and rebuild the package from source (to mitigate the possible version incompatibility).

But, I want this mechanism to be more generic and standardized, so the end-user could mark any dependency to be "system-installed" (using Conan profile, for example) and all packages, dependent on this will be rebuilt, assuming, that this dependency is present on the system.

[memsharded]

Hi Alexander,

I think there are couple of different approaches for this:

• Have a different Qt recipe, that acts only as a wrapper for the system installed one. It might take advantage of the system_requirements() method and the SystemPackageTool helper: http://conanio.readthedocs.io/en/latest/reference/tools.html?highlight=SystemPackageTool#tools-osinfo-and-tools-systempackagetool. Put that package under the system channel. Then you would have to different versions for your Qt/version library, one would be the full conan one, able to build from source code and manage the binaries within conan, and the other would be just the wrapper. Packages can then use conditional dependencies to depend on one or the other, or use dependency overriding from downstream, to force the usage of the one you want.
• Make your Qt recipe, configurable with an option, something like system=[True, False], and use that option inside the recipe to act accordingly. build() and package() methods would do no thing for system=True.

I think both could be valid solutions, depending on the personal preferences and particular team workflows, would choose one or the other. What do you think? Could be a solution?

[hoxnox]

I tried to move through the second approach (see my recipes, for example https://github.com/hoxnox/conan-libressl), but I had to use my own extended ConanFile (see https://github.com/hoxnox/conan-nxtools) to make system and root options available in all my packages.

[GamePad64]

@memsharded

• The first approach is using native override mechanism, it appears difficult to use on an everyday basis: there are too many libs out there inside the system, and writing a separate (dummy) conanfile.py for each of them I want to override will lead to a great clutter.
• The second approach is too intrusive. It will require the maintainers to write completely similar code (like provided by @hoxnox)

@hoxnox You are moving in a right direction, and all your packages can be overridden by the system-specific package. I propose adding this functionality directly into Conan, so every package could be explicitly overridden (not by default, of course).

Note: maybe, restrict this option to _package consumers_, so Conan packages would depend only on other Conan packages (not on the system ones).
Then, every package could still be installed on a clean system using a full dependency graph without making any assumptions about system-installed libs.

[hoxnox]

I propose adding this functionality directly into Conan, so every package could be explicitly overridden (not by default, of course).

It would be great. I think it's important feature. The lack of it produces problems in building packages for OS distributives (.deb, for example). And it should be conan level, not package level because of transitive dependencies. If, for example, I wish zlib would be taken from system, I don't whant it to be staticaly compiled in by my dependencies.

[lasote]

To summarize, are you suggesting something like an always available option system defaulted to Falseand a package_info_system() method that will just declare the system's library names and any other flag needed or am I missing something?

[hoxnox]

There are situations in wich there may be several system packages with the same name (very often on WIndows). So there should be ability to force root directory helped to find the lib. In my code it looks like that:

    def package_info(self):
        self.do_package_info()
        if self.options.system:
            self.cpp_info.includedirs = []
            self.cpp_info.libdirs = []
            if len(str(self.options.root)) != 0:
                self.cpp_info.includedirs.append(str(self.options.root) + "/include")
                self.cpp_info.libdirs.append(str(self.options.root) + "/lib")

package_info() and package_info_system() should not conflict. Who is responsible for writing package_info_system()? Package creator? Or package user? What if package creator know nothing about system option...

[lasote]

But if you think that the package user has to take the decision about which system package link and the root...(I see it reasonable) It's almost the same effort than create a dummy recipe and override it, right?

Actually, the package creator won't need to introduce anything special, so all packages would be "compatible".

[GamePad64]

Personally, I am thinking only about an always available option "system" or "skip", that basically skips the build(), source(), package() and import() steps from the package, without any package_info_system() methods.
Searching the "system" package location is what the build system exists for.

[GamePad64]

@lasote basically, yes, but without writing dummy recipes by hand.

[hoxnox]

For complex situations (root change) dummy package should be created, for simple cases - option system would be great.

[ZaMaZaN4iK]

@lasote @memsharded what is the status of the issue?

[memsharded]

For me, it is still confusing this feature request.

Lets start with the UI. Lets say we have some package "LibA", with a conanfile.py and a conan package we can install, and such LibA is also available by other means at the system level.

Lets say that we implement a command-line skip option, it would be something like this?

$ conan install .. --skip=LibA

And then, even if I have listed LibA/version@user/channel in my conanfile, or it is in my transitive dependencies, it won't get installed at all, because it would be at the system level? Or I get installed the recipe, but it does nothing, but the package_info()? But package_info() will be most likely incompatible/wrong for the system packages.

So there are a few questions:

• If I created a LibB conanfile.py package that depends on LibA, that of course uses the conan information (e.g. with the cmake generator and the conanbuildinfo.cmake) to link with it, it will fail if I provide the skip option, right?
• I could add some manual stuff to LibB build scripts, but that doesn't scale at all if I have some other packages LibC, LibD, etc that depends on LibA
• What happens if LibA has other transitive dependencies. I don't think this is possible at all, so the suggested approach is only for packages with no transitive dependencies? That would quickly break, even for system packages. Gathering the transitive dependencies information for all system package managers is totally unfeasible, it is not realistic to implement that in conan.

Please @GamePad64 @hoxnox @lasote feedback.

[hoxnox]

If I created a LibB conanfile.py package that depends on LibA, that of course uses the conan information (e.g. with the cmake generator and the conanbuildinfo.cmake) to link with it, it will fail if I provide the skip option, right?

Why? libA package should find libs and headers in the system and provide correct package_info, so libB could use the provided information for link with.

I could add some manual stuff to LibB build scripts, but that doesn't scale at all if I have some other packages LibC, LibD, etc that depends on LibA

Some manual stuff should be added in libA package. It will not affect libB, libC and libD.

As for me - the new argument doesn't needed. Just wide ConanPackage with default system and root options, for example like I do in my packages and describe them in the documentation to force package creators use it. This helps a lot with building packages for various OS (windows, debian, gentoo).

[memsharded]

@hoxnox Your approach is very interesting indeed, as it is related to the "conan project" information. Please have a look: https://github.com/conan-io/conan/issues/1377

The idea was to be able to define in a file a set of mappings, from PkgName->Root folder. This allows not only for system dependencies, but also for user dependencies in user space. That root approach in "conan-project" is already implemented as a PoC and does not require to code or modify at all the package recipes.

[clssn]

Hey @memsharded , please allow me to put my use-case into the mix, hoping to revive this discussion.

I've got a library A requiring boost v1.69.0 (conveniently getting it from Conan Center).
A plugin library B to be loaded at runtime by A has a dependency to ROS (Robot Operating System). ROS is quite popular, but I can't find ROS libraries in any Conan repository. So I have to use the one installed on Ubuntu (still 18.04, so ROS melodic).
Using a find_package(ros_cpp REQUIRED) this is not a problem. Problem is though, that ROS melodic depends on the older boost 1.65.1 whose include directory is /usr/include. Building package B CMake gets the boost 1.69.0 include directory first in the include list, effectively hiding the system's boost 1.65.1. Unfortunately I need to use 1.65.1 here because when I compile with the 1.69.0 headers, I get a segfault at runtime.

tl;dr: In package B I'd like to override the transitional boost 1.69.0 requirement from package A with the system installed one for use in package B. But I don't want to maintain a whole lot of wrapper packages for these kinds of things.

Bottom line: In an ideal world, I would get ROS via Conan and have Conan handle dependencies one or the other way, but as long as there are libs that we need to use from the system, I'd like to have a means to express that with Conan.