Conan-center-index: [question] Approach for system requirements

[danimtb]

There are some recipes that may need to install tools that can be considered as system requirements.

There are already some recipes in the index that check for system tools installed but do not install them. Others like libusb do install system requirements as libudev.

We feel it is intrusive and risky to install system requirements when using a Conan dependency and we want to avoid this in Conan (specially asking for sudo permissions 😩).

The question here is what approach should be taken when packaging recipes that require any kind os system requirement:

• Should we create recipes for system requirements and package binaries?
• Should we just check the existence of that tools installed and just raise?
• Should we create "proxy recipes" that perform those install or checks to model that dependency in the conan graph?

Feedback please!

[SSE4]

for the reference, I am copying here problems/disadvantages/limitations caused by system requirements we previously faced on bincrafters:

---

System Requirements in conan known to cause lots of various issues, they are hard to maintain, and don't scale well. some problems with System Requirements:

• they require running system package manager from sudo, which is not a desired in many cases. they pollute build machine, they require user interaction, and they require permissions which developer might not have at all.
• system requirements don't support cross-building at all, especially cases like cross-compiling from Windows/Mac to Linux are impossible, and even Linux -> Linux is problematic.
• due to the fragmentation, there are many major package managers to support, like yum, apt, pacman, etc. support of each new package manager requires additional baby-sitting (and possibly modifications to the conan itself, to add new distributions like Gentoo, Slackware, etc.)
• within single package manager, there are various architectures to support - and this adds an additional level of scalability problem.
• even within single package manager, various distributions may have different names of packages. for instance, Fedora, RHEL and CentOS are known to have different package names for certain packages.
• even within single distro, packages are deprecated and removed from time to time, so it requires continuous maintenance with every new release.
• the code to use system package manager in recipes looks ugly - it has lots of nested conditions.
• due to the various bugs in distributions, packages often conflict with each other, making it impossible to build a conan package on the system. for instance, on Ubuntu, 32-bit python-development package conflicts with 64-bit one.
• installing additional software from system package manager, may confuse conan build. for instance, libprotobuf-dev might be installed as a dependency of some other package, then it may override protoc executable from the conan package, which is not a desired effect.
• builds are no longer truly reproducible, because various distros have different versions of system packages, therefore behaviour may change after recompiling on another distro.
• additional point of failure and additional dependency on the availability of system package manager registry - e.g. apt repositories are known to have downtimes. given the fact conan builds on CI are very unstable for other reason, it would be nice to have to remove that factor.
• conan builds are no longer self-contained - installation of executables/libraries will not run or link because of dependency of system libraries, which might not be available on another machine.
• use-case simply doesn't work on systems with no system package managers (e.g. source-only distros, like LFS systems), or systems with poor support for package managers or lack of packages (e.g. Windows).
• in order to use conan package, now it's needed to use pkg-config to locate system requirements (or locate them by other means, some system requirements don't even have .pc files!) - this adds additional complexity to the integration of conan package into the build system.

right now, the following conan packages are heavily based on system requirements:

OpenGL-related: GLU, GLUT, GLEW, GLFW, GLM, etc.
ffmpeg: X11, vaapi, vdpau
SDL2: X11, wayland, mir, audio APIs (pulseaudio, alsa, esound, nas, jack, etc.)
wxWidgets: GTK

[SSE4]

my two cents:

Should we create recipes for system requirements and package binaries?

if it's easy and straightforward - yes, that should be the first choice (as we did with libalsa, for instance)

Should we just check the existence of that tools installed and just raise?

I'd say that's much better than system requirements, and should be the second choice

Should we create "proxy recipes" that perform those install or checks to model that dependency in the conan graph?

such things sometimes are needed, e.g. OpenGL support might be abstracted away in conan (on Windows, you have to link with opengl32.lib, on Mac you have to link with OpenGL.framework, and on Linux you need some system libraries such as libGL.so). this also applies to few other things which are system-specific (e.g. Python, OpenCL, Vulkan, CUDA, etc.). it would be much simpler for other recipes to just include opengl-support conan package and don't worry about system specifics and system requirements. in such case, we delegate all baby-sitting and complexity to single recipe, rather copying ugly fragile conditions to every recipe which requires OpenGL.

[Hopobcn]

I agree with the ''proxy recipes'' concept. Take the case of CUDA, they ship it with an EULA which makes it nearly impossible to be packaged by conan (If I remember well, the only redistribution allowed is via docker image). Also, you don't want to make an installer package because not only it requires sudo and downloads several GBs, but also installs some kernel modules which could be a big NO if the user is not informed previously.
Shoulnt "proxy recipes" require changes in the CCI hooks?

[SSE4]

they do require changes in hooks - in practice, such proxy recipe may contain no files at all (no headers or libraries copied), but it only may have package_info method specifying platform-specific includes and libraries. however, currently our hooks complain if no files are copied. but that should be an easy change (or we can simply white-list such packages).

[Croydon]

@SSE4 pretty much summarized it and I agree: system_requirements should be avoided whenever possible.

In the specific case of libudev @uilianries and I were of the opinion that as part of systemd it is too low-level to be packaged with Conan (maybe not?). And libusb needed it, as it is non-optional, but also not pre-installed in the CCI building workers.

At Bincrafters we slowly create packages for the system_requirements in our recipes. This sometimes takes an awful lot of time, but I don't see any reason to loose patient now just because we have the Conan Center Index.

At first glance it might save a lot of time because you don't need to create all of these packages, but precisely because of all the downsides @SSE4 summarized I believe that the time in supporting users and for adding workarounds for specific issues will cost much more time long term. And quality wise it is much more better to have Conan packages anyway (again; see @SSE4 summary).

That said, I support https://github.com/conan-io/hooks/pull/112 to make sure that nothing slips through review. We can add specific packages to a whitelist if we -absolutely- have to at some point.

[czoido]

In the end, it seems that for the case of X11 (and probably more typical libs that are close to the system) the best would be to have such "proxy" recipe in ConanCenter that is xorg/system (encompassing several X libraries), and simply check that it is installed and inject the flags but not building or packaging from source.
When someone from a Linux desktop requires a package from ConanCenter, like GLFW or the like, that requires the X11 windows system installed, they will certainly want to be using the X11 of their system. Having X11 as real packages from Conan could be conflicting/incompatible with the existing X11 runtime.

[ericLemanissier]

I started working on wayland : https://github.com/bincrafters/conan-wayland. The package is quite self-contained, and ubuntu distro seems to build it "vanilla". Also, wayland is just the implementation of a protocol (named wayland too...), and I don't know of another implementation.
This makes me think we could simply create a conan-center-package for it, without creating a proxy for the system version. Do you agree ?

[ltjax]

I think you definitely want the option of using proxies for system level dependencies like that, regardless of whether it's technically a good idea. The end decision should be up to the application developer/stakeholders.
There's really a few concerns with this. Deployment is only one of them. Sometimes you also do not want to many deep per-project dependency hierarchies in conan and instead rely on a baseline system for compilation. E.g. I might want to link to wayland, but I really don't wanna see that it drags in expat or libffi, much less have those libs generate conflicts further down the line.
Maybe it can be done in a more "conan" way by deploying the proxy recipe for a specific user/channel like system/stable?

[uilianries]

for the reference, I am copying here problems/disadvantages/limitations caused by system requirements we previously faced on bincrafters:

System Requirements in conan known to cause lots of various issues, they are hard to maintain, and don't scale well. some problems with System Requirements:

* they require running system package manager from sudo, which is not a desired in many cases. they pollute build machine, they require user interaction, and they require permissions which developer might not have at all.

* system requirements don't support cross-building at all, especially cases like cross-compiling from Windows/Mac to Linux are impossible, and even Linux -> Linux is problematic.

* due to the fragmentation, there are many major package managers to support, like yum, apt, pacman, etc. support of each new package manager requires additional baby-sitting (and possibly modifications to the conan itself, to add new distributions like Gentoo, Slackware, etc.)

* within single package manager, there are various architectures to support - and this adds an additional level of scalability problem.

* even within single package manager, various distributions may have different names of packages. for instance, Fedora, RHEL and CentOS are known to have different package names for certain packages.

* even within single distro, packages are deprecated and removed from time to time, so it requires continuous maintenance with every new release.

* the code to use system package manager in recipes looks ugly - it has lots of nested conditions.

* due to the various bugs in distributions, packages often conflict with each other, making it impossible to build a conan package on the system. for instance, on Ubuntu, 32-bit python-development package conflicts with 64-bit one.

* installing additional software from system package manager, may confuse conan build. for instance, libprotobuf-dev might be installed as a dependency of some other package, then it may override protoc executable from the conan package, which is not a desired effect.

* builds are no longer truly reproducible, because various distros have different versions of system packages, therefore behaviour may change after recompiling on another distro.

* additional point of failure and additional dependency on the availability of system package manager registry - e.g. apt repositories are known to have downtimes. given the fact conan builds on CI are very unstable for other reason, it would be nice to have to remove that factor.

* conan builds are no longer self-contained - installation of executables/libraries will not run or link because of dependency of system libraries, which might not be available on another machine.

* use-case simply doesn't work on systems with no system package managers (e.g. source-only distros, like LFS systems), or systems with poor support for package managers or lack of packages (e.g. Windows).

* in order to use conan package, now it's needed to use pkg-config to locate system requirements (or locate them by other means, some system requirements don't even have .pc files!) - this adds additional complexity to the integration of conan package into the build system.

right now, the following conan packages are heavily based on system requirements:

OpenGL-related: GLU, GLUT, GLEW, GLFW, GLM, etc.
ffmpeg: X11, vaapi, vdpau
SDL2: X11, wayland, mir, audio APIs (pulseaudio, alsa, esound, nas, jack, etc.)
wxWidgets: GTK

I will print this and put it on a board

[jgsogo]

opengl recipe with a system approach has been out for a while: https://github.com/conan-io/conan-center-index/blob/master/recipes/opengl/all/conanfile.py, no big issues so far and it is been consumed by some other packages.

Should this approach be added to the docs?

[uilianries]

Yes! I think it's a good reference. I usually send its link when users ask for system requirements.