Conan-center-index: [question] Consistency of settings in tools only packages

Created on 9 Oct 2020 · 8Comments · Source: conan-io/conan-center-index

As of 2020/10/09, here are the settings contributing to package_id in pure utilities packages:

| Utilities only packages | settings | settings contributing to package_id |
|-----|----------|----------|
| 7zip | "os", "arch", "compiler" | "os", "arch" |
| autoconf | "os", "arch" | "os", "arch" |
| automake | "os" | "os" |
| b2 (portable) | "os", "arch" | "os", "arch" |
| b2 (standard) | "os", "arch" | "os", "arch" |
| cc65 | "os", "arch", "compiler", "build_type" | "os", "arch" (not with Visual Studio), "compiler", "build_type" |
| cccl | "compiler" | None |
| cmake | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| depot_tools | "os_build" | "os_build" |
| doxygen | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| flatc | "os_build", "arch_build" | "os_build", "arch_build" |
| ftjam | "os", "arch", "compiler", "build_type" | "os", "arch" |
| gettext | "os_build", "arch_build", "compiler" | "os_build", "arch_build" |
| gnulib | "os_build", "arch_build" | "os_build", "arch_build" |
| gperf | "os_build", "arch_build", "compiler" | "os_build", "arch_build" |
| guetzli | "os", "arch", "compiler" | "os", "arch" |
| jom | "os" | "os" (Windows only anyway) |
| kcov | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| lemon | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| lzip | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| m4 | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| make | "os_build", "arch_build", "compiler" | "os_build", "arch_build" |
| makefile-project-workspace-creator | "os" | "os" |
| meson | None | None |
| mozilla-build | "os_build", "arch_build" | "os_build", "arch_build" |
| msys2 | "os_build", "arch_build" | "os_build", "arch_build" |
| nasm | "os_build", "arch_build", "compiler" | "os_build", "arch_build" |
| ninja 1.9.x | "os_build", "arch_build", "compiler" | "os_build", "arch_build" |
| ninja 1.10.x | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| nodejs | "os_build": ["Windows", "Linux", "Macos"], "arch_build": ["x86_64"] | "os_build", "arch_build" |
| premake | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| ragel | "os", "arch", "compiler" | "os", "arch" |
| re2c | "os", "arch", "build_type", "compiler" | "os", "arch", "build_type", "compiler" |
| scons | "os" | "os" |
| strawberryperl | "os_build", "arch_build" | "os_build" (only Windows anyway), "arch_build" |
| swig | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler", "build_type" |
| verilator | "os", "arch", "compiler", "build_type" | "os", "arch", "compiler" (not the version), "build_type" |
| waf | "os_build", "arch_build" | "os_build", "arch_build" |
| winflexbison | "os", "arch", "compiler", "build_type" | "os" (Windows only anyway), "arch", "compiler", "build_type" |
| yasm | "os_build", "arch_build", "compiler" | "os_build", "arch_build" |

Packages are not really consistent with each other. What should be the proper settings?

(I would like also to centralize discussions about settings in utilities packages in this thread)

Docs question

Source

SpaceIm

👍5

Most helpful comment

Hi! We've been talking internally about this issue again, and these are the conclusions:

Note.- We are always talking about _installers_, packages that provide an executable that is used as a tool. If, at some point in time, that package provides a library and it is also used as a regular requirement it will no longer be considered a _installer_.

A bit of context

Conan v2.0 will likely be using the new approach with the two profiles/contexts (build and host) by default. Some details are to be defined, but it looks like it is the only way to handle tools properly as build_requires. Old os_build and arch_build seems not to be flexible enough and make recipes harder to reuse in _host_ contexts.

As said above by @ohanar , typically a user will use build_type=Release in the _build_ context, so for sure we need to provide those binaries. But, as @madebr has pointed out several times, sometimes it is useful/needed to debug a tool running in the _build_ context (attach the running process and debug step by step).

Proposal

We need to list all the required settings in the recipe like any other regular package: this is needed in order to generate the binary matching the configuration provided in the CLI
```
settings = "os", "arch", "compiler", "build_type"
```
```
conan install cmake/3.18.2@ -s build_type=Debug  --build=cmake  # Generates Debug binaries
conan install cmake/3.18.2@ -s build_type=Release --build=cmake  # Generates Release binaries
```
Of course, if the package is not compiling from sources, some settings are not needed (like any other regular package).
When debugging a tool, usually one needs to switch from Debug to Release, it is not enough to be able to compile the package in Debug, we need to switch from one to the other and it is not affordable to compile the binary each time (if they have the same package_id, the binaries are overriding). That's the reason we agree we should preserve the build_type in the package_id()
We remove the compiler in the package_id(self) method, we think no one finds it useful to have 20 different versions of CMake just because the package_id is different for every different compiler version. One of the value propositions of Conan is that you can _remove_ some combinatorial from the settings and reuse the same binary for different configurations.
```
def package_id(self):
   del self.info.settings.compiler
```

Future

In ConanCenter we will generate and provide binaries only for the Release configuration of these tools, that movement will require two developments:

we need to implement in the CI something like https://github.com/conan-io/conan-center-index/issues/2813 to be able to select the packages to compile. Basically, we will remove every package-ID corresponding to a Debug build (assuming the corresponding Release one is not raising a ConanInvalidConfiguration).
recipes for these _installers_ will include a line to declare that the Release package is compatible with the build_type=Debug. This will be achieved via compatible_packages(docs). By that time it should be a stable feature.

Hope this makes sense. Thanks for all the effort you are dedicating to the ecosystem! 🙌

jgsogo on 22 Oct 2020

👍2

All 8 comments

@jgsogo commented some time ago to use full settings (os, arch, build_type, compiler), but change the package_id as we need. Seems be reasonable, we will need to add one more line, customizing the package id, but, we will have a pattern for settings.

uilianries on 9 Oct 2020

Croydon on 9 Oct 2020

My 2c:
The package_Id of tools-only packages should include os, arch, compiler, build_type (where applicable) because they also might need to be debugged.
These recipes should be as tools and c3i should know to only build Release+MT for them.

madebr on 9 Oct 2020

👍1

tools are tools and should be be debugged as part of a build.

It is an unneeded complication to build N configurations for ninja for example while just Release is sufficient/Preferred.
It does not sounds correct to use Debug ninja to build a debug configuration of a library.

boussaffawalid on 20 Oct 2020

My 2c:
In an ideal world, package_id of tools should include all 4: arch, build_type, compiler, os. Using split build and host profiles, as will likely be the default for conan 2, this shouldn't really be an issue, as build profiles will almost always have build_type=Release.

That said, this doesn't work great for the status quo, so I would be inclined to remove build_type and compiler in package_info.

ohanar on 21 Oct 2020

Keeping only os and arch is the easy solution, but is not possible once a build requirement also provides libraries.
e.g. python.

For example:
Python can be used as a build requirement for executing python script (e.g. to fix a certain version)
But can also be used as a requirement for creating python modules or embedding the interpreter.

So keeping only arch and os is a short-term solution. The problem will not go away.

madebr on 21 Oct 2020

Why not keeping only arch and os, and If a requirement provides libraries, and extra package can be used. Something like

python: tools only
python-dev: also include dev libraries

boussaffawalid on 22 Oct 2020