Conan-center-index: [question] Projects with an in-source recipe

[Minimonium]

What's the strategy for the inclusion of such projects? For example - https://github.com/catchorg/Catch2

[uilianries]

I would say keep all them as third-party and deprecate in-source recipes.

pros:

• author don't need to maintain the recipe/ci/upload
• the project don't need to receive an extra step on their ci script (travis, ...)
• any issue related to the recipe/conan the author don't need to care about or look for help

cons:

• the author can't manage the recipe as owner (less power)
• the license won't follow the same from the project
• the author will need to observe an external project to know about the recipe

[Minimonium]

There is one moment of transitive Conan dependencies. For projects that don't have dependencies having a fully separate recipe is okay. But what about projects that use other Conan projects? They'd like to have the recipe in-source.
Would the advice be to submodule the recipe, for example?

[uilianries]

As these new packages are uploaded to Conan Center, the author will need to update the recipe only, with new references.

If some author want to keep in-source recipe, I see no problem. However, someone will need to maintain the recipe.

[Minimonium]

The problem here is the duplication of the recipe, which is unfortunate. I think it's worth it to think about it. :thinking:

[ghost]

I think having 2nd in-source recipe is useful for "git" versions and maybe other repositories.

[Minimonium]

Related https://github.com/conan-io/conan/issues/4734

[KerstinKeller]

I feel a bit ambiguous about the in source vs out of source recipe topic.
But I looked at the structure of the repo (because internally we keep almost the same structure), and the build logic for Jenkins. Unfortunately that one is hidden in a library :disappointed:

Anyways, depending on how the build logic works, it could certainly be suitable to integrate libraries with an insource recipe by specifying urls instead of folders in the config.yml file:

Take the boost example:

versions:
    1.70.0:
      folder: all
    1.71.0:
      folder: all

If boost had an insource recipe you could do:

versions:
    1.70.0:
      url: https://github.com/boostorg/boost/commit/8f9a1cf1d15d262e09c16a305034d8bc1e39aca2
    1.71.0:
      url:: https://github.com/boostorg/boost/commit/afb333b7c5101041f0280b2edf155c55114c9c95

And the job builder would checkout that commit and continue from there.

[SSE4]

the author can't manage the recipe as owner (less power)

this is questionable argument, as author may decide to force push to his repository, or even remove it completely, but to keep packages stable and always available it might not be desired.

[uilianries]

the author can't manage the recipe as owner (less power)

this is questionable argument, as author may decide to force push to his repository, or even remove it completely, but to keep packages stable and always available it might not be desired.

I mean, the author is not able to force a push here in conan-center-index, or use any admin power.

[SergiusTheBest]

Can we make an out-of-source recipe that references in-source recipe?

[uilianries]

Do you mean alias ?

[SergiusTheBest]

@uilianries It looks very close if it could reference a github repo instead of another package.

[Minimonium]

So, friends, to sum up.

Why the out-of-source strategy for the Index important:

• Any patches could be applied to _all_ recipes at once for best practices in a unified manner. Repo maintainers are responsible for the quality of recipes.
• Any security updates could be applied to everything at once too.
• Any contribution process is visible to all and can get help from the community in a simple manner.
• Importantly, Conan plans breakages (Conan 2.0). The Index allows migrating the whole repo at once.
• Library authors don't need to know about the recipe or the Index.

Moments to think about:

• Release is bottlenecked. The process of publishing the new version after the upstream requires a manual effort and audit. I'm sure it's not an unsolvable problem, there are ways to work around it. For example, granting the library owners the fast-track rights or maybe making something with the Github API to allows authors to add a command into their CI to automatically make a PR with a new version. Would be pretty cool.
• Duplication. I think that's a problem only for projects that use Conan as a vehicle to get dependencies. So the conanfile.txt and the same functions of the conanfile.py. Right now, the whole machinery for calling the source(), build(), package() is used to make a package. My opinion on it is that the upstream package should back-port all best practices from the Index, which means that leaving the window for referencing the out-of-Index recipe compromises the quality of the recipe. More at https://github.com/conan-io/conan/issues/4734#issuecomment-531577918

The out-of-source way is clearly the way to go for the long-term maintainability of the repository. The goal is to figure out how to find a way to solve some shortcomings with it.
It would be good to have some understanding of what's Conan's plan in the near future to address moments with it.

[ghost]

* Library **authors don't need to know** about the recipe or the Index.

Doesn't that kinda defeats the point? For my libraries Conan is the only supported way to consume them because Conan is the only sane package manager. Using anything else is disaster.

[danimtb]

Generally, I agree with the summary of @Minimonium and believe this repository and build service is the best choice to have a strong conan-center and better community. Not to say that the contribution process is transparent and the curation is done community wise.

On the other hand, we knew the in-source approach was going to be compromised in the early stages and authors having the recipes in their projects (like @Lyberta) could end up with a duplicated recipe.

However, this is something we will solve in the near future and approaches like the one proposed by @KerstinKeller could fit nicely. Will need to look into this issue and discuss different approaches further so we come up with the right solution for the different scenarios.

[Minimonium]

@Lyberta The point is to have a large number of libraries to consumers. If authors want to engage with the Conan community - great. If they don't - doesn't matter. :slightly_smiling_face:

[ghost]

@Minimonium And what if they use other horrible package manager such as APT, DNF, Pacman, vcpkg?

[Minimonium]

@Lyberta We migrate their dependencies and patch them out of their hard bindings to specific package managers if needed. But it's not useful to fight windmills. :slightly_smiling_face:

[Minimonium]

Users brought up an additional use case of when in-source recipes are important: workspaces
https://github.com/skypjack/entt/issues/380#issuecomment-569468194

Removing the recipe from EnTT repo will make it a bit cumbersome for me to test things eg. the experimental or other branches.

Having the recipe there allows me to add the repo as a part of a Conan workspace, without it I can't do that.

What can we do about it?

[memsharded]

Thanks for this new bit of information.

I see the issue, but I think that at the moment, I would still recommend keeping an in-source recipe for development, while submitting a recipe for conan-center-index. It might be a bit of duplicate work sometimes, but to summarize some of the above, it also helps in other ways:

• it is possible to have a very simple recipe, e.g. for header-only packages for conan-center-index, while keeping a more complicated one that can run tests and other stuff (like workspaces) in-source.
• can streamline the contribution of many versions at the same time, and specially to backport package fixes to all versions very easily, highly simplifying the work of contributors and collaborators.

Thanks for the feedback, even if it seems difficult to solve the issues for both worlds at the same time, we keep these things in mind.

[Croydon]

While I would also like the idea that projects keep a conanfile in-source for dev versions, I think that reality is slowly catching up.

For people better familiar with Conan it might be clear which Conan recipe you want to use, when both CCI and in-source recipes exist, but for the wast majority of people this seems to be confusing. People start reporting bugs in which it isn't clear what they actually use. Even worse when the recipes diverge from each other. Maintenance and support is hell. (I have realized this e.g. with Google Benchmark)

Some upstream projects also removed their own conanfile and now refer to CCI. Latest example is probably Paho, see #1395

I also can't think of any solution to refer/link from CCI to an in-source recipe that would really work. Working across so many repositories, platforms, systems, workflows to contribute would be living hell too. And "quick fixes" would probably take pretty long.

Personally, my current point of view is, that in-source recipes only pay off if the project itself is so specific that there is no general interest in packaging it, or if legal reason don't allow to add it CCI or something exceptional along those lines. In any other case, adding a recipe only to CCI is probably much easier to discover, maintain and support.