Conan: [RFC] Package recipe declare remotes to auto-add for requirements

[jinq0123]

I created a gRPC package gRPC/1.4.2@jinq0123/stable and uploaded it to bintray https://api.bintray.com/conan/jinq0123/test. My grpc_cb library requires this gRPC package. How can I write this requirement in conanfile.py?

class grpccbConan(ConanFile):
    url = "https://github.com/jinq0123/grpc_cb"
    # How to require a package on https://api.bintray.com/conan/jinq0123/test ?
    requires = "gRPC/1.4.2@jinq0123/stable"

Or how to upload gRPC/1.4.2@jinq0123/stable to conan-center?

[memsharded]

The package requirements declaration is decoupled from the origin. This is a typical orthogonal design, the same as the #includes, they say what they need, but it is the build system the one providing the include-paths to find them, so the same code can work in different computers.

So, in the package recipe, you cannot specify a remote, because that would break that decouple and users wouldn't be able to use the remotes they want. A typical approach in conan is users copying and storing packages in their own remotes (like in-house remotes, so they don't depend on external infrastructure).

To use the packages from your remote, you need to add such remote:

$ conan remote add myremote https://api.bintray.com/conan/jinq0123/test 

You have more details about it in: http://docs.conan.io/en/latest/packaging/bintray/move.html

[jinq0123]

Whatever the design, orthogonal or non-orthogonal, it must be designed to make the usage simple.

In the #include case, conan makes it simple to set the include path. Without conan, I must set the include path for every dependency library.

And there can be some tool to make conan remote management simple, for example, using

conan install . --auto_add_remote

to add remote temporary for library that missed in the current remote list.

I suggest to set a default remote in connanfile.py, like:

class GrpccbConan(ConanFile):
    requires = "[https://api.bintray.com/conan/jinq0123/test] gRPC/1.4.2@jinq0123/stable"

[memsharded]

I am not sure that users would like remotes being added on the fly.
But I am open to this suggestion, lets see what other users think about it.

[promgamer]

I'm trying to set up a CI environment for our project (that uses conan) and I think this would be a good feature.

Right now, I've phase on the CI which adds the remotes we need, which is OK, I guess. Thing is, when we try to have incremental builds (for fast CI), adding a remote that already exists gives an error, which makes the build fail.

[memsharded]

@promgamer I think your suggestion is more in the line of the latest suggestion of https://github.com/conan-io/conan/issues/2618, which is a --force flag, though there are some things to polish there. Please follow conversation there.

In any case, I'd like to suggest that for CI (and for developers) to use the conan config install feature, with a remotes.txt file, that will completely redefine remotes. Even for incremental builds, it is fine, it should be a very fast operation, and it ease the task of better reproducible environments.

[jinq0123]

Is is hard (or impossible) to find all the remotes of a package which depends on many other packages. For example, grpc-lua/0.1@jinq0123/testing must add 5 remotes to install it successfully, because it depends on gRPC/1.8.3@inexorgame/stable which depends on protobuf/3.5.1@bincrafters/stable and other packages.

[allquixotic]

As a veteran Maven user, I was expecting something equivalent to the repository behavior of Maven in Conan. Here's how it works with Maven:

• Commonly used Maven repositories include Maven Central (like conan-center I guess), Bintray, a few vendor-specific ones like Red Hat's repository, and of course in enterprise environments, their own local Artifactory or Nexus IQ server.
• Default Maven without any configuration will try to pull from Maven Central only.
• Authors of a project's pom.xml (basically does what Conanfile does) can specify a repository, or multiple repositories, to use for the current project. Dependencies to be acquired will first be taken from a repo specified in the project (if they exist there), before Maven Central or any user-defined repositories are searched.
• Users can "redirect" repositories by creating a repository of type mirror in their local ~/.m2/settings.xml file. Basically what this does is override any previous definition of a repository of a given name. For example, if the default repo named central points to Apache's Maven Central instance, you can define a mirror of central in your local ~/.m2/settings.xml to point to an Artifactory server on your LAN. Enterprises do this because they usually have very strict firewalls that don't let random Java processes make outbound Internet connections, or require annoying proxy setting config first. They can also control which dependencies are made available this way.

So with Maven, in a case like jinq0123's situation, for his gRPC, he'd define a series of repository statements in his gRPC pom.xml, pointing to bintray, bincrafters, etc. If the user building from his pom.xml had public Internet access, then gRPC and its dependencies would preferentially be downloaded from the repos specified in the gRPC POM, and the user would not need to customize or setup anything before running mvn package (the equivalent-ish of conan install .). If the user were in an enterprise environment, they could have custom repositories in ~/.m2/settings.xml with <id> set to the same ids as jinq0123 chose for his third-party repos, but redirecting any downloads from them, to his LAN's Artifactory or Nexus mirror.

Another nice thing about Maven is that it's pretty easy, relatively speaking, to get your own software into Maven Central, assuming it's open source and you want it to be available to the public. The fact that seemingly all of the dependencies I care about for my own projects are not in Conan Center makes me wonder if packagers are feeling that Conan Center is too difficult to get into due to policy constraints. If that's the case, it should really be made to be no harder than uploading to Maven Central. Since so many packages are in Maven Central at this point, a typical Maven user very, very rarely ever even thinks about repositories -- 99% of the stuff you need is in the default repository, and if your enterprise mirrors Maven Central on your Artifactory/Nexus server, you're good to go.

Edit: I don't know what "reviewed/curated" specifically means (it's on the conan.io homepage), but if that implies that packages are declined to be included due to quality / need / impact judgments (like, "no one's ever going to use that" or "this package has no tests so we don't think it's reliable enough to be included"), that is a pretty bad thing, IMO. Maven Central does not judge the quality of packages, nor do they guarantee the functionality of them. The few constraints they do have are designed to ensure that dependencies' metadata is in a consistent state; that there isn't an artifact or group ID conflict (so that I can't just upload a random library as the latest version of Google Guava, for instance); and to make sure that uploaders are using a reasonable license.

Maven Central is also a bit decentralized in that there are multiple approved hosts that all get combined together to form the global Maven Central repository (they're technically rsynced together). There are a few common ones like Apache's own Maven repository for Apache projects, but then there's Sonatype OSSRH, which has very sensible but ultimately quite lenient guidelines for getting your stuff uploaded to Maven Central. OSSRH does not perform any testing for quality, functionality, or judgment calls like "is this really something we want?" before including packages in Central. This ensures a level playing field and makes the concept of Maven repositories, other than Central, a very rare thing to think about for developers.

So far it seems like Conan-Center is at least enough of a walled garden that very well-meaning packagers producing high quality work, like bincrafters and jinq0123, feel that it's less effort to chuck their things in a separate repository on bintray than to try and go through conan-center. Otherwise I imagine they would probably want their stuff in conan-center if it were easy to do so - I mean, who wouldn't? Not having to specify a bunch of remotes before you do your build is highly desirable from a user workflow perspective.

I apologize in advance if I've got the wrong idea about how open conan-center is to external package contributions. If it's indeed as non-judgmental and easy to upload to as, say, OSSRH, then that's awesome. Maybe I'm just unlucky then that all the packagers who produce packages I care about so far have chosen to use their own custom repos instead of conan-center.

Until Conan can address some of these issues, honestly, it's pretty annoying to use relative to its Java-land competitor (well, not really competitor, since Maven doesn't work well with C/C++ and Conan isn't designed to work with Java out of the box. But you get the idea.)

[matthew-d-jones]

What about the ability to add a suggested remote for each required package? If the package is not found in current configured remotes then Conan could at least return the suggested remote in the error.
I currently keep a list of remotes to configure in the README for each of my projects, the problem is that this can easily get out of sync with the real requirements of the conanfile. If suggested remotes could be added to the conanfile then at least everything is defined in one place.

[memsharded]

Hi @matthew-d-jones

Yes, that would be related to what is described in https://github.com/conan-io/conan/issues/2091.
Still we believe it shouldn't be in the conanfiles, but in another file, though transitivity has to be considered.

[groovyd]

We have run up against this issue a number of times mostly because people have stewn about a bunch of repos they expect everyone to add to their registry and then sometimes they have same named and version forks of other packages that are not always actually producing the same binaries for whatever reason. In this case it is hard to pin down the problem but it would be very nice if you could force conan to pull certain requirements from hard-coded repos in the conanfile itself as suggested by the op. While it is true this then places non-orthogonal aspects into recipes it can solve the user's problem without requiring them to construct private repos with further copies of said packages which only further inflates the issue for everyone else. The only way to fix this is for everyone to use the exact same registry of repos in the exact same order.

[Maddimax]

Allowing requires to specify the remote is something I wish for as well.

[ruipacheco]

+1

[muzammil360]

+1. I am trying to push a package to gitlab and it gives me the error that username is too long. (should be less than 51 char). Now for gitlab, I have to add a new project based remote. If only I can add it in conanfile, it will make my life simpler. But now I need to manually add that remote everytime I try to pull this. Furthermore, a new user in team will have no idea why he can't pull it unless he really knows what's happening.

[memsharded]

Hi all

This is a long standing request, and we have learned a lot since then. We have learned from many companies having projects with thousands of packages, and from open source users.

We can conclude that putting the remote information into recipes is a bad idea. It completely breaks possible ownership of packages (the flow that conan download + conan upload a package, to store a copy of it in your own repository, to isolate your builds, is an extremely popular flow). Packages should be able to be moved between repositories without problems. The "promotion" mechanism of package is a best practice is package management and devops, and this feature will violate it.

So I am going to close this feature request as "wont fix".

This doesn't mean that other approaches are possible. Of course the conan config install is the best current resource, it is extremely convenient and easy to manage, and it can manage and define remotes easily. This means, that you can easily have a "remotes.txt" file in your repo and do conan config install remotes.txt.