Conan: Automatically upload package to release-channel-specific remote?

[ghost]

Hi,

In my setup I have my own artifactory server running with two artifactory conan repositories. One conan repo shall contain all package that use "stable" channel, the other repo shall contain all other channels (e.g. "testing").

I'm referring to the documentation, where conan upload is specified as follows:
"If no remote is specified, the first configured remote [..] will be used."

I would ask for following behavior:

• A user creates a package: myPackage/0.1@orga/stable => conan upload shall automatically push it to the remote "stable" repo
• A user creates a package myPackage/0.1@orga/testing => conan upload shall automatically push it to the remote "testing" repo
• A user creates the testing package, wrongly calls conan upload -r stable => artifactory returns an error that release-channel/maturity-level is not matching. Requires changes in artifactory?
• This behavior should be configurable

What do you think?

Thanks and have a nice day :)

[memsharded]

There are two different things here:

• First, permissions like "you cant upload this to this server", should never be a client side feature. I think you can already define permissions in Artifactory so it will reject packages not matching some patterns. cc/ @markgalpin

• Regarding automatic uploading, conan is kind of similar to git decentralized model. The remote name is just an shortcut to the real URL, so it can barely be used for any logic. This sounds like very specific for your own setup, but I don't see it that relevant for the wider conan community. It would be like making git automatically to push to different remotes depending on the branch name. I strongly suggest using the explicit way, that is very robust, clear and understandable for users:

$ conan upload myPkg*stable --r=mystablerepo --all
$ conan upload myPkg*testing --r=mytestingrepo --all

Cheers!

[Aalmann]

@memsharded
I think it is not a that specific use case. We are doing the same by having one repo for all users (consumers) which only contains stable packages and one for developers and CI which contains "testing" and "integration" channels. We currently have this logic in CI scripts.

However, comparing to other package managers: Let's take maven. You can specify a repo for SNAPSHOT and one for RELEASE builds. Maven is selecting the right one by evaluating the version "x.y.z-SNAPSHOT" or "x.y.z".
So IMHO conan should do the same. Probably an additional and optional remote flag would be helpful.
Something like conan remote add my-stable-repo https://my-stable-url True channels="stable" and conan remote add my-private-repo https://my-private-url True channels="testing,integration,nightly".
This could be auto evaluated by the upload command (when no remote is given) depending on the given channel.

Best
Aalmann

[memsharded]

@Aalmann thanks for the feedback.

I understand the use case, migth make sense, though it must be carefully analyzed. I am opening the issue for feedback, and discussing with the team.

[jokram]

Hello,

I agree to the described use case. We have to protect release versions of packages so that they cannot be deleted or overwritten. But for intermediate versions like CI testing we want to be able to delete them after a certain time to clean up the storage (= Artifactory) behind to save storage costs.
And in the context of conan the channel seems to be appropriate to distinguish between release versions and intermediate versions.
Also in the context of git we have taken counter actions using protected branches on GitHub.
So also there a user could locally create new commits on a branch and when pushing to GitHub gets the feedback that the branch is protected. I see this very similar to what we need for conan, but instead of protecting branches I would need to protect channels.

Background:
In the automotive industry we have to fulfill regulations like "Automotive SPICE". There we must guarantee that a release can be re-build and that we have taken counter measures to protect releases against deletion or modification.

[memsharded]

Thanks for the feedback @jokram !

So also there a user could locally create new commits on a branch and when pushing to GitHub gets the feedback that the branch is protected. I see this very similar to what we need for conan, but instead of protecting branches I would need to protect channels.

I think this is relevant, the permissions and the control of such thing is done server-side. Doing things client side is fragile and always prone to human errors. So the most important feature here is just to configure permissions in the respective servers, so this can't happen. This can already be done with Artifactory, and is definitely the way to go.

So any automation client-side, is basically a small user-convenience, so developers don't have to think if the should conan upload -r=testing_repo or conan upload -r=prod_repo. As we see it, the second case is already not the best practice, developers shouldn't be able to upload to the production or releases repo, only CI can do that.

This is why we are looking into this feature, it will require some code and complexity into conan client, but the gain is relatively small, as configuring it properly server-side is what should be done in any case.

[Aalmann]

Hi,

@memsharded :
Can you provide a link or description how to configure an Artifactory conan repo to just accept "stable" packages? I was unable to find a solution with google and tried a lot of different include patterns while setting up the repo but without success.

@jokram : (a bit off topic but it's related to the "do not delete stable packages" and recipes and packages in remotes)
There is also another issue in "Automotive SPICE" context. There was already a discussion in #1249 because of the "chicken egg" problem concerning scm information. Depending on your workflow either you have create a release tag first or you have to create a release branch and finally put this information into the recipe. Using a tag could only really work if you are using a dedicated scm repo for recipes (first tag the product/module, then update the recipe), because otherwise this information can't be added to the recipe which was already tagged (or you guess the tag first, commit/push the change and create the "real" tag). Using a release branch requires the current revision information for the created package which has to be stored beside the package for each platform built (CI build on different machines for different compiler/os/arch settings could end in a non-parallel build, because of availability of build slaves/agents). If you also want to store the test results in scm (e.g. the release branch to have them together with the code) you have to use a branch instead of a tag. But then, using <conan command> --build=myReleasedModule could end in a different binary package.
However, we have a small scm helper for that to put at least the current scm information into the binary package to have a detailed information about the revision, url, branch etc. from which the package was created. But IMHO this whole workflow should be part of conan to be able to create always the same package (either by creating from recipe or by rebuilding a package from remote recipe).
@memsharded :
Do you think it is worth to start a new issue/discussion about scm in general, all scm use cases and conan long term strategy/desired workflow concerning this issue? I have seen that you already created a PoC in #2371 which might be something in the same direction.

Best Aalmann

[memsharded]

Hi @Aalmann

Yes, the process would be something like:

• Create the user that will upload, with write permissions, etc.
• Create a "Permissions Target", and define a "Include Pattern" as **/stable/**.
• Select the repository for that Permission Target, and assign the previous user
• Use that user to try to upload non stable packages

Note: the pattern is basic, if some package is called "pkg_stable/1.2@user/testing", it will be uploaded, but I thing the idea is there. Please try and tell me.

Also, regarding the SCM thing, I think the best is continue conversation in: https://github.com/conan-io/conan/issues/2371 at the moment.

Thanks!

[Aalmann]

Hi @memsharded

it works with **/stable/** (the only combination I didn't try last time) both as permission target and directly in repo settings. Many thanks for that hint.
Btw. the package "pkg_stable/1.2@user/testing" will not be uploaded because "stable" is surrounded by "/". So only "stable/1.2@user/testing" or "whatever/stable@user/testing" or "whatever/1.2@stable/testing" matches, which is good enough for me.
The include pattern thing should be part of any conan or Artifactory documentation. And/Or Artifactory should provide a more "conan-specific" configuration field, because this pattern is not really obvious. ;-)

Regarding SCM: Ok, I will continue in #2371
Thanks.

Best Aalmann

[memsharded]

it works with /stable/ (the only combination I didn't try last time) both as permission target and directly in repo settings. Many thanks for that hint.

Great, you are welcome!

The include pattern thing should be part of any conan or Artifactory documentation.

Yes, probably not explicitly about the include pattern, but some documentation about good practices, common patterns and flows using conan with Artifactory: https://github.com/conan-io/docs/issues/625

[memsharded]

This is also related to: https://github.com/conan-io/conan/issues/2091