Conan: Allow promotion of packages in Artifactory

[memsharded]

Comes from https://github.com/conan-io/conan/issues/2414

Right now it doesn't seem possible to do a copy in Artifactory to promote a package from one user/channel to a different user/channel, it complains about the path. This is a very necessary feature, because otherwise users need to download tons of large package, do a conan-copy, and upload again, which is much slower and fragile.

[mdegel]

As an addition to the explanation from @memsharded, this would be the complete error when attempting to promote a package via copy directly in Artifactory (5.8.3):

Target path 'conan-server/user/package/version/channel/export/conanfile.py' for Conan files must match the source path'conan-server/user/package/version/test/export/conanfile.py' (excluding repo):
Target path 'conan-server/user/package/version/channel/export/conanmanifest.txt' for Conan files must match the source path'conan-server/user/package/version/test/export/conanmanifest.txt' (excluding repo):
Target path 'conan-server/user/package/version/channel/export/conan_sources.tgz' for Conan files must match the source path'conan-server/user/package/version/test/export/conan_sources.tgz' (excluding repo):
Target path 'conan-server/user/package/version/channel/package/package_id/conaninfo.txt' for Conan files must match the source path'conan-server/user/package/version/test/package/package_id/conaninfo.txt' (excluding repo):
Target path 'conan-server/user/package/version/channel/package/package_id/conanmanifest.txt' for Conan files must match the source path'conan-server/user/package/version/test/package/package_id/conanmanifest.txt' (excluding repo):
Target path 'conan-server/user/package/version/channel/package/package_id/conan_package.tgz' for Conan files must match the source path'conan-server/user/package/version/test/package/package_id/conan_package.tgz' (excluding repo):

Some Trivia, how and why channel promotion might occur:

• Users may deploy their conan packages for testing purposes, e.g. zlib/1.2.11@user/testing
• The buildserver continously builds packages from designated branches (e.g. master, release/*, ...). These packages may be quite big in size and build time, so a rebuild is usually not desired (e.g. toolchains or entire kernels). Also an unnecessary download/repackage/upload of this package would usually be avoided where possible.
• Built packages become part of a bigger software package and are tested as a whole. Testing a kernel+libs for example doesnt make much sense without any running applications due to unrealistic load.
• Assuming positive tests, the final software package would be released. When releasing this software, it would also be a practical option to promote all/specific contained conan packages e.g. from stable to release. This seems somewhat unnecessary at first, yet this promotion makes it far easier to distinguish between packages someone might need in the future (hotfix, release, debugging, ...) and packages that can be removed by a garbage collector or something similar. Garbage Collection on the other hand is quite important when releasing often and much (big packages). For example my company currently produces around 20-50 Gigabytes worth of artifacts per week, from which the most part usually isn't needed longterm (let's say we keep 10%). Storing all of it would be an excessive overhead, which is neither suitable for backups nor for storage management and package resolving on the conan server.

Therefore a simpler solution for promoting conan packages trough different channels would be highly appreciated.

[markgalpin]

Absolutely a valid use case.
https://www.jfrog.com/jira/browse/RTFACT-14165

A question for the community: I am considering removing the restriction on copy/move entirely. In this case you would be able to do what you want with a package but the metadata would NOT be updated. A possibility to cover both would then be to add a custom API (like we have for docker) for "promote" that will both move the artifact and update the extracted metadata. In theory conan could also then leverage this API if available. WDYT?

[markgalpin]

To clarify my early comment, we are considering three options:
Option 1: Leave behavior as is, and force all changes via a new REST API purely for conan promotion. This would be very unusual for artifactory, and I'd like people who like this one to also indicate their preference for 2 vs. 3 as well, but it does ensure that both the conan recipe data in the conanfile.py and the artifactory property metadata remains consistent with the package name.

Option 2: Copy/Move protects package name and version changes, but not user/channel versions. This is because the package recipe itself typically specifies name/version. Copy/Move changes that impact user/channel would NOT change the artifactory properties, users would have to do that manually as a separate step, but we could provide an additional REST API or create a user plugin to do this either automatically on copy/move or instead of copy/move [note that a user plugin using the Copy/Move command would currently NOT work, which is annoying and precludes some obvious workarounds]

Option 3: No protection at all. In this case you can copy/move as you please. This may result in conan objects in coordinates that do not agree with the package/version listed in the conanfile.py, as well as in incorrect properties. User would have to modify these things manually (Note that changing the conanfile.py implies changing the checksum) Again this gives maximum flexibility as it allows user plugins and other capabilities to do what they want for whatever reason.

For any of these options we could pair it with a new API (which could even be introduced via User Plugin to artifactory if need be) that would do a "conanPromote" which would allow changes of user/channel and adjust the artifactory metadata accordingly. In theory we could probably take it further and try to update the conanfile.py, but I do not favor any internal Artifactory API that changes the checksum of a package. That's not really a 'promotion' anymore, but an outright transformation, which generally speaking violates Artifactory design principles unless being executed in the context of a virtual repository (And at the moment this is by definition a local repository)

so WDYT?

[solvingj]

I hope a lot of the more experienced packagers think long and hard about this now and add their opinions , but it might take a release of one strategy for people to get their hands on before they can really make an informed opinion. It's an important topic, and to the best of my knowledge, you guys are really the first to come to the community and devote significant time/attention/discussion of creating a first-class package promotion strategy. It's great to see this kind of progress and initiative . Thanks!

So far I favor number 2. When we promote, we generally just want to change the channel, and I can see wanting to change the user. I am sure there will be countless ideas and use cases that want the flexibility of 3, but I don't think it's supportable.

[e-shreve]

I also favor number 2. I don't personally have a use case for changing the version or package name. (I'm using conan for deployment within my company.)

[sorf]

I'll add another vote for option 2 - the ability to change a user or channel directly on the server.

I'm not sure if this was covered by another issue or is implied by the current one, but the ability to change the user or the channel should be atomic - all the packages under a package-reference change at the same time.

This would be so that packages that depend on the current one and use a version range that resolves to the latest build of it, do not accidentally pick up the build/upload in progress one. In my use case, the Jenkins build fails if a dependency package is missing, it doesn't attempt to build it. Thanks!

[fedterzi]

I would favor number 2 as well, I think the most requested use case is to just change user and/or channel but not package name/version.

Also, is there an ETA for this feature? It would be really helpful to us.

[rkibria]

Another vote for number 2 here.

[thorsten-klein]

Another vote for number 2 here.

[lasote]

I'll try to get an ETA today. Thanks all for the feedback and interest.

[lasote]

Sorry, I didn't get yet an ETA. We need the help of @markgalpin here ;)

[joechrisellis]

Another vote for #2 here -- any news on an ETA? Cheers :)

[lasote]

ping @markgalpin

[mdegel]

Another vote for number 2. Like @e-shreve I dont't think there should be any suitable use cases for changes in version or package name in the future. I would even go as far and say you wouldn't usually need to even change the user. However changes in the channel should be rather common.

[nikola-kocic]

If option 2 is implemented, would changing versions work if version is not specified in conanfile.py?

Use case would be to first build and publish to testing channel and random patch version (for example timestamp), with version supplied in command line. Then, if packages creation and publish succeeds for all platforms (with build parallelized to multiple machines), CI would generate and tag new version and promote packages to the stable channel and the tagged version.

[lasote]

Please @danimtb check if in the conan-dev snapshot of Artifactory this is working or not.

[danimtb]

Seems this is still not possible. When you change user or channel artifactory-dev still reports an error:

[danimtb]

Hi! I tested this in the feature branch of Artifactory and the promotion is now completely allowed.
It will be available soon in an Artifactory release.

Cheers!

[danimtb]

The Artifactory release with this feature implemented will be 6.9.0

[GordonJess]

Hi @danimtb,

Did you test that the promoted packages can then be resolved by the conan client?

I have moved some packages to another channel using the Artifactory UI, but the client cannot resolve them:

PS C:\my_repo> conan search -r my-repo
Existing package recipes:

my-dependency/1.0.0@user/channel
my-dependency/1.0.0@user/new_channel

PS C:\my_repo> conan create . user/channel
ERROR: Error loading conanfile at 'C:\my_repo\conanfile.py': Version range '~=1.0.0' from requirement 'my-dependency/[~=1.0.0]@user/new_channel' required by 'python_require' could not be resolved

It works fine if I do conan copy and upload again.

[danimtb]

HI @GordonJess,

I tried again with Conan 1.18.2 and Artifactory 6.11.3 and it worked fine:

I promoted the kk/1.0@us/chpackage in the same repository to a different user and channel kk/1.0@other_us/other_ch (I used the "copy" option in the UI).

The search worked fine:

$ conan search * -r my-repo
Existing package recipes:

kk/1.0@other_us/other_ch
kk/1.0@us/ch

Then I installed the new package and it worked:

$ conan install conanfile.txt

kk/1.0@other_us/other_ch: Not found in local cache, looking in remotes...
kk/1.0@other_us/other_ch: Trying with 'conan-center'...
kk/1.0@other_us/other_ch: Trying with 'my-repo'...
Downloading conanmanifest.txt: 100%|##########| 58.0/58.0 [00:00<00:00, 19.4kB/s]
Downloading conanfile.py: 100%|##########| 553/553 [00:00<00:00, 61.6kB/s]
kk/1.0@other_us/other_ch: Downloaded recipe revision 0
Version ranges solved
    kk/*@other_us/other_ch versions found in 'my-repo' remote
    Version range '~=1.0.0' required by 'conanfile.txt' resolved to 'kk/1.0@other_us/other_ch' in remote 'my-repo'

conanfile.txt: Installing package
Requirements
    kk/1.0@other_us/other_ch from 'my-repo' - Downloaded
Packages
    kk/1.0@other_us/other_ch:6cc50b139b9c3d27b3e9042d5f5372d327b3a9f7 - Download

kk/1.0@other_us/other_ch: Retrieving package 6cc50b139b9c3d27b3e9042d5f5372d327b3a9f7 from remote 'my-repo'
Downloading conanmanifest.txt: 100%|##########| 59.0/59.0 [00:00<00:00, 14.7kB/s]
Downloading conaninfo.txt: 100%|##########| 438/438 [00:00<00:00, 88.1kB/s]
Downloading conan_package.tgz: 100%|##########| 63.0/63.0 [00:00<00:00, 3.71kB/s]
Decompressing conan_package.tgz: 100%|##########| 63.0/63.0 [00:00<00:00, 21.1kB/s]
kk/1.0@other_us/other_ch: Package installed 6cc50b139b9c3d27b3e9042d5f5372d327b3a9f7
kk/1.0@other_us/other_ch: Downloaded package revision 0
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo

Here ypu have the contents of the conanfile.txt with the version range used:

[requires]
kk/[~=1.0.0]@other_us/other_ch

Did you use a different method to "promote" the package? Maybe through the Artifactoy API or with jfrog CLI?

[GordonJess]

Nope, I used the same method, and with the latest version of Artifactory.

During the copy I did get error messages for some of the larger packages, about the timeout of a file lock or something (sorry I don't have a screenshot on hand). The packages looked ok after, but maybe something got corrupt..!

[danimtb]

Well, in my case it was a small package.... I opened a new issue with your experience to better track the details. Please continue the conversation there: https://github.com/conan-io/conan/issues/5779 Thanks