Conan: [question] Managing package revisions on Artifactory

[ericriff]

Hello,
I have a question regarding the management of package revisions on my Artifactory server. The way we're using conan consists on a Jenkins pipeline that builds a bunch of recipes and uploads them to artifactory. We have revisions activated because we're using the recipe revisions as locks for the dependencies, which works great for us.
The issue that I'm facing is that my Artifactory server is getting polluted with old packages, because sometimes the package revision change, resulting on a new package being uploaded.
Is there an tool to remove old package revisions? Like the --outdated flag on conan search?
Or a way to tell conan upload to override the package if the RREV is the same?

Thanks!

• [x] I've read the CONTRIBUTING guide.

[datalogics-kam]

I was about to open an issue as well; I just had to go save our Artifactory server from running out its datastore (some of our packages are big). We also run nightlies and the RREV/PREV combo gives us an explosion of old packages.

What I'd like to have is a conan purge command, which would have the ability to:

• Purge old RREVs (keeping the latest)
• Purge old PREVs of the latest recipes.
• Be able to iterate over wildcards like conan search

Example use cases:

$ conan purge -r conan-local "*@datalogics/nightly" --recipes
$ conan purge -r conan-local "*@datalogics/nightly" --packages
$ conan purge -r conan-local "bigpackage/1.0.0@datalogics/nightly" --packages
$ conan purge -r conan-local "bigpackage/1.0.0@datalogics/nightly#<some RREV>" --packages
$ conan purge -r conan-local "*" --recipes # really clean up everything

[ericriff]

That's exactly what I'm looking for.

[memsharded]

This might take a while to consider, as we will probably be discussing to incorporate this into the command line for Conan 2.0. In the meantime, you can get the revisions information with:

$ conan search zlib/1.2.11@ --revision -r=conan-center
f3b17fb16cd1b7fbc969400a2ba1bc08 (2020-04-16 10:21:36 UTC)
ddccdddea098293f5202c5e8eb29967b (2020-03-13 17:07:40 UTC)
...
bf8cd01f8b018a3c0871562ae5251bff (2019-09-10 07:54:59 UTC)
c42d82150723a4bb1baca4757fb939ed (2019-08-30 13:32:07 UTC)

And for packages:

conan search zlib/1.2.11#f3b17fb16cd1b7fbc969400a2ba1bc08:ff82a1e70ba8430648a79986385b20a3648f8c19 -r=conan-center --revisions
Revisions for 'None' at remote 'conan-center':
69e8d72ae2418f7626e21529b3d00bd3 (2020-04-16 10:21:45 UTC)

The --json=file.json might also help to automate this together with conan remove calls. So it seems that a couple of python scripts for the admins of the Artifactory repository should be doable without big problems, so you can start doing that maintenance already.

[memsharded]

Just in case, possibly a good practice is to use more than one Artifactory repository. We are recommending it for CI, for example, where most of the builds are uploaded to a temporary "tmp" Artifactory repo. The purge policy there could be automated for example with an Artifactory plugin that removes artifacts older than a given time, automatically.

Also some notes for ideas for implementations and our future selves:

• It might be a bit dangerous to hardcode purge policies based only on just the "latest" revision available. That might break lots of build-infos or lockfiles using previous revisions.
• Maybe the solution is to provide a better way of chaining "search" results and piping them into "remove" command, instead of kind of building the same logic inside (because the next feature request is "I want to run the purge, but without really purging, only telling me what is to be removed", which is basically a "search" command)
• It seems that policies based on time, not just the latest, will be requested.
• It might be that the best approach is to do a lockfile or buildinfo purge behavior.

[datalogics-kam]

ETA: Thanks for taking this under consideration.

that removes artifacts older than a given time, automatically

That sounds great, but actually wouldn't work for us. If the nightlies are failing for a few days, we need to be able to have our devs get packages (we have things that are way too big to build in the cache). If we set the timeout to a week or two, we run out of space. Also, I've been getting the strong vibe that Conan package management should be done from the client.

In other words, we need the ability to delete things because they were superseded, not just because they're old.

It might be a bit dangerous to hardcode purge policies based only on just the "latest" revision available. That might break lots of build-infos or lockfiles using previous revisions.

conan purge would be deliberate. If it doesn't fit your workflow...don't do it. conan remove is dangerous too; if it doesn't fit your workflow, don't do it.

I could do a lot with a more comprehensive search or a search that could cascade. And more qualifiers on purging would be ok as well, number of revs to keep, purge by date but keep N revs, etc.

It might be that the best approach is to do a lockfile or buildinfo purge behavior.

What would that entail? So far, we don't really use lockfiles...too many different platforms and configurations of things. I suppose our use of Conan is not like most open source projects.