Conda-forge.github.io: Package deletions

[183amir]

I wanted to know that the policy of deleting broken packages are still in place. If so, what to do to avoid it?

[183amir]

We are in the process of writing our continuous integration based on conda and we want to create an environment, freeze it, and keep testing our software against that environment.
These environments use the below packages from conda-forge:

https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py27_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py34_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/boost-1.61.0-py35_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ffmpeg-2.8.6-2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/giflib-5.1.2-1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/icu-56.1-4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipdb-0.10.1-py27_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipdb-0.10.1-py34_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ipdb-0.10.1-py35_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libblitz-0.10-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libmatio-1.5.6-4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libsvm-3.21-1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/sox-14.4.2-libpng1.6.22_6.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/vlfeat-0.9.20-1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/x264-20131217-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/boost-1.61.0-py27_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/boost-1.61.0-py34_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/boost-1.61.0-py35_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ffmpeg-2.8.6-2.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/giflib-5.1.2-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/icu-56.1-4.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ipdb-0.10.1-py27_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ipdb-0.10.1-py34_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/ipdb-0.10.1-py35_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/libblitz-0.10-0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/libmatio-1.5.6-4.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/libsvm-3.21-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/sox-14.4.2-libpng1.6.22_6.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/vlfeat-0.9.20-1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-64/x264-20131217-1.tar.bz2

We want those packages to be never deleted nor replaced/changed so that we can keep testing our software against those.
Can @conda-forge/core guarantee us that those packages will be kept forever? What is your policy of keeping packages as of now?

[patricksnape]

Relying on specific builds may be a bad idea as they may be rebuilt in the future when a build bug is discovered.

[mwcraig]

My recollection is that we don't delete packages; if a new build is needed then the build number should be increments. I don't think there is a mechanism to enforce this in anaconda cloud like there is (sort of) on pypi.

The conda-forge account has a very high storage limit so I don't see a need to delete old packages.

I don't think anyone can guarantee this kind of thing forever :wink:

[183amir]

If you want people to rely on conda-forge, you have to have clear rules and policies. As of now, I have seen some packages being deleted. I want to know if we have a clear policy on this.

[jakirkham]

Normally deletion has happened before when the pinning is bad and the package is interfering or if the package was deemed to be breaking peoples environments. Though normally this has been by request as opposed to admin initiated action. Previously I had suggested we label broken packages instead of deleting them in issue ( https://github.com/conda-forge/conda-forge.github.io/issues/181 ). Though it hasn't really gotten much feedback at this time. Personally that has been how I have handled all ejections from main since. While we could design a more sophisticated system and set down rules for how it behaves, at least relabeling broken packages is amenable to being changed; whereas, removal is not. However, this isn't really the question being asked.

[jakirkham]

FWICT the question is (reformulating it a bit)...

What is conda-forge's retention policy?

To be clear, I think this is a 💯 reasonable question to ask. Though it is a bit different than please don't delete these packages forever. There are two reasons it is different.

1. It acknowledges there are causes for removal (if nothing else storage limitations).
2. It requires that deletions must proceed under systematic guidelines that are transparent.

Given this is not the first time that deletion has been a controversial topic for the community, it seems totally reasonable for us to have a contract about when and how deletions occur. As such, this feels to me like an Enhancement Proposal (EP) in the making.

We are just starting the work on our Enhancement Proposals (EPs). That being said, this will still need to be written up before it can proceed through such a mechanism. Given your interest in this @183amir, I would suggest that you start an org hackpad or other shareable medium to start writing on this point about what seems reasonable in an unbiased manner. Feel free to invite others to work with you on this. Once you have that roughly written out, we can try to run that through the EP system.

Does this sound reasonable?

[patricksnape]

Agreed that this sounds like a CFEP candidate!

[edisongustavo]

@183amir At the company that I work we've set up internal channels:

• mirror-conda-forge (mirrors the conda-forge channel)
• mirror (mirrors the default channel)
• esss (our own non-open source packages go here)

Notice that an internal channel are simply directories being served over apache. So it is very easy to create your own channel.

We're in the process of making the "mirroring" script available as open-source. It is simple too, but maybe it could help you.

[ocefpaf]

@edisongustavo that is probably the best strategy as both conda-forge and defaults are hosted by continuum and both may be subject to deletion at any point in time due to any resources limitations. (Although I am pretty sure that won't happen it is better to be prepared :wink:)

[183amir]

@edisongustavo any news on sharing the mirroring script?

[ericdill]

@183amir I have also written a conda channel mirroring

[ericdill]

Oops, commenting on my phone is rough! What I was trying to say is that I've also written a mirroring script. You can find it at github.com/MaxPoint/conda-mirror and it's available on pypi.

[183amir]

Thank you! for anyone who is interested, there is also this repository to help you setup a local channel: https://github.com/danielfrg/docker-conda-channel

[edisongustavo]

@183amir sorry about that, I forgot about this.

I've left the company where the script resides, but @tadeu and @gqmelo can help you with that.

[tadeu]

cc @nicoddemus

[nicoddemus]

@183amir after taking a look at https://github.com/MaxPoint/conda-mirror, we decided to ditch our internal scripts in favor of using it. If you are really interested in our scripts we can share them with you, but MaxPoint/conda-mirror is in a much better state than our (messy) internal scripts, so we suggest that you do the same. 😁

[183amir]

@nicoddemus thank you the https://github.com/MaxPoint/conda-mirror is fine.

[ericdill]

@nicoddemus @183amir Please feel free to request features (or implement them and submit a PR!) if there are use cases that are not covered in conda-mirror.

[nicoddemus]

@ericdill will do! And thanks for creating and sharing that project.

[jakirkham]

Oops, commenting on my phone is rough! What I was trying to say is that I've also written a mirroring script. You can find it at github.com/MaxPoint/conda-mirror and it's available on pypi.

Should we get it packaged on conda-forge too, @ericdill? 😜

[ericdill]

Should we get it packaged on conda-forge too, @ericdill? :stuck_out_tongue_winking_eye:

@jakirkham Hah, yes :disappointed: . For a time it was being rev'd so quickly that it didn't make sense to get it going on conda-forge. Will submit a recipe this week

[jakirkham]

No worries. Actually recent issues with S3 have motivated me to put one together. 😄 Please let me know your thoughts.

xref: https://github.com/conda-forge/staged-recipes/pull/2526

[ericdill]

It's been on our teams kanban board since Jan 18. Thanks for doing this :grin:

[jakirkham]

Thanks for writing and sharing this nice utility.

[jakirkham]

Resolved with PR ( https://github.com/conda-forge/conda-forge.github.io/pull/538 ). If not, please let us know and we can reopen.