Conda-forge.github.io: Channel for pre-releases

[astrofrog]

It would be cool to have a way to build packages for pre-releases and have them be uploaded to a e.g. conda-forge-pre channel. This could be done by having a prerelease branch in a feedstock repo, which if present will build as normal but upload to this pre-release channel instead of the main channel.

It might be reasonably easy to configure this in the CI config?

[Cadair]

+1 to the prerelease branch idea, that's a really nice way of doing it.

[jakirkham]

Thanks for adding this, @astrofrog.

[Cadair]

I have setup a branch on the sunpy-feedstock repo which uploads to the "pre" label: https://github.com/conda-forge/sunpy-feedstock/tree/prerelease

It wasn't hard, but it also wasn't smooth either, it would be good if there were a better way to handle it.

[jakirkham]

@Cadair, this is something I would like us to discuss as a community please. I strongly urge you to hold off on traversing new ground here without some feedback and healthy discussion.

[jakirkham]

Also, should add that making these changes directly to the repo (instead of your fork) results in an immediate deploy of packages from any CI build. There is no guarantee what has been released at this point. So, this may have resulted in the creation of packages that are undesirable. This could have been totally avoidable if we discussed this beforehand and reached some sort of consensus. So, I would really appreciate that we hold off any more changes until we can get some more people's thoughts on this.

[Cadair]

@jakirkham I made sure the builds were working by issuing a PR first: https://github.com/conda-forge/sunpy-feedstock/pull/2 but then to demonstrate that the upload would work to non-main label I obviously needed to push to the main feedstock.

I appreciate it could have gone wrong, but I thought it was worth a punt especially seeing how the repo is new and there are no users at the moment, we haven't advertised it yet.

[pelson]

The problem with having a conda-forge-pre channel is that you have no obvious control of _what_ you are letting yourself in for. One thought that I have had on this in the past is that if we want to be able to do pre-releases (and we do for those packages which provide them) then maybe the best way is for each package to fork the conda-forge feedstock into their own organisation and push to an organisation specific channel. A few other ideas:

• we check the version string at upload time and if it is rc like we upload to a specific user and/or channel
• we maintain a rc branch (as per @Cadair's approach) which configures the conda-forge.yml to upload to a different channel (I think conda-smithy should just work with this usecase, though I fully accept that conda-smithy's configure_feedsock isn't the most pleasant of reads 😄

Just a few thoughts. Would be a good topic for a hangout... 😉 https://conda-forge.hackpad.com/conda-forge-meetings-2YkV96cvxPG

[Cadair]

another potential approach would be to push to a label with the package name in it, so where as I uploaded to the pre label I could have uploaded to sunpy-pre or similar so that if I wanted prerelease SunPy then I could install with conda install -c conda-forge/label/sunpy-pre sunpy?

[jjhelmus]

I too would be interested in having a method of serving up pre-release conda packages. My particular use case would be for making Python 3.6 alpha/beta/rc packages available. Have not worked through the workflow yet so I have no opinion on the best manner of implemented this.

[pelson]

Just wanted to note that I made a devel branch on matplotlib-feedstock in order to see how maintainable it is (that and I really wanted to help out getting a RC for matplotlib). It is clear that we need a solution/policy on this. The next planned meeting is 2016-06-03 14:00UTC and we should get around to discussing it some more there (all welcome to these meetings).

[pelson]

Another idea: we put an expiration policy in place on the rc channel (something suitably long to give people a chance to try stuff out, and suitably short to stop package maintainers avoiding RC cycling/releasing 😉 - e.g. 6 weeks).

[jakirkham]

Something else to think about. I know others have expressed a need for NumPy pre-releases before ( cc @jreback ). It would be good if we could find a way to do that here as well.

@rgommers, do you have any thoughts on this? Basically, this would supersede this PR ( https://github.com/numpy/numpy/pull/7036 ).

[rgommers]

Project specific channels would mainly be useful for CI, but:

• there's a lot of overhead to maintain those channels
• users probably won't bother with them.

A single pre-releases channel will have lower overhead and be used more widely I'd expect. Downside is maybe that users want pre-releases only for particular packages, and if it's all in one channel then they have less/no control over which packages those are. On the other hand, if it's limited to rc's, then the number of packages that release those is probably not huge. Many smaller projects just directly tag new releases without bothering with pre-releases.

So I see more benefits than downsides of a single pre-release channel. If projects want to make alpha/beta releases or dev builds available, they are still free to maintain their own channel.

[ChrisBarker-NOAA]

On Sat, Jun 4, 2016 at 10:05 AM, Ralf Gommers [email protected]
wrote:

• A single pre-releases channel will have lower overhead and be used
  more widely I'd expect.

+1 I like this idea.

I've been trying to figure out where to put packages for the wxPython
Phoenix builds.

I assume that we don't need to be pedantic about what 'pre-release" means
-- i.e. anything that's not a proper release, not only release "candidate".

-CHB

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

Chris.[email protected]

[rgommers]

I assume that we don't need to be pedantic about what 'pre-release" means -- i.e. anything that's not a proper release, not only release "candidate".

Not pedantic, but we do need to be clear I think. I'd say only rc, or rc + beta. With PEP 440 compliant names. The idea (at least my idea) was that users could add the channel and get pre-releases to test. They shouldn't run the risk of getting a random dev build.

[ChrisBarker-NOAA]

On Sun, Jun 5, 2016 at 12:07 PM, Ralf Gommers [email protected]
wrote:

I assume that we don't need to be pedantic about what 'pre-release" means
-- i.e. anything that's not a proper release, not only release "candidate".

Not pedantic, but we do need to be clear I think. I'd say only rc, or rc

• beta. With PEP 440 compliant names. The idea (at least my idea) was
  that users could add the channel and get pre-releases to test. They
  shouldn't run the risk of getting a random dev build.

hmm, I see your point, but if we have pre-releases from arbitrary packages
in there, then simply adding the channel would get folks who-knows-what. I
was imaging that use case of folks instaling a particular pre-release, and
specifying the channel on the command line for just that install.

Not a big deal for wxPython Phoenix, as there are ONLY dev versions
available, and I'd make sure to give it a different name, like wxPhoenix so
no one would accidentally get it when they wanted a stable wxPython.

-CHB

—

You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/147#issuecomment-223831037,
or mute the thread
https://github.com/notifications/unsubscribe/AA38YFAUzP8dCWB_pq_AEMSnPcNdUDgdks5qIx5dgaJpZM4IibIK
.

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

Chris.[email protected]

[jakirkham]

Honestly, I think having one channel label (i.e. dev) sounds sensible from a maintenance standpoint. If we feel that it is difficult to control what one gets out of it when using conda, then it feels like a problem we should address in conda not conda-forge. Does that make sense?

[jakirkham]

cc-ing @danielfrg because of PR ( https://github.com/conda-forge/tensorflow-feedstock/pull/8 ), which would add an rc of tensorflow.

[patricksnape]

What if conda added something similar to what pip already has and required a special flag in order to grab 'pre-release' versions? By default pip won't install pre-releases - perhaps conda should do the same? Then we wouldn't need a new channel?

[jankatins]

If a prerelease switch is added, it needs to identify prereleases and that's not as easy as in an all Python world where versions have a specific meaning (JPEG 8c is not a prerelease).

[ChrisBarker-NOAA]

What if conda added something similar to what pip already has and required
a special flag in order to grab 'pre-release' versions?

+1

By default pip won't install pre-releases - perhaps conda should do the
same? Then we wouldn't need a new channel?
The other trick is dependencies -- if you want a pre-release, do you want
pre-releases of the deps???

CHB

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/147#issuecomment-224527418,
or mute the thread
https://github.com/notifications/unsubscribe/AA38YOQFu2g4LIXEHd4mNxASih5G2U7Iks5qJoGDgaJpZM4IibIK
.

[ChrisBarker-NOAA]

On Jun 8, 2016, at 7:23 AM, Jan Schulz [email protected] wrote:

If a prerelease switch is added, it needs to identify prereleases and
that's not as easy as in an all Python world where versions have a specific
meaning (JPEG 8c is not a prerelease).

Yup. But the conda package version does not need to exactly match the
underlying package version, does it?

But there would need to be a conda versioning standard to identify
pre-releases.

CHB

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/conda-forge/conda-forge.github.io/issues/147#issuecomment-224549502,
or mute the thread
https://github.com/notifications/unsubscribe/AA38YIHrZdSxgJ-XGwh1iryexztcVFdtks5qJpgigaJpZM4IibIK
.

[jakirkham]

So, a flag is nice. Setting a versioning standard with conda dev/rc releases may be a bit tricky as its versioning system has been extended to handle many things beyond the PEP440 versioning scheme. Though it is certainly worth thinking about

Let me take a quick poll so I can get a sense of what is we think we still need. Currently one can already do this.

conda install -y matplotlib
conda install -y --no-deps -c conda-forge/label/rc matplotlib

This installs matplotlib as an rc. What do we think about this? Personally, I see a few problems, but they are resolvable. What if the rc has new dependencies? Basically, we don't get them. What happens when we run conda update --all? We lose our rc. What if we could just say we want to get the matplotlib package from the rc channel? Would that not solve our current problems?

cc @msarahan @mcg1969

[jjhelmus]

Dicussion on this topic seems to have stalled without a concrete proposal. Would it make sense to discuss this during tomorrow's (Jun 24, 2016) conference call or a later date?

[ocefpaf]

Tomorrow would be good. We are already doing this for matplotlib, right? So it is a matter of laying out a plan (I hope :grimacing:).

[jakirkham]

So, another interesting facet is packages that depend on RCs (even if they are not RCs themselves). See this PR ( https://github.com/conda-forge/staged-recipes/pull/1118 ) for an example of this use case.

cc @SylvainCorlay

[jakirkham]

I've opened this issue ( https://github.com/conda/conda/issues/3125 ) at conda to try and get some thoughts on the pre-release point. It feels to me the sticking point is having control of what development releases one gets. Most of the discussion at the end seemed to be about how conda could be changed with flags and other things to accomplish this. So, it felt like time to change the forum of the discussion. I have cc'd everyone on the ticket for awareness (and everyone's apparent interest). Please feel free to chose your level of involvement though.

[jorisvandenbossche]

I would like to make a rc release of pandas on conda-forge, but it seems there is not yet a final discussion on this?
In the meantime that changes to conda are discussed, is it OK to use a similar approach as done in matplotlib (https://github.com/conda-forge/matplotlib-feedstock/pull/24#issuecomment-221496870) or python (https://github.com/conda-forge/python-feedstock/pull/45)? Or does it need to wait until a final decision?

[jakirkham]

Would we be willing to try a beta of NumPy? 1.12.0b1 has been out for ~20 days.

[rgommers]

That would be interesting I think. 1.12.0rc1 may be just around the corner though. @charris do you have a date in mind for that already?

[charris]

@rgommers I was going to do it yesterday, but waiting on a ma.median fix. I'd like to have the final out at the end of the year.

[ocefpaf]

Closing this. We do allow for pre-releases in labels for a while.
(Although we still need to write some docs on how to do it.)