Conda-forge.github.io: Request for clarification of conda-forges role in development.

[artPlusPlus]

This issue is intended to help clarify how developers should approach using conda-forge in terms of development and releases. My goal is to balance my own efficiency and the efficiency of conda-forge. I do not want to abuse the time of the maintainers or the CI services. Ultimately, it would be useful for this information to make it into the docs.

Recently, I staged my first recipe (conda-forge/staged-recipes#2499). This package is part of a greater whole, and all of it is still pre-alpha at best, so it was marked as dev.

A comment was left by @jakirkham asking if a non-dev version was available, which caused me to rethink my perception of where conda-forge sits in the development/release cycle and how I should be using it.

My previous perception was because conda-forge integrates many CI platforms, it was part of the development process; part of setting up a new python project would be to get a (potentially empty) package into the conda-forge feedstock in order to leverage the CI services during development. As it is, I manually manage CI configurations and integrations for Appveyor and Travis. I saw conda-forge as a means to not have to do that anymore.

However, after this comment, I began to think maybe conda-forge is not a development tool, but instead, a release tool. I should be providing my own CI solution for evaluating dev grade builds (commits to master), and only using conda-forge when a package has been deemed safe for public consumption.

These are my questions:

• How would conda-forge like me to use their service?
• What grade of release is appropriate for conda-forge?
• In a development process, what, if anything, should happen between a commit hitting master and that commit getting picked up by conda-forge?
  
  
  
  • Should these commits be making it to conda-forge at all?
  
  
  • What about a non-master branch?
  
  
• If conda-forge is not intended to be part of the development cycle, are there any aspects of it that can be leveraged to make setting up and maintaining CI for development easier?

[ChrisBarker-NOAA]

This is just my personal opinion-- but no, conda forge is NOT there to
support development.

And it seems we are frequently hitting various limits, so we shouldn't
extend its use, even though it might be convenient.

• How would conda-forge like me to use their service?

To make your package easily available to end users.

• What grade of release is appropriate for conda-forge?

I'm not inclined to be pedantic about this -- so anything you'd want your
users that are not involved with development to use. Usually this is
"releases", but you may want to make previews available if the project is
young.

• In a development process, what, if anything, should happen between a
  commit hitting master and that commit getting picked up by conda-forge?
  
  
  
  • Should these commits be making it to conda-forge at all?
  
  

No

-
- What about a non-master branch?

No!

I'd say only if you tag a release.

-
- If conda-forge is not intended to be part of the development cycle,
are there any aspects of it that can be leveraged to make setting up and
maintaining CI for development easier?

The code is all open source, it may be helpful as a framework to borrow
from.

-CHB

[artPlusPlus]

Thank you, Chris. That was exactly what I was looking for.

[astrofrog]

@artPlusPlus - if you are interested in using conda as part of your Travis/AppVeyor builds in your own project, you may instead be interersted in https://github.com/astropy/ci-helpers, which removes a lot of the required boilerplate code.

[jakirkham]

Thanks @ChrisBarker-NOAA for the nice explanation.

Would expand on this a little bit on a few fronts.

First periodically people want to do a release, but they want to check if a commit will work before they tag it and do other release actions (e.g. deploy to PyPI, etc.). As it is difficult to know if this will work without testing it at conda-forge, I would argue this is ok. However, this can't be used to test every commit only as part of release preparation. Once it works it has to be tagged and deployed the standard way.

Second it is totally possible to use conda-smithy to create feedstocks in one's org and use those to release to your own channel. I'm not sure whether this fits your development needs, but it is an option to keep in mind. As these will use CIs on your own org, likely the release process will be faster as they will spend less time queued than they would here. Take a look at conda-smithy's Readme for more details.

Third we do allow rcs and the sort. There is a process to do this, but it is a bit fuzzy. Still I think what @ChrisBarker-NOAA said is very important to keep in mind namely these would be, "...anything you'd want your users that are not involved with development to use." So if it is something you want to package and use internally, creating your own feedstocks in your org (as explained above) is more appropriate. However if it is something you would like external users to try and just let them know with an asterisk that this is still rough around the edges, then deploying rcs here may be appropriate.

Hope this helps clarify things a bit.

[artPlusPlus]

Thank you both, @ChrisBarker-NOAA and @jakirkham for taking the time to clarify. Both of your comments are very useful.

Additionally, thank you for pointing me at conda-smithy. Between that and @astrofrog's ci-helpers, I should be able to optimize the development side without impacting conda-forge. I will add a bit to the docs to include conda-smithy as an option.