Conda-forge.github.io: Automatically open issue/PR on version update

[jakirkham]

The idea here is to cut down the maintenance burden of feedstocks a little more. In particular, notify the user that a new release is out and/or make the needed changes. If we do make the changes, we need to make sure we are following are guidelines ( https://github.com/conda-forge/staged-recipes/issues/139 ) very closely.

This may not be practical in all cases, but I think there are a number of cases where this would be beneficial and possibly quite doable. Here are some cases to consider.

• GitHub release tag
• New release on PyPI
• New release to CRAN
• New release to CPAN
• New release to LuaRocks ( see Lua support in conda https://github.com/conda/conda/pull/1934 https://github.com/conda/conda-build/pull/719 )

In some cases, we may be able to use a webhook. In others, we may need to do some parsing. We will need to make some decisions on how we want to handle these cases or find some other way that fits well with what we already do.

[pelson]

:+1:

Having de-centralised the ownership has made this slightly better (as a package owner can take some responsibility for a single feedstock, and it is easy for them to update the version with confidence), but I agree, there are still plenty of packages which we want the version to be checked and a notification made.

I'm kind of wondering if we should simply do this against the distributions in the conda-forge anaconda.org channel and some clever interactions with requires.io (or some other service).

[jakirkham]

Having de-centralised the ownership has made this slightly better (as a package owner can take some responsibility for a single feedstock, and it is easy for them to update the version with confidence)...

Yes, the model of one repo per package works very nicely I think. Especially with the use of conda-forge teams.

...there are still plenty of packages which we want the version to be checked and a notification made.

I would hope that we get pinged by upstream, which is why I have tried to get in touch with significant developers where possible. Of course, we also should keep our eye out. That all being said, having automated pings via issues and/or PRs to track on this point would make things a little more manageable.

I'm kind of wondering if we should simply do this against the distributions in the conda-forge anaconda.org channel and some clever interactions with requires.io (or some other service).

That is a very interesting idea. Tracking dependency relations (and their version) could open a very nice way to think about package management.

[minrk]

While auto-PR is tempting, I think opening Issues should be sufficient, and a PR that doesn't quite work could get annoying. The key issues for me would be:

1. identify the source for checking for new releases (PyPI, etc.)
2. check if the current recipe is out of date
3. check if a comment should be made
   
   
   
   • has an Issue already been made (by the bot)?
   
   
   • has someone already opened a PR with the new version?
   
   
   • bonus: has someone _else_ already opened an Issue about the new release?
   
   

Some questions/refinements along the way, largely to do with content of the Issue:

• should the bot include things like downloading the file and producing sha256 checksums in the Issue?
• it could go so far as to include a proposed meta.yaml (this is getting awfully close to a PR)
• should it ping maintainers listed in the recipe?

Starting with PyPI (https://github.com/conda-forge/conda-forge.github.io/pull/197) seems like a reasonable way to go, and it can be expanded to other mechanisms later once the kinks are worked out with the rest. The question then becomes how to identify the 'provider' for listing new releases in the future. One version would be to check the URL of the source (e.g. pypi.io -> pypi, github.com -> github, etc.). Another would be to be super explicit about it, with metadata like:

- upstream:
  - source: github
  - name: jupyter/jupyter_client

Questions include:

• Do we need manual override of upstream package name, or is inference from the recipe enough?

[jakirkham]

Nice summary, @minrk.

Completely agree about starting with an issue. That may be enough and is certainly better than what we have now. 😄

Couple notes. 3.a should be doable. We do that with PRs and issues raised now. 3.b and 3.c are a bit tricky as it comes down to dealing with a variety of different formats "Bump version", "New release", "v0.17.1" (is there a "v"?), etc. I'd be ok if that were dealt with in a second pass. Though do understand the sensitivity to noise. In short, agree with the tasks and am just suggesting they move to refinements along the way.

The refinements are interesting. Keeping them in mind, but not worrying about for v1 or whatever we call it.

Like the provider being listed explicitly. That seems like the only sensible way to start. Interesting side note (that it looks like was briefly noted in there), should multiple providers be allowed for one recipe? Again this is down the road, but worth thinking about. Use cases include when a tag is forgotten, but a PyPI release is made or vice versa. In the context of raising issues, this is straightforward. If we go down the PR route, this may get tricky (which source do we use/prefer? do we switch sources if one is missing?). Anyways this is down the road, but just worth keeping in mind.

[jakirkham]

cc @goanpeca

[goanpeca]

I have a POC for this :-)

[jjhelmus]

I have two scripts in the conda_recipe_tools repository which might be helpful. The first compares packages available in a conda channel against PyPI and lists any that are out-dated. The second has some rough logic to update a recipe's meta.yaml file to a given or the latest version.

I don't have too much time to work on these but am happy to review PRs to that repo. If nothing else I would be happy if these were useful for others.

[goanpeca]

I definitely dont have time this week, but next week I can show something I hope

[jakirkham]

@pmlandwehr has done a nice job putting together PR ( https://github.com/conda-forge/conda-forge.github.io/pull/375 ), which is a conda-execute script for updating PyPI packages' feedstocks. This should work nicely as a web service as it is using conda-execute already. Please take a look. Feedback welcome.

[jakirkham]

Would direct people interested in this issue to please take a look at issue ( https://github.com/conda-forge/conda-forge.github.io/issues/512 ).

[jakirkham]

Thanks to work done by @CJ-Wright, @justcalamari, and others this exists. Would encourage people interested in seeing that evolve to contribute to that work (ref below). Thanks all. :)

ref: https://github.com/regro/cf-scripts