Conda-forge.github.io: All package uploads are broken

[ngoldbaum]

Currently all package uploads are failing with the following traceback:

anaconda_upload is not set.  Not uploading wheels: []
+ upload_or_check_non_existence /recipe_root conda-forge --channel=main
Using Anaconda API: https://api.anaconda.org
Using Anaconda API: https://api.anaconda.org
[ERROR] 
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 102, in add_package
    aserver_api.package(username, package_name)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 288, in package
    self._check_response(res)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 216, in _check_response
    raise ErrCls(msg, res.status_code)
binstar_client.errors.NotFound: ('"whitenoise" could not be found', 404)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/anaconda", line 6, in <module>
    sys.exit(binstar_client.scripts.cli.main())
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/scripts/cli.py", line 88, in main
    description=__doc__, version=version)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/scripts/cli.py", line 66, in binstar_main
    return args.main(args)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 242, in main
    args=args)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 179, in upload_package
    add_package(aserver_api, args, username, package_name, package_attrs, package_type)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 126, in add_package
    license_family=package_attrs.get('license_family')
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 356, in add_package
    self._check_response(res)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 216, in _check_response
    raise ErrCls(msg, res.status_code)
binstar_client.errors.Unauthorized: ('Authorization token is no longer valid', 401)
Traceback (most recent call last):
  File "/opt/conda/bin/upload_or_check_non_existence", line 122, in <module>
    main()
  File "/opt/conda/bin/upload_or_check_non_existence", line 111, in main
    upload(cli, meta, owner, channel)
  File "/opt/conda/bin/upload_or_check_non_existence", line 62, in upload
    env=os.environ)
  File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['anaconda', '--quiet', '-t', 'binstar.token', 'upload', '/feedstock_root/build_artefacts/linux-64/whitenoise-3.3.0-py27_0.tar.bz2', '--user=conda-forge', '--channel=main']' returned non-zero exit status 1.
+ exit 1
Exited with code 1

I'm not sure if this is a problem with conda-forge or with the anaconda platform.

[jakirkham]

Thanks for opening this @ngoldbaum. Will try to keep the discussion about this problem in this issue to make it easy to find info and to avoid having the discussion get too dispersed. Though will cross link this issue from other discussions, gitter, the mailing list, etc. to get the info out there.

What happened: We discovered recently that the Anaconda token was exposed in some recent CI logs. As soon as this issue was discovered, fixed the code responsible for the vulnerability, attempted to pull any associated logs (but were unable to), and pulled the token. As this token is responsible for uploads from all feedstocks, this has caused any recent uploads from feedstocks to fail. Further any new feedstocks created by staged-recipes were DOA due to this issue.

Action Plan: At our dev meeting earlier today, we discussed this issue and outlined the following plan. First generate a new token. Second change out the token at staged-recipes so that any new feedstocks created will not suffer from this issue (otherwise they will pile up with the existing broken feedstocks). Third for any feedstocks in need of an immediate update, readd the recipe through staged-recipes. Fourth write a script to update the remaining feedstocks focusing on those that have been recently updated first.

Qualification for near term fix: To elaborate a bit more on the second point (as that seems relevant to you @ngoldbaum and likely others who will find this issue soon), this should only be done if you are trying to get a release out today or if you have tried and failed to make a release in the last 24hrs. As for recipes that have been recently converted by staged-recipes, we will try to handle these directly ourselves (core and staged-recipes teams) for a number of reasons including to make sure we have gotten them all, to avoid confusion for new contributors, and to avoid stressing the system. If you do not fall in this category, please do wait for the mid term fix to be applied to your feedstock. If you know a release will be coming out soon, please raise an issue on your feedstock with a link to this issue and ping @conda-forge/core and we will try to move your feedstock up in the midterm queue.

Near term: If you do fall into this category of having tried to make release from a feedstock, please simply copy the recipe from your feedstock's master branch and place it in staged-recipes again under the recipes directory. Ensure that your recipe directory name uses your package name. Please make sure to include [ci skip] [skip ci] in the commit message to try and avoid having CIs run on these recipes. In your PR title, please write "Readd <package name>" where <package name> will be the package in question. Also in the PR description, please include that you are readding your recipe to get a new token in the feedstock and include a link to this issue. Doing all of this should help ensure that we are able to find your PR and merge it in a timely manner.

Mid term: For everything else that is broken, but is not in need of an immediate release, we will be writing a script to send PRs to affected feedstocks in small batches. These will update the token and re-render feedstocks. Priority will be given to feedstocks that have been changed recently.

Long term: We will begin discussions at dev meetings on how to better handle and manage uploads in the future to minimize the fallout from similar issues (e.g. expired tokens).

As a general reminder, please keep in mind that we were all affected negatively by this issue. So it will take some time to get everything back in working order for everyone. That said, we are eager to do exactly that. Thanks for your understanding and cooperation.

cc @conda-forge/core @conda-forge/staged-recipes

Edit: Added a link to staged-recipes.

[nicoddemus]

Thanks @jakirkham for the update! I'm sure all conda-forge users appreciate your dedication and efforts in keeping everything running smoothly! 👍

[jakirkham]

At this point, feedstocks created since the incident began have now been fixed. Also any new feedstocks created by staged-recipes should not have this issue. However the bulk of the feedstocks will still be using the revoked token. So there is still much more work needed to resolve the issue.

[asmeurer]

I would also suggest fixing either binstar_client or upload_or_check_non_existence to give a better error message when this happens.

[jakirkham]

Feel free to raise an issue and/or open a PR over at conda-forge-build-setup to fix this.

[jjhelmus]

Provided valid Travis, AppVeyor, CircleCI and Anaconda.org tokens are available, I believe the script in this gist an be run in a feedstock's root directory to update the upload token in all the correct locations.

My though was that this could be incorporating this into a shell script which loops over the directories in the conda-forge/feedstocks repositories, updates the feedstock, creates a commit, and pushes the change to the feedstock. Whoever does this would need full read/write access to the conda-forge organization and therefore would likely be limited to the @conda-forge/core.

I'm not sure how much CI traffic this action would create but adding these one-by-one via staged-recipes will take quite some time.

[wesm]

Any chance you could do me a kindness and update the tokens in:

• arrow-cpp-feedstock
• parquet-cpp-feedstock
• pyarrow-feedstock

? We just made an Arrow release and users are waiting on the updated packages. Otherwise we will open PRs to readd them

cc @cpcloud @MaxRis

[ocefpaf]

I can do those in my next PR. Hold on...

[blink1073]

@ocefpaf, could I trouble you for jupyterlab_launcher-feedstock and jupyterlab-feedstock as well? We'd like to make a release before JupyterCon next week.

[ocefpaf]

I'll do them today @blink1073. Let me just wait for Travis-CI to clear a few recipes.

[blink1073]

Much obliged!

[pkgw]

@ocefpaf Would appreciate fast-tracking of aipy and casacore if you have the chance. Anything useful non-core people can do to help the cleanup process?

[ocefpaf]

@ocefpaf Would appreciate fast-tracking of aipy and casacore if you have the chance.

Sure. As soon as the CIs clear the current recipes I'll submit those.

Anything useful non-core people can do to help the cleanup process?

You can send the PRs to staged-recipes to re-add them and ping me there to merge.

[jakirkham]

Anything useful non-core people can do to help the cleanup process?

Also help triaging staged-recipes.

[pkgw]

So it sounds like the expectation that most of the feedstocks will be fixed up by resubmission to staged-recipes, rather than running some repair script that will patch up all of the feedstock repos independently? Is that right?

[ocefpaf]

@jjhelmus script would work and I think we should go for it. I am not sure how to get the right Travis-CI token though and I am swamped by the day job for a few days...

[jschueller]

Is updating the appveyor token manually equivalent to readding the feedstock through staged recipes ?

[ChrisBarker-NOAA]

OK -- I'm lost.

What can a regular old feedstock maintainer do now to update the tokens in a feedstock?

https://github.com/conda-forge/gridded-feedstock

being the task at hand.

[mwcraig]

@ChrisBarker-NOAA -- as I understand it, open a PR on staged-recipes following the "near term" instructions here: https://github.com/conda-forge/conda-forge.github.io/issues/426#issuecomment-321897876

Ideally those PRs are only being opened if a release of the package just happened or is imminent. I don't think we are checking that, just relying on the good will of the community to be patient if there is is no need to update.

[isuruf]

I used @jjhelmus's script and modified it for staged-recipes and pushed to tokens branch. All the feedstocks were updated (the ones with the new token were left alone) except for 2 repos trmm_rsl, textract. First one had a very old conda-forge.yml without an appveyor token and git push fails for the second one.

[ocefpaf]

All the feedstocks were updated (the ones with the new token were left alone) except for 2 repos

@isuruf I'll take a look at these two tomorrow morning. Thanks for the awesome work!

[jjhelmus]

trmm_rsl is one of my feedstocks from the early days of conda-forge. There is no windows build so updating the AppVeyor token is not needed. I'll look into re-rendering it with conda-smithy.

[isuruf]

All the feedstocks are updated now.

[asmeurer]

The commit was pushed with ci skip. Should feedstock maintainers manually trigger builds if the packages aren't up-to-date, or will you be automating that as well?

[ocefpaf]

It should not be needed b/c that commit triggers a re-rendering in the feedstock, the skip happens only on staged-recipes.

[isuruf]

@asmeurer, recipes that were fixed using the script needs a manual trigger. (recipes that were re-added through staged-recipes need no trigger)

[jakirkham]

Thanks so much for doing this, @isuruf!

[jakirkham]

As some feedstocks were being hit by this midrelease, would recommend that people check if their feedstocks were affected and failed to release. Can just re-render your feedstock to get a new release out.