Staged-recipes: HTTP 403 error during conversion

[jakirkham]

Appears we have been getting this error for the past 2 days at staged-recipes during conversion of any feedstocks or registration of any feedstock with Travis CI. This has also been blocking our ability to retrigger Travis CI (have been issuing these myself manually, maybe others have too?). Recent log linked below.

ref: https://travis-ci.org/conda-forge/staged-recipes/builds/378003067 (recent failure)
ref: https://travis-ci.org/conda-forge/staged-recipes/builds/377391161 (last working conversion)

cc @conda-forge/core @conda-forge/staged-recipes

[isuruf]

Is the Travis-CI API key still working? It might have expired.

[jakirkham]

Do you remember which bot it is?

[djsutherland]

TRAVIS_TOKEN says it's for the conda-forge Travis user: https://github.com/conda-forge/staged-recipes/blob/master/.travis.yml#L30

Blame says it was last updated 2 years ago (in https://github.com/conda-forge/staged-recipes/commit/5c3123a6153ad66c4bec75327f246cd3783ec861).

conda-smithy also looks like it can automatically get a token, but it doesn't try to if it's already given one, which it is.

Not sure how to get the token myself, so going to temporarily turn off the caching and see if that fixes it....

[djsutherland]

Nope.

[djsutherland]

The response received is (from here)

{
  "@type": "error",
  "error_type": "insufficient_access",
  "error_message": "operation requires create_request access to repository",
  "resource_type": "repository",
  "permission": "create_request",
  "repository": {
    "@type": "repository",
    "@href": "/repo/5842170",
    "@representation": "minimal",
    "id": 5842170,
    "name": "staged-recipes",
    "slug": "conda-forge/staged-recipes"
  }
}

Can't keep working on this right now, but will try again maybe tomorrow if nobody else figures it out first. :|

[jakirkham]

Hmm...@pelson, would you have time to update the Travis CI GitHub token at staged-recipes?

[isuruf]

Looks like it was 2 errors at once. Travis-CI token getting expired and the conda-smithy issue.

[jakirkham]

Went ahead and released conda-smithy.

However we still need to replace the Travis CI token. Not having the Travis CI token will quickly rate limit us (as was the case long ago when we added it and is still the case now based on recent builds).

ref: https://travis-ci.org/conda-forge/staged-recipes/builds/379050702

[jakirkham]

Now cleared! 🎉

Though the Travis CI token still should be fixed. Let's leave this open for that fix.

[jakirkham]

To give more of an idea about the importance of using a Travis CI token vs. not using one w.r.t. GitHub rate limiting, converting a single recipe went from 5000 requests allowed to 227 requests allowed (relevant lines linked below). Using the Travis CI token cuts this by roughly 2 orders of magnitude!

ref: https://travis-ci.org/conda-forge/staged-recipes/builds/379444909#L839
ref: https://travis-ci.org/conda-forge/staged-recipes/builds/379444909#L923

[pelson]

@jakirkham - do you need the token from me? I don't want to be the bottleneck on this kind of thing, so it is important we have some mechanism for sharing credentials securely...

[isuruf]

We can generate the travis-ci token as in https://github.com/conda-forge/staged-recipes/pull/5878

[jakirkham]

Hi @pelson. Good timing. Was just reading all the notifications I missed.

Yeah I think you had added this one. Guessing it is based off the GitHub token used at staged-recipes (looks like @conda-forge-manager). A new token for Travis CI should fix it.

It’s a good point and something we discussed at AnacondaCON for basically the same reasons. Namely the larger core group needs to be able to handle these sorts of things and bot access is one where things are difficult to share. Would be open to any good suggestions. :)

[pelson]

I've given it a shot in #5879. As I say there, I'm not 100% confident I've remembered how that token should be generated... 😞

[jakirkham]

Thanks Phil. Much appreciated. 😄

[jakirkham]

Closing as this has been resolved with PR ( https://github.com/conda-forge/staged-recipes/pull/5879 ). 🎉 Thanks all! 😄

[jakirkham]

Just for posterity, here's a log using the Travis CI token to convert a single recipe. Note that this uses 32 requests vs. the 4773 requests without the token ( shown above ). Hopefully this provides context as to the value of storing the Travis CI token.

Side note: There is probably still something weird going on as 32 requests still sounds too large for converting a single feedstock, but it is small enough compared to our 5000 request limit and typical throughput rate at staged-recipes that there has been little investigation into this.

[mstimberg]

Recent builds seem to be failing again (last one: https://travis-ci.org/conda-forge/staged-recipes/builds/388256208), preventing the creation of new feedstocks.

[jakirkham]

Thanks @mstimberg. It did seem to go through a day ago. That said, am seeing a lot of these errors today as well. Going to reopen.

[jakirkham]

For example, here's one from ~6 mins ago. Seems to have been going on for ~9 hrs continuously.

[jakirkham]

Going to drop the Travis token again for a bit. ( https://github.com/conda-forge/staged-recipes/pull/6097 ) This is not a long term fix. Will need a new Travis token and probably need to figure out why the last one went bad so quickly.

[jakirkham]

Dropping the token doesn't help. Seeing this...

{
  "@type": "error",
  "error_type": "wrong_credentials",
  "error_message": "access denied"
}

ref: https://travis-ci.org/conda-forge/staged-recipes/builds/392559745#L1048-L1050

[jakirkham]

FWIW wound up adding the Travis CI token back in PR ( https://github.com/conda-forge/staged-recipes/pull/6099 ). The issue of course persists. Just dropping it didn't totally work and it was late. Though it may work along with this other change, @isuruf mentioned.

[jakirkham]

Token was dropped again with PR ( https://github.com/conda-forge/staged-recipes/pull/6105 ), but still doesn't seem to be working.

[isuruf]

@pelson, can you try one more time to update the GH_TOKEN with conda-forge-manager?

[pelson]

I already did so in #6149. I'm confident that I've encrypted them fine, so can only conclude that the tokens themselves are non-functional. FWIW Have you had success with other tokens?

[isuruf]

FWIW Have you had success with other tokens?

Yes, I've had success with a bot I manage. (@conda-forge-bot), but for some reason @conda-forge-manager token you added doesn't work.

[isuruf]

Fixed now