Qtox: Nightly builds failing

[nurupo]

@iphydf has pointed out on IRC that nightly builds have stopped being made. It looks like CIRP is hitting this error, failing cron builds. The link explains how to work around the error. In short - set CIRP_TRAVIS_ACCESS_TOKEN secure env variable on qtox/qtox Travis-CI to @qTox-bot's API token. CIRP needs the API token mainly to avoid being rate-limited, it doesn't need any permissions to cancel/restart build.

[sudden6]

Added the token to CIRP_TRAVIS_ACCESS_TOKEN, I'll keep this open until nightlies are working consistently again.

[anthonybilinski]

The cron job still failed after sudden6 setting the token. I re-set the token in case there was a mistake and kicked off the cron job again, but I still see too many 403 error responses.
The build in question is here: https://travis-ci.org/github/qTox/qTox/jobs/671550203?utm_medium=notification&utm_source=github_status
any idea on next steps, @nurupo?

[nurupo]

Setting the token did work, it no longer fails on the /auth/github?github_token endpoint. That's the most problematic endpoint, and setting the token lets you avoid calling it. It fails on something else now, which looks like Tavis-CI's API had a hiccup, so I'd wait for another build. Btw, CIRP retries failed requests up to 7 times, with increasing backoff, so those errors indicate that API returned 403 seven times in a row for the same request.

[anthonybilinski]

That's the most problematic endpoint, and setting the token lets you avoid calling it.

Ah I see.

which looks like Tavis-CI's API had a hiccup, so I'd wait for another build.

We still seem to be hitting

09:59:08 RetryError: HTTPSConnectionPool(host='api.travis-ci.org', port=443): Max retries exceeded with url: /repo/qTox%2FqTox/builds?sort_by=finished_at%3Adesc&offset=0&limit=100&branch.name=master (Caused by ResponseError('too many 403 error responses'))

on every cron build for the last week: https://travis-ci.org/github/qtox/qtox/builds

I enabled cron builds on the v1.17-dev branch recently, in case that's somehow related.

[nurupo]

Hm, that's weird, my vlc cron builds are not failing and I can't reproduce those API endpoints 403'ing locally no matter how hard I spam them.

Could you try adding curl calls after cleanup1.sh and cleanup2.sh scripts? After those scripts, not inside the scripts, as they have -e set and will exit right away on error. Maybe also do it in a separate testing branch as not to pollute master with debugging commits, adding --latest-release-draft --numbered-release-draft to publish.sh so that the releases for that testing branch don't show up on qTox-nightly-releases's Releases page if they somehow manage to pass. These should do, one for cleanup1 and another for cleanup2:

curl -i -X GET \
-H "Content-Type: application/json" \
-H "Travis-API-Version: 3" \
-H "Accept: application/json" \
'https://api.travis-ci.org/repo/qTox%2FqTox/builds?sort_by=finished_at%3Adesc&offset=0&limit=100&branch.name=master'

curl -i -X GET \
-H "Content-Type: application/json" \
-H "Travis-API-Version: 3" \
-H "Accept: application/json" \
'https://api.travis-ci.org/build/674008422?include=job.allow_failure%2Cjob.state'

I took those URLs from this failed job. Just want to see what the API server responds with exactly when it responds with an error -- headers and the body. That should tell more about what is going on.

If curl doesn't reproduce the error, I could release a new version of ci-release-publisher with an option to print response headers and body on a failed request, however that will take a bit more effort to do so it's easier to give curl a try.

[nurupo]

Actually, before you do any of what I suggested in the post above, try setting Travis-CI API token again. You see, HTTP 403 is what Travis API returns when you give it an invalid API token. I think that is likely the cause of all this.

Here are curl examples with an invalid API token:

curl -i -X GET \
-H "Content-Type: application/json" \
-H "Travis-API-Version: 3" \
-H "Accept: application/json" \
-H "Authorization: token xxxxxxx_xxxxxxxxxxxxxx" \
'https://api.travis-ci.org/repo/qTox%2FqTox/builds?sort_by=finished_at%3Adesc&offset=0&limit=100&branch.name=master'

curl -i -X GET \
-H "Content-Type: application/json" \
-H "Travis-API-Version: 3" \
-H "Accept: application/json" \
-H "Authorization: token xxxxxxx_xxxxxxxxxxxxxx" \
'https://api.travis-ci.org/build/674008422?include=job.allow_failure%2Cjob.state'

You can test those and see that they return HTTP 403 with "access denied" body. Then replace xxxxxxx_xxxxxxxxxxxxxx with a valid Travis API token and see, locally on your machine, if you can get HTTP 200.

[anthonybilinski]

Trying to hit the API myself, I still get access denied. I'm using the qTox-bot's "API authentication" token from https://travis-ci.com/account/preferences. If I do a POST to start a build off master instead of the get you provided, I get back "operation requires create_request access to repository". I wasn't involved with the initial linking of qTox-bot and travis's qTox repo, but maybe there's an issue there?

[nurupo]

Try the API token from .org instead of .com, since we are using the .org API, i.e. get it from https://travis-ci.org/account/preferences instead.

[nurupo]

Oh, I linked to .com in my initial post. Welp. Must have copy-pasted from the CIRP's README. .com is the default for new projects so the README uses .com everywhere, sometimes mentioning about existence of .org.

[anthonybilinski]

The problem was indeed the .org / .com mixup. I re-set the token from the .org site, and cron jobs and now completing again, including the new v1.17-dev branch nightly: https://github.com/qTox/qTox-nightly-releases/releases/tag/ci-v1.17-dev-latest which can be used for release testing.

Thanks nurupo!

[nurupo]

No problem, feel free to ping me if anything happens again.