Test-infra: Tide status controller query frequently gets a 502

Created on 13 Oct 2020 · 6Comments · Source: kubernetes/test-infra

What happened:
The tide status controller query frequently gets a 502 from github which in turn renders it unable to post the tide status which confuses uses. Error looks something like this:

{
  "component": "tide",
  "controller": "status-update",
  "error": "non-200 OK status code: 502 Bad Gateway body: \"{\\n   \\\"data\\\": null,\\n   \\\"errors\\\":[\\n      {\\n         \\\"message\\\":\\\"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `B5BF:7B03:9DCC9:112CDB:5F85F07D` when reporting this issue.\\\"\\n      }\\n   ]\\n}\\n\"",
  "file": "prow/tide/status.go:566",
  "func": "k8s.io/test-infra/prow/tide.(*statusController).search",
  "level": "error",
  "msg": "Search failed",
  "query": "is:pr state:open sort:updated-asc org:\"cri-o\" [many repos]",
  "severity": "error",
  "time": "2020-10-13T18:23:04Z"
}

Github support advised us to shard the query so that is what we should do.

/kind bug
/area prow
/area prow/tide

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Please provide links to example occurrences, if any:

Anything else we need to know?:

areprow areprotide kinbug

Source

alvaroaleman

All 6 comments

In https://github.com/kubernetes/test-infra/pull/19743 we reduced the number of results per page for the tide queries from 100 to 37 which should reduce githubs backend load to a bit more than a third of what we have. Lets defer doing anything here until we saw the 502s again after that change got rolled out

alvaroaleman on 29 Oct 2020

👍1

@alvaroaleman what's the status of this? Have you seen some 502s since?

matthyx on 11 Jan 2021

Hey, nope, hasn't happened anymore for us

alvaroaleman on 11 Jan 2021

ok so maybe we can close?

matthyx on 13 Jan 2021

👍1

/close

If someone else sees this happening again with a current version of Prow, feel free to re-open

alvaroaleman on 13 Jan 2021

@alvaroaleman: Closing this issue.

In response to this:

/close

If someone else sees this happening again with a current version of Prow, feel free to re-open

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.