Test-infra: Triage is serving stale data - feed stopped on Nov 21st

Created on 24 Nov 2020 · 9Comments · Source: kubernetes/test-infra

What happened:

On Triage, aggregated test error results are not up to-date.

https://storage.googleapis.com/k8s-gubernator/triage/index.html

What you expected to happen:

See aggregated test results from today, Nov 24th.
There are no results since Nov 21st.

How to reproduce it (as minimally and precisely as possible):

Visit https://storage.googleapis.com/k8s-gubernator/triage/index.html
Read charts

kinbug

Source

RobertKielty

👀3

All 9 comments

/assign
Kettle is having a number of issue that are being slowly mitigated. I am not sure if there are a number of bad jobs that are causing failures or the sheer number of jobs are causing it to lock up. I can manually stream results to BQ form kettle to update data. I have been working on the few issue I still see. But has been a war of attrition.

MushuEE on 30 Nov 2020

👍1

This is being caused by #20024

I think I have a partial fix that will at least keep data fresh for ~3 month periods. Though there is a LOT of data now stored in kettle's DB and it is having a lot of trouble keeping up.

MushuEE on 1 Dec 2020

Checked triage today and it is still only serving data up to Dec 1st
https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=instrumentation

RobertKielty on 14 Dec 2020

@RobertKielty Waiting on #20091

Kettle is locking up (not 100% sure why, but seems related to size of data) that PR should limit it to only look for new build in the last 30 days and I was successful in running this manually. Just need lgtm

MushuEE on 14 Dec 2020

👍1

It looks like the Triage upload job has been failing for the past day and a half. I can't tell for certain that these two issues are related, only that the timeline kind of matches up.
https://github.com/kubernetes/test-infra/issues/20259

chases2 on 16 Dec 2020

100% kettle on the Triage failures. "Fix" merged yesterday and updated kettle last night. I do see data in Triage again. Will keep this open for a while to make sure data is consistent.

MushuEE on 17 Dec 2020

This looks like it is now being consistently updated

MushuEE on 29 Dec 2020

/close

MushuEE on 29 Dec 2020

@MushuEE: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.