Test-infra: triage is failing

[BenTheElder]

the triage periodic is now failing continuously with errors https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-test-infra-triage/1157331701362331652

+ bq extract --compression GZIP --destination_format NEWLINE_DELIMITED_JSON k8s-gubernator:temp.triage gs://k8s-gubernator/triage_tests.json.gz
BigQuery error in extract operation: Error processing job
'k8s-gubernator:bqjob_r7ddb74232fc611d3_0000016c533a4156_1': Table
gs://k8s-gubernator/triage_tests.json.gz too large to be exported to a single
file. Specify a uri including a * to shard export. See 'Exporting data into one
or more files' in https://cloud.google.com/bigquery/docs/exporting-data.

https://prow.k8s.io/?job=ci-test-infra-triage
xref: https://github.com/kubernetes/test-infra/issues/13730

[spiffxp]

https://testgrid.k8s.io/sig-testing-misc#triage&width=5&graph-metrics=test-duration-minutes

this seems like a "bad data in" situation, what has been growing to trip us over this threshold?

[spiffxp]

/area triage
/sig testing
/priority critical-urgent
go.k8s.io/triage is going to be stale until this is fixed

[clarketm]

As the error suggests, bigquery doesn't allow extraction from a table to a single file if the table is 1GB+ and this table is just over that threshold.

We can extract to multiple files, but just need to ensure we aggregate before feeding to summarizer.py since it expects a single file.

- gs://k8s-gubernator/triage_tests.json.gz
+ gs://k8s-gubernator/triage_tests*.json.gz

[cjwagner]

need to ensure we aggregate before feeding to summarizer.py since it expects a single file.

I don't think this is true? https://github.com/kubernetes/test-infra/blob/6a85e1e9718d1d43aaa4b756826b36e8d22eff71/triage/summarize.py#L563

[cjwagner]

Green run: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-test-infra-triage/1158455335996887042

[BenTheElder]

🎉

On Mon, Aug 5, 2019 at 12:54 PM Cole Wagner [e-mail]
wrote:

Green run:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-test-infra-triage/1158455335996887042

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/test-infra/issues/13744?email_source=notifications&email_token=AAHADK5MY4T272DTO3P3YTLQDCANPA5CNFSM4IJAIXZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3S4OVQ#issuecomment-518375254,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHADK3HMN2YWLFOYUP4JD3QDCANPANCNFSM4IJAIXZQ
.