Test-infra: Add data from PRs to 24 hr test history

[fejta]

Right now we only aggregate results from the ci-* jobs. Would love to include information from the pull-* jobs as well.

• The dashboard is here
• Code that generates these pages is here.
• The CI configuration for this job is here and here

Other improvements are also welcome!

[fejta]

I suspect this work is

a) Adding gs://kubernetes-jenkins/pr-logs/pull to buckets.yaml
b) speeding up the whole process so it doesn't take a million years to complete
c) most likely by

• actually adding gs://kubernetes-jenkins/pr-logs/directory to buckets.yaml
• making the GCSClient in gen_json.py aware of the pattern where instead of gs://BASE/JOB containing a list of BUILD folders, you instead have a bunch of BUILD.txt files (example) that contain the name of the gs://OTHER_BASE/JOB/BUILD/ path that contains the actual data files we need to parse.

[grodrigues3]

@crimsonfaith91 this could be a good project as well

[crimsonfaith91]

I will start working on this issue. :)

[spxtr]

Note that it looks funky since we just got rid of merge-blocking jobs.

[fejta]

@crimsonfaith91 thanks! Sent you an invite to collaborate on this repo. Once you receive it you can /assign this issue to yourself.

[crimsonfaith91]

@fejta noted! i will work on this issue first. thanks! :)

[crimsonfaith91]

/assign @crimsonfaith91

[crimsonfaith91]

@fejta Do you mean changing this when you mentioned BUILD.txt?

I apologize if it is not a good question. I am new to the Jenkins repo (and Python). Thanks! :)

[fejta]

Intent here is to note the differences:

• CI results use paths like logs/JOB_NAME/BUILD_NUMBER and in that folder is a bunch of files like started.json, finished.json, etc.
• PR results use paths like pr-logs/directory/JOB_NAME path has a pr-logs/directory/JOB_NAME/BUILD_NUMBER.txt file, the contents of which are a gs://kubernetes-jenkins/pr-logs/PR_NUMBER/JOB_NAME/BUILD_NUMBER that store a link to the actual started.json etc files

So we will need to make the gen_json.py file aware of this directory pattern -- read the poor man's symlink at BUILD_NUMBER.txt and read the actual path to get the json/junit files.

[crimsonfaith91]

thank you very much for the very detailed explanation! many thanks!

[crimsonfaith91]

@fejta I will get back to this when I resolve the inactive reviewer issue. Sorry for the delay!

[fejta]

Testgrid and triage sites have deprecated the 24 hr test history