Test-infra: Prow: jobs trees should be able to share data

[BenTheElder]

We'd like to be able to use run_after_success to do builds and then run tests against them, previous discussion here: https://github.com/kubernetes/test-infra/pull/4047

In order to do this we need a common ID or path from which to determine a common path between jobs and their run_after jobs.

[BenTheElder]

/area prow
/assign

[BenTheElder]

Previously mentioned options:

• inform job runs of their UID and "parent" UID for the job they ran after
• use an "invocation ID" for a tree of jobs
• use git SHAs

[BenTheElder]

After some more discussion I'm in favor of using gs://some-bucket/$PULL_REFS/ and allowing jobs to take it from there. All jobs should know this without modification to Prow and this will be more useful to debug than a UUID. We can have bootstrap handle uploading a dir similar to _artifacts/ -> gs://<log path>/artifacts (_results/ -> gs://some-bucket/$PULL_REFS/$job/ ?) and then child jobs can fetch from gs://some-bucket/$PULL_REFS/* as needed.

Bootstrap can provide a flag to override this location for local testing.

[spxtr]

Sounds good, and I'm a fan of not complicating prow's interface for a feature that is already possible.

[BenTheElder]

In the future we'll probably want something else like this but for now it looks like the only use case is separating build jobs and test jobs which works without even bootstrap knowing about this, just those scenarios. We can circle back to this later if there is more interest.

If anyone has a different concrete need please reopen or reference this issue and I'll look again.