Test-infra: gubernator can't handle build logs bigger than 256MB ?

[yutongz]

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_istio/3459/istio-pilot-e2e/3012/
The full log can be accessed by directly going into the GCS bucket.

Not sure if that's designed to be, or because we are using some old prow version. We probably shouldn't dump out such a big log, but just curious if we know that limitation.

[BenTheElder]

1) gubernator's deployed to appengine, the instance is probably only 256mb, but it could be deployed to a larger isntance
2) why are your logs more than 256MB?

[BenTheElder]

/area gubernator
/cc @rmmh

[stevekuznetsov]

See also https://github.com/kubernetes/test-infra/issues/6092

[BenTheElder]

some options:

• port gubernator to something else (PRs welcome?)
• use ginormous gubernator deployments
• don't have crazy enourmous logs (which, we also parse these for highlighting etc, so maybe just don't have crazy huge logs?)
• maybe gubernator can just refuse to display large logs and link to GCS. not sure if the APIs are there to check size before accessing. (probably?)

As an aside I'm actually still waiting for https://storage.googleapis.com/istio-prow/pull/istio_istio/3459/istio-pilot-e2e/3012/build-log.txt to load 2 minutes later, just how large is this? Somehow I don't think other hosted CIs support log files this large.

[ldemailly]

1. is being addressed as https://github.com/istio/istio/issues/3499

[ldemailly]

as a side note, circleci does print the first (or last I forgot) 40k lines and then puts a download link

the issue here is the log is stored compressed (26Mbytes) and you need to decompress it to find out it's so big (doing 2 passes just to get the size seems too much)

maybe a heuristic is to just refuse to handle any file > 10Mbytes compressed and tell the users (nicely) to fix their damn build in that case, without blowing up (and with a link to the gcs bucket) ?

[stevekuznetsov]

Porting Gubernator away from Python and onto a k8s deployment is a nice far-out goal, for sure. In the meanwhile I think we just need to break off all of the (really) rough edges when Gubernator encounters a non-normal situation. Have it show a nice page and link to the GCS log. Maybe even put some CSS on the page :)

[mattdelco]

The different test attempt triggers the stack backtrace variation of the issue:

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_istio/3459/istio-pilot-e2e/3011/

[BenTheElder]

yeah, I'd love to see it ported off and deployed with the rest of prow to k8s but nobody is working on this right now.
CSS is being addressed by @qhuynh96

Detecting the size and handling it gracefully SGTM, I don't think I have bandwith for this currently though and Ryan is out, I'll ask him when he's back or happily review PRs with such a heuristic.

[BenTheElder]

/assign @qhuynh96 @rmmh

[BenTheElder]

/kind bug