Test-infra: PR testing jobs not launching with "Error finding Jenkins job."

[viglesiasce]

Hey Team!

Test jobs are unable to launch and the status in Github shows "Error finding Jenkins job."

Two examples from different projects:
https://github.com/kubernetes/heapster/pull/1907
https://github.com/kubernetes/charts/pull/2962

[viglesiasce]

cc @mattfarina

[mattfarina]

/kind bug

[BenTheElder]

xref: https://github.com/kubernetes/test-infra/issues/190 heapster and cadvisor have special jenkins VMs apparently. I'll try to take a look at this, but I suspect the jenkins-operator in prow @cjwagner @kargakis

edit: I do not suspect the operator anymore, it looks like PR jenkins is just hosed.

[BenTheElder]

I can access the agents but not the control plane currently, the VM is up however. Tentatively it looks like PR Jenkins gave out around 3PM last night. 😕

Please consider helping move heapster and charts presubmits to Prow where we have better (kubernetes!) tools for this sort of problem.

[piosz]

Thanks @viglesiasce for filling this.

@BenTheElder we will definitely migrate to Prow, but at this point we are about to launch v1.5.0 and the problem with e2es blocks the process. I doubt whether we will be able to migrate to Prow by Monday, so any helps with making it work would be appreciated.

cc @kawych @x13n

[BenTheElder]

I've gotten onto the master and eventually decided to reboot, it seems to many files were open, however the control plane still does not seem to be healthy after reboot.

I will continue to look into this, and I'm also trying to find someone with more knowledge of the Jenkins PR cluster but we really should consider moving these jobs instead, depending on what they are doing it may be trivial and not many presubmits depend on Jenkins at this point.

[BenTheElder]

PR jenkins should be back online now, please let me know if you see any futher issues.

[fejta]

Migrate to prow? 😈

[viglesiasce]

Got any pointers? We'd be happy to.

On Fri, Dec 8, 2017, 2:50 PM Erick Fejta [e-mail] wrote:

Migrate to prow? 😈

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/test-infra/issues/5874#issuecomment-350368724,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZCp2s9LVMYvlR_tY4nTEat5NC-0ScOks5s-aEmgaJpZM4Q7TSG
.

[BenTheElder]

@viglesiasce Prow runs a kubernetes pod with one container for each job, we then build an image based on our runner which currently handles logging, artifact upload etc (to be moved out potentially in the future).
These jobs should already be running with the runner script (jenkins/bootstrap.py) but we need to get them containerized and away from using anything special in the jenkins projects.

[BenTheElder]

Some bit of pointer here: https://github.com/kubernetes/test-infra#create-a-new-job

I'd be happy to create the actual configuration entries in prow/config.yaml and jobs/config.json but I'd appreciate help figuring out what these tests need to run and making sure we can containerize that and eliminate any dependency on the GCP project they are currently in with Jenkins.