Runner: expose runner-status in API

[davidkarlsen]

Describe the enhancement
Today the api only exposes on/offline status: https://developer.github.com/v3/actions/self-hosted-runners/#list-self-hosted-runners-for-a-repository

It would be better if the API could show if the runner is occupied with a job or not (like in the UI).

We want to use this to dynamically provide self-hosted runners based on demand.

usecase:

1. package runner as docker image
2. have a k8s controller: https://github.com/evryfs/github-actions-runner-operator/
3. make runner spawn runners based on CRD high/low water marks and demand for runners.

Also - ideally there should be a possibility to add labels to runners, so that we can have different pools of runners for different environments. These labels should be reflected in the API so we can distinguish them both when listing runners, as well as using in jobs to select the appropriate type of runner.

Code Snippet
status, enum of (online,offline,queued) (offline,idle,queued)
labels, map of
key: val

Additional information
Killer feature for providing dynamic build-pools which again can help park Jenkins for good :)

[davidkarlsen]

Any thoughts on this @bryanmacfarlane ?

[davidkarlsen]

Bump. @bryanmacfarlane

[missedone]

i'm looking for the similar features.
i'd like to implement auto scaling out for the runners based on metrics like how busy it is.

[bryanmacfarlane]

@hross to look at priority

[bryanmacfarlane]

I pinged some folks to at least get some 👀 on it :)

[hross]

I added this for internal discussion. I like the feature, I think it deserves some priority. Let me see where we can fit it...

[hross]

one thing to note here is even if you get this API you may have a race condition unless you can disable an agent (stop it from picking up more builds) because it might pick up another build while it is idle, before you kill it.

[davidkarlsen]

@hross Yes - it won't be bullet-proof. It's possible to add a preStop hook: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/ which could tail the status of the build log in order to let the running build finish - but it's a bit of a hack. But at least it should be easy to scale up this way.

[lomkju]

@davidkarlsen @hross We run actions on VMs rather than on k8s (Even though all other infra is on k8s). We don't want the overhead of running k8s on runners (ds, kubelet ...).

It would be great if the runner could understand different signals.

SIGTERM: Cancel the current job and EXIT.
SIGWINCH: Wait for job to finish and EXIT.

We would want the latter as when we downscale our autoscaling group would want the job to finish before the runner exits.

[davidkarlsen]

It seems this has been added to the API now... https://github.com/google/go-github/issues/1589