Actions-runner-controller: Support ephemeral runners to minimize committed resource usage

[andyfeller]

From studying the current code base, it seems that Runner, RunnerDeployment, RunnerReplicaSet are backed by always on Pod resources. If this is the case, there is a concern about having committed resources sitting idle within the cluster for individual projects, not to mention doubled or tripled as more concurrency is desired.

I'm open to any options on the table to reduce the amount of resources committed while being idle, not simply via ephemeral runners. Any thoughts and ideas would be appreciated!

[mumoshu]

@andyfeller Hey! Thanks for the suggestion and the proposal.

I thought ephemeral containers would help at first. But now this is looking more like #10. Can we merge this into #10?

To avoid consuming the committed resources while idling, I believe that you either need to launch an ephemeral container or a pod on demand.

This means that you would need to poll Actions API to see there are any pending jobs so that the controller can react by adding the requested resource via ephemeral containers or pods.

Choosing pods for the provider of demanded resources, maybe it's more understandable as it looks very similar to how HPA works. If we choose ephemeral containers, perhaps we'll have another pro/cons?

[summerwind]

Ephemeral runners is an interesting idea to me.

Running a runner using Kubernetes' Job resource can save resource usage and recreating the Runner every time will force the workflow to run in a clean environment. Controller implementation will also might be simple.

On the other hand, as described by @mumoshu, it is necessary to poll the Actions API to detect pending jobs. This will take longer to start the Job than keeping Runner running. It seems to be a factor to limit the number of polls due to the rate limit of GitHub API.

It might be nice to have a solution that achieves both 'auto scaling' and 'scale to zero' :grinning:

[kuuji]

Just adding my 2 cents to this conversation.
How would you guys feel about a webhook approach instead of polling the API. I know that implies that you now have to expose the controller to the outside (or it could just be a webhook service) but that would rule out any sort of rate limit on the GitHub API.

[andyfeller]

Just adding my 2 cents to this conversation.
How would you guys feel about a webhook approach instead of polling the API. I know that implies that you now have to expose the controller to the outside (or it could just be a webhook service) but that would rule out any sort of rate limit on the GitHub API.

TBH, I’m not going to bat for opening up the entire Azure list of public CIDR blocks to my corporate and production networks. No way Security or Network engineers will go for that.

[kuuji]

Just adding my 2 cents to this conversation.
How would you guys feel about a webhook approach instead of polling the API. I know that implies that you now have to expose the controller to the outside (or it could just be a webhook service) but that would rule out any sort of rate limit on the GitHub API.

TBH, I’m not going to bat for opening up the entire Azure list of public CIDR blocks to my corporate and production networks. No way Security or Network engineers will go for that.

Well that's how prow works for example (see details here)
And that's the CI a lot of major kubernetes project use (including kubernetes itself).
Not saying that it means it's the best way to do it but it's definitely a pattern widely used.

You would be exposing a webhook service that only github would call (I believe they do provide a list of their cidrs, so you could restrict to that only). And you should also protect the webhook itself with a secret key.

[mumoshu]

Yeah, I understand you both. Anyways, security-vs-operatability-wise, there's no one thing that fits all.

Also - @kuuji, you may be confusing this with a regular CI that receives webhook events from GitHub. AFAIK there's no event for receiving GitHub Actions jobs which is what we want here.

https://developer.github.com/webhooks/#events

And the github actions runner itself(not controller) uses polling for receiving jobs. So exposing a webhook endpoint from the controller wouldn't affect the overall architecture.

[kuuji]

Yeah, I understand you both. Anyways, security-vs-operatability-wise, there's no one thing that fits all.

Also - @kuuji, you may be confusing this with a regular CI that receives webhook events from GitHub. AFAIK there's no event for receiving GitHub Actions jobs which is what we want here.

I did mean github actions jobs event. That's unfortunate.

So what would be the idea? Call the api regularly for pending jobs and publish the result as a metric to the metrics api. That way it can be used by hpa?

developer.github.com/webhooks/#events

And the github actions runner itself(not controller) uses polling for receiving jobs. So exposing a webhook endpoint from the controller wouldn't affect the overall architecture.

[mumoshu]

Call the api regularly for pending jobs and publish the result as a metric to the metrics api.

Yes I believe that's the only way at the moment.

That way it can be used by hpa?

I thought HPA already supported scaling pods. Here we need to scale runners(our own custom resources). So probably we would end up implementing a purpose-built autoscaler that is partly similar to HPA, but for runner resources?

[kuuji]

Call the api regularly for pending jobs and publish the result as a metric to the metrics api.

Yes I believe that's the only way at the moment.

That way it can be used by hpa?

I thought HPA already supported scaling pods. Here we need to scale runners(our own custom resources). So probably we would end up implementing a purpose-built autoscaler that is partly similar to HPA, but for runner resources?

Doesn't the RunnerDeployment resource create a replicaSet? ReplicaSet can be the target of HPA -> https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#replicaset-as-a-horizontal-pod-autoscaler-target

Edit: Nope, I see how it works after checking the code. You recreated your own replica set that controls the number of runner resource. Then HPA won't work.

[mumoshu]

@kuuji Yes, your observation is correct. We need to hook into the pod creation, so that we can register a actions runner token and inject an envvar containing the token to the pod.

We use a variety of custom resources for the hook today. Alternatively, we could use a mutating webhook and statefulsets(#4). But it seems to require a plenty of engineering.

[iagomelanias]

I would love to work on that, but i'm very focused on a specific project right now. 😕
@mumoshu is there any reason for not using standard Deployments instead of creating a CRD?

[MPV]

This blog post describes some issues with scaling runners to zero, might be helpful being aware of?
https://dev.to/wayofthepie/hacking-together-an-actions-runner-orchestrator-5ef9

[andyfeller]

Closing in favor for https://github.com/summerwind/actions-runner-controller/issues/10

[dudicoco]

Would https://github.com/actions/runner/pull/660 make this possible?

[mumoshu]

@dudicoco Nope. But that would make our already ephemeral runners more reliable, by removing the rare chance that GitHub Actions assigns jobs to runners that are in the progress of terminating.

[dudicoco]

@mumoshu how are the current runners ephemeral if they can't be scaled down to 0?

[mumoshu]

@dudicoco They are currently ephemeral in the sense that they get recreated after each job run, and can get auto-scaled depending on pending jobs or else using our HorizontalRunnerAutoscaler.

On the other hand, it is GitHub Actions API's limitation that we cant scale down to zero. All we can do today is scale-down-to-one, because there's no (at least official'ish) way for us to "defer" running jobs when there is no registered runners at all. You need to submit a feature request to GitHub for that.

[dudicoco]

@mumoshu i'm aware of the Github API limitation, hopefully they will provide a better solution eventually.
Thank you!

[dudicoco]

Yeah, I understand you both. Anyways, security-vs-operatability-wise, there's no one thing that fits all.

Also - @kuuji, you may be confusing this with a regular CI that receives webhook events from GitHub. AFAIK there's no event for receiving GitHub Actions jobs which is what we want here.

https://developer.github.com/webhooks/#events

And the github actions runner itself(not controller) uses polling for receiving jobs. So exposing a webhook endpoint from the controller wouldn't affect the overall architecture.

@mumoshu can't we listen to workflow_run and workflow_dispatch events and check the queue on such events, rather than at a fixed interval?

[mumoshu]

For anyone interested, I've written another comment on why we use our own RunnerDeployment and other CRDs instead of regular deployments and HPA https://github.com/summerwind/actions-runner-controller/issues/245#issuecomment-754991647

[mumoshu]

can't we listen to workflow_run and workflow_dispatch events and check the queue on such events, rather than at a fixed interval?

@dudicoco I'm unsure - for what? For the scale-to-zero use-case, it doesn't help. Any run should immediately marked as failed by GitHub when there're zero runners, which makes scaling to zero impossible for us.

[dudicoco]

@mumoshu i'm not referring to the scale to zero use case.
I'm referring to how to controller scales out the runners - currently it's checking the queue on a fixed interval, the interval can't be too short as that can cause API throttling.
My suggestion is to watch for workflow_run and workflow_dispatch events, which will be sent via a github webhook, and check the queue each time such event is triggered.
What do you think?

[mumoshu]

@dudicoco How would you calculate e.g. "the current number of pending jobs" then, because that's required for autoscaling?

[dudicoco]

@mumoshu the webhook is just a trigger for the controller to check the queue, it would then check the queue exactly the same way it does it today with the fixed interval.

[mumoshu]

@dudicoco As we need to call some GitHub API to determine the queue length, wouldn't that result in excessive API calls proportional to number of webhook events received?

[dudicoco]

@mumoshu I guess you're right. How are you handling jobs waiting in the queue today?
The way I see it is either over provision the number of runners which can result in a lot of idle runners, or have jobs waiting in the queue for the controller to check the queue (which can take a long time in order to avoid throttling).