Runner: Queueing of self-hosted runners is busted

[timharris777]

UPDATE: Updated title and description of issue based on further findings.

Summary:

"If a job runs but there are no runners available (either they are all busy with other jobs or all offline) then it no longer waits but rather aborts immediately. This doesn't appear to be anything to do with the runners themselves but rather be whatever is responsible on GitHub's severs to dispatch job requests to available runners." - @Bo98

Details:

We have ephemeral self-hosted runners running in AWS. When a new workflow is triggered on a repo we have github webhooks calling an api that runs ephemeral runners in AWS ECS Fargate. We are using a custom label to choose which ephemeral runner to use. In order to do this we had to register a runner with that label and then leave the "offline" runner entry available in the action runner list for the repos. If we didn't do this we would get an error saying that no runner with the label existed. This has worked just fine up until a few days ago. Now we have to have a runner with the "label" that is online or idle. Otherwise we get the following error:

An image label with the label menuintegrationdev does not exist.

The remote provider was unable to process the request.

[timharris777]

@wasdin and @joncfa

[peterkline-es]

I can confirm that we are seeing the same error but with slightly different circumstances. One of our runners is hosted in AWS but it is not ephemeral (the system is online 24/7). We seem to fail the action stating the same error above but if I re-run the job manually it works just fine. Also our runner has NOT been updated to 267.1 but is still at 263.0 as the runner hasn't updated yet. So its server side?

[Bo98]

Queueing of self-hosted runners is busted right now and has been for a couple days now.

If a job runs but there are no runners available (either they are all busy with other jobs or all offline) then it no longer waits but rather aborts immediately. This doesn't appear to be anything to do with the runners themselves but rather be whatever is responsible on GitHub's severs to dispatch job requests to available runners.

Other reports: https://github.community/t/gha-internal-error-an-image-label-with-the-label-name-does-not-exist/121725
(+ various emails and tickets)

[timharris777]

@chrispat and @bryanmacfarlane, are y'all aware of this issue?

[lokesh755]

We've identified the bug and currently rolling out the fix. Will update again once the fix is rolled out.

[lokesh755]

Fix was rolled out.