Runner: Support for autoscaling self-hosted github runners

[jwcmd]

Describe the enhancement
I'm looking for a way to put a self-hosted github runner into an autoscale group.

I've discussed with Github Support and they've explained that the tokens are only valid for one hour. That's problematic for an autoscale group because it means they will fail to bring up a runner an hour after I deploy the autoscale group. They recommended raising my issue here, I apologize if we've both missed an obvious solution for this.

Code Snippet
Not Applicable.

[j3parker]

In AWS we do this:

GitHub App tokens:

• We register a GitHub App which has permission to register runners
• We store its secrets in AWS secrets manager
• A cloudwatch scheduled event triggers a lambda which generates a github token for that app and stores it into secrets manager

Runner registration:

• A cloudwatch event trigger is setup for EC2 instances in the pending state.
• Those events trigger a lambda which looks at the Org and Repo tags of the launched instance to decide where to register the runner (this lambda has permission to read the github token secret)
• The lambda fetches a registration token and puts it into the EC2 Parameter store, prefixed by the instance name

The runner instance:

• The IAM role for that instance has permission to read parameter store values for its instance id
• On boot it polls parameter store waiting for a registration token

The ASGs:

• We run an ASG for each kind of runner we want, and it uses the Org/Repo tags for its instances
• When running ephemeral runners (waiting on GitHub to finish that feature so we don't do this yet) when a job gets picked up by the runner it removes itself from the ASG, causing a new runner to be launched/boot/etc. (in this mode, "desired" = how many pre-warmed runners do we want to have on standby to pick up new jobs... our max # of concurrent jobs is only limited by EC2/our accounts EC2 limits)
• When running a fixed pool of re-usable runners we use scheduled scaling events

The way we do registration is so that:

• Runner VMs never have access to GitHub app creds or tokens
• The registration lambda only has access to the token, not the long-lived creds
• If GitHub has a blip (heh) we have the cached token.... we try to refresh it every 30 mins but its good for an hour and we will keep using the old one for that long if new tokens are failing
• The instance polls for the registration token... its usually there on the first poll but if GitHub or AWS were to have a blip it will still work fine
• It's also easy to manually launch a runner in the console if needed... just specify the Org/Repo tags