Runner: Auto updater wants to update to a pre-release version

[steven166]

Describe the bug
The auto updater of the gha runner want to upgrade to a prerelease

To Reproduce

1. Start the runner on version 2.263.0
2. It will show the following message in the output:

Runner update in progress, do not shutdown runner.
Downloading 2.267.1 runner

Expected behavior
I don't want auto update at all, but being auto updated to a pre-release version is far from acceptable.
At least I only want to be updated to a stable version.

Runner Version and Platform

2.263.0

OS of the machine running the runner? OSX/Windows/Linux/...
Linux

[j3parker]

We're fine with auto-updating, but when it happens it can mean extra latency before accepting builds since we auto-scale our runners and plan to terminate them after each build (the --once option). We'd like to know which exe we should be baking into our VMs (with Packer) so that we can typically avoid this.

We have the runner version pinned in our Packer build. We poll https://api.github.com/repos/actions/runner/releases/latest hourly. When there is a new version we update our build. We use virtual-environments to configure our VMs and all-told it takes about 4 hours to build (for Linux). So in total it can take us up to 5 hours before we have the latest runner on our VMs.

Should we be using the pre-release versions instead? We don't care either way.

In a perfect world we'd get ~5 hours to update our VMs on our own before an update was forced but it's not the end of the world for us if they update automatically sooner (e.g. when you have urgent patches). We'd just like clarity on which channel to track for our VMs.

[npalm]

We face the same issue, auto upgrade is ok but auto upgrade to a pre release is not OK to us. We auto scal up and down runners and cache the distribution for perfornamce reasons. This cache is sync with releases, and not with pre releases. Please can you explain why the 263 runner version is auto updating to a pre release?

[erulabs]

Auto-upgrade is a bit of a nightmare - causes github-runner to be fairly unreliable when running inside a container. I was previously using a hack where I would upgrade to the latest runner whenever my runner boots up, thus avoiding the default crash-loop behavior (update -> crash -> start with old version -> repeat). The entrypoint for my runner looks like:

#!/bin/sh

# Try to download latest update on startup
GITHUB_RUNNER_VERSION=$(curl -s https://api.github.com/repos/actions/runner/releases/latest | jq '.tag_name' --raw-output | cut -c2-)
echo "Downloading Github Runner ${GITHUB_RUNNER_VERSION}"
curl -Ls https://github.com/actions/runner/releases/download/v${GITHUB_RUNNER_VERSION}/actions-runner-linux-x64-${GITHUB_RUNNER_VERSION}.tar.gz | tar xz

registration_url="https://api.github.com/repos/${GITHUB_OWNER}/${GITHUB_REPOSITORY}/actions/runners/registration-token"
echo "Requesting registration URL at '${registration_url}'"

payload=$(curl -sX POST -H "Authorization: token ${GITHUB_PAT}" ${registration_url})
export RUNNER_TOKEN=$(echo $payload | jq .token --raw-output)

./config.sh \
    --name $(hostname) \
    --token ${RUNNER_TOKEN} \
    --url https://github.com/${GITHUB_OWNER}/${GITHUB_REPOSITORY} \
    --work ${RUNNER_WORKDIR} \
    --unattended \
    --replace

remove() {
    ./config.sh remove --unattended --token "${RUNNER_TOKEN}"
}

trap 'remove; exit 130' INT
trap 'remove; exit 143' TERM

./run.sh "$*" &

wait $!

However, because it appears the auto-updater wants to update to pre-release versions, I've modified it to not use the _latest_ release, but the _newest_ release:

GITHUB_RUNNER_VERSION=$(curl -s https://api.github.com/repos/actions/runner/releases| jq '.[0].tag_name' --raw-output | cut -c2-)

It would be quite nice to be able to disable the auto-updating. Will look into this eventually. Note that this hack is published as kubesail/github-actions:v19 if anyone else would like to make use of it :)

[myoung34]

Im unrelated to the project but I'm curious what makes 2.267.1 a pre-release. This was opened 10 days ago but 2.267.1 was released 19 days ago and doesn't show anything about being a pre-release afaict

I can try to work with you on this from my docker runner: https://github.com/myoung34/docker-github-actions-runner
by pinning to non-prerelease for specific tags

[npalm]

@myoung34 indeed the release was created 19 days ago, but it was marked as a pre-release till July the 15th or 16th. Unfortunately you cannot see anymore that the pre-released was promoted to a release.

[bryanmacfarlane]

The runner rolls out across rings. It's a service side config change and it's been tesedt, baked in all previous rings (including self host, github load, etc). So if it progressed beyond ring 0 into customer rings it's baked and not pre-release. My guess is someone didn't uncheck that box. @TingluoHuang

@erulabs as the related issues noted, run.sh should only be used for interactively running. In a container, you should call ./runsvc.sh. That gracefully handles updates.

[arash-bizcover]

@bryanmacfarlane How can I pass --once to the runsvc.sh ?

I'm using ./run.sh --once which help me have clean new containers after each build.

BTW on the latest release: v2.272.0 runsvc.sh is inside bin bin/runsvc.sh

[pietroalbini]

This happened again: 2.273.0 is being rolled out while being marked as a pre-release (we hit it in rust-lang-ci/rust).

[j3parker]

It would be nice to have some official guidance on this. E.g., "it is safe* to use the pre-release versions"

What we're looking to avoid is baking in an out-of-date version into our VMs. When we spin up to run a job the runner will sit idle with the out-of-date build until the first job comes in. It will accept that job, download the update, restart and then run the job. We are about to switch to disposable VMs which means that we will download the update for every job run.... we just want to avoid this latency.

Right now we poll the GitHub releases API hourly to get the download URL and pre-download the runner into our VM template (AWS AMI).

In a perfect world there would be some standard window of time we have to update. E.g. we can run the previous release or pre-release build for 24 hours/an hour/whatever before an auto-update (which would give us time to update our VMs)

[arash-bizcover]

Since Microsoft purchased Github ....

[hross]

Apologies for any confusion caused by our runner release process and how it may impact rollout. For transparency's sake, here is the current process that has been followed since actions GA (as far as I'm aware):

1. Runner updates are made to this repo and committed to master
2. Changes are tested internally at GitHub
3. Changes are rolled into a release as a prerelease
4. The prerelease runner is slowly rolled to different rings of customers via autoupdate, starting with internal customers and moving to larger and larger rings of customers
5. When the changes have rolled to every customer on the service and our telemetry indicates there are no issues, we mark the runner as "released"

Prerelease versions of the runner are considered stable, working and are rolled out to customers (obviously bugs can happen and we might have to roll back a runner version, which is why we would consider it prerelease until it has made it to all of our customers).

We could change the runner version to a "release" after it makes it through our internal customers and testing. That might help with the confusion on the terminology, although it would not change our testing process (just marking it earlier as released).

[j3parker]

Thanks! That's perfect.

We'll change our stuff to opt-in to the pre-release runners.

[nanddalal]

@hross do we think the closing criteria for this issue is an official recommendation (perhaps by a line in the README) about how to subscribe to the proper version (via curl for example)?

[npalm]

@hross thanks for the clarification. For starting our dynamically runners fleat we cache ther runner binary. Therefor we need to find out which version we put in cache. So htis requires we are able find automatically which binary we should download. Two scenarios would work for us

1. Upgrade to the pre-release as soon it is available, runners should keep working even if we or not in the upgraded ring.
2. At the moment you start rolling out to customers you mark the pre release as release. But this requires also that we should be able upgrade to the latest version, even if we are not in the upgraded ring.

[npalm]

@hross could it be the case that issue #592 and #679 are related to the way of pre-releases and the updates in rings?

[hross]

@npalm those issues are unrelated to runner releases and have to do with infrastructure issues. I would recommend using the prerelease right now. I think it's likely we will just mark the runner as released after we internally release it rather than make other changes (but I am following that up internally before I comment here).

[hross]

We are going to start marking the runner as "released" after we go through our internal customers. That should eliminate any confusion and allow people to continue to work on released versions of the product.

Thank you for all of the input and feedback.

TLDR: no changes needed on the community side and our release process should give you the right version of the runner. If you are downloading the latest releases continue to do so.

[nanddalal]

@hross thanks for the update! Just for clarification and following up @erulabs solution, does this mean it is always safe to use the following command to get the current version of the runner?

curl -s https://api.github.com/repos/actions/runner/releases/latest | jq '.tag_name' --raw-output | cut -c2-

[hross]

@nanddalal that seems reasonable, yes

[npalm]

@hross thanks for the follow up and giving a clear direction and expectations.