Actions-runner-controller: Runner upgrade to 2.276.0 failed

[jolestar]

Runner auto-upgrade to 2.276.0 failed

2021-01-20 01:59:58Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.276.0 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should back online within 10 seconds.
renamed '/runner/_diag/SelfUpdate-20210120-020013.log' -> '/runner/_diag/SelfUpdate-20210120-020013.log.failed'
Runner listener exited with error code 4
Runner listener exit with undefined return code, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process
events.js:187
      throw er; // Unhandled 'error' event
      ^

Error: spawn /runner/bin/Runner.Listener ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
Emitted 'error' event on ChildProcess instance at:
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: 'ENOENT',
  code: 'ENOENT',
  syscall: 'spawn /runner/bin/Runner.Listener',
  path: '/runner/bin/Runner.Listener',
  spawnargs: [ 'run', '--startuptype', 'service', '--once' ]
}

[mumoshu]

@jolestar Hey! I was thinking that this might be due to #252 that can be fixed via #253. I thought I'd release it soon but had no time to test it. Could you confirm/test it if possible?

[jolestar]

@jolestar Hey! I was thinking that this might be due to #252 that can be fixed via #253. I thought I'd release it soon but had no time to test it. Could you confirm/test it if possible?

Ok, I try to build a new image. the latest base image is still v2.275.1 ?

[jolestar]

@mumoshu I read #253, If I want to test it, do I just need to build a new runner image? or also need to upgrade the controller?

[mumoshu]

@jolestar You need both runner and controller images upgraded, as the new controller code depends on the new runner image this time.

[shadiramadan]

@jolestar were you able to resolve the self update issue?

[jolestar]

@shadiramadan resolve via #253

[terry-yolabs]

I am hitting the same error. I wasn't able to resolve it by creating containers using the branch in #253. This error is blocking development work on a new CD workflow so I'm hoping to find a solution quickly. Is there something I can do to assist testing?

[achedeuzot]

Hi !

@mumoshu I can confirm it works with the PR on the branch fix-self-update-failure with a few minor fixes.

This is what I did:

• USER runner instead of USER runner/runner in the runner Dockerfile
• adding sudo chown -R runner:docker /runner and mkdir ./externals in the runner/entrypoint.sh file
• updating the controller image in my k8s cluster to my custom built one
• updating my custom runner image to build from the patched runner image

➡️ everything is working again 😁

[shadiramadan]

@mumoshu Do you have an ETA on when you can incorporate these fixes into #253 for a release? Just want to know if I need to follow @achedeuzot recommendations and make a local build or wait.

[kaykhancheckpoint]

Also experiencing this now

[kaykhancheckpoint]

@achedeuzot do you have your docker image available publically somewhere?

[hfuss]

Hi folks, we just tested @achedeuzot's fixes on https://github.com/hfuss/actions-runner-controller/tree/testing-self-update-fix-2.276.0 and also had success. You can find the images here in the interim:

• Patched to handled auto-update: brix4dayz/actions-runner:v2.275.1
• To avoid auto-update altogether: brix4dayz/actions-runner:v2.276.0
• Controller image with runner volumes needed by newer images: brix4dayz/actions-runner-controller:v0.16.1-prerelease

Example output from self-updating runner:

 No newline at end of file
.path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Starting Runner listener with startup type: service
Started listener process
Started running service
√ Connected to GitHub
2021-01-21 19:24:31Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.276.0 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should back online within 10 seconds.
Runner listener exited with error code 4
Runner listener exit with undefined return code, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process
√ Connected to GitHub
2021-01-21 19:28:51Z: Listening for Jobs
2021-01-21 19:28:54Z: Running job: Tests
2021-01-21 19:29:17Z: Job Tests completed with result: Succeeded
Runner listener exited with error code 0

Opened #269 to get #253 updated and ready to :shipit:

[kaykhancheckpoint]

@hfuss great

The actions-runner-controller image needs to be replaced with the image found in https://github.com/summerwind/actions-runner-controller/releases/latest/download/actions-runner-controller.yaml is that right?

[hfuss]

@kaykhancheckpoint correct!

[jonico]

2.276.0 is not compatible with GitHub Enterprise Server - see https://github.com/actions/runner/pull/896#issuecomment-764016208

I would recommend upgrading to the latest version: https://github.com/actions/runner/pull/931

[int128]

I've updated actions-runner-controller to v0.16.1 and this issue has been fixed. I manually deleted the old runners (in crash loop) and the controller automatically created new runners. Thank you for the great work.

[e36-cicd]

Just a quick note on this - we noticed that the deployment for the controller has an image pull policy of "IfNotPresent", which doesn't update the image if it already exists. This has caused issues for us after merging this fix, and we had to make it pull again for this fix to work (since the old controller still created an emptyDir of '/runner/externals', crashing the update).

Edit: probably need to refresh the docker image for the controller?