Actions-runner-controller: Autoscaler do not scale down
runner config:
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: starcoin-runner-deployment
spec:
template:
spec:
nodeSelector:
doks.digitalocean.com/node-pool: ci-pool2
image: starcoin/starcoin-runner:v2.275.1.20210104
repository: starcoinorg/starcoin
resources:
requests:
cpu: "24.0"
memory: "48Gi"
# If set to false, there are no privileged container and you cannot use docker.
dockerEnabled: true
# If set to true, runner pod container only 1 container that's expected to be able to run docker, too.
# image summerwind/actions-runner-dind or custom one should be used with true -value
dockerdWithinRunnerContainer: false
# Valid if dockerdWithinRunnerContainer is not true
dockerdContainerResources:
requests:
cpu: "24.0"
memory: "48Gi"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
name: starcoin-runner-deployment-autoscaler
spec:
scaleTargetRef:
name: starcoin-runner-deployment
minReplicas: 1
maxReplicas: 6
scaleDownDelaySecondsAfterScaleOut: 120
metrics:
- type: TotalNumberOfQueuedAndInProgressWorkflowRuns
repositoryNames:
- starcoinorg/starcoin
- --sync-period=2m
kubectl get HorizontalRunnerAutoscaler
NAME MIN MAX DESIRED
starcoin-runner-deployment-autoscaler 1 6 1
But if the runer autoscale to 6, it does not scale down, even I delete a runner pod manual, it will auto-create a new runner pod.
jolestar
All 18 comments
kubectl get RunnerReplicaSet
NAME DESIRED CURRENT READY
starcoin-runner-deployment-tqxmv 1 6 6
If I delete the RunnerReplicaSet, it will scale down to the desired runner count.
jolestar
on 5 Jan 2021
@jolestar Hey! Thanks for reporting.
If I delete the RunnerReplicaSet, it will scale down to the desired runner count.
Does this mean that the autoscaler did updated the desired count for your RunnerDeployment as expected, but it didn't update RunnerReplicaSet's desired count?
mumoshu
on 6 Jan 2021
@jolestar Hey! Thanks for reporting.
If I delete the RunnerReplicaSet, it will scale down to the desired runner count.
Does this mean that the autoscaler did updated the desired count for your RunnerDeployment as expected, but it didn't update RunnerReplicaSet's desired count?
the RunnerReplicaSet's DESIRED count is right, but the CURRENT & READY is always 6. How can I get more info for diagnostic it?
jolestar
on 6 Jan 2021
@jolestar Thanks! That's helpful.
How can I get more info for diagnostic it?
runnerreplicaset_controller embedded in the actions-runner-controller pod should react as fast as possible to delete redundant runner pods, so that CURRENT would eventually decrease to the count same as DESIRED.
So perhaps your runnerreplicaset controller isn't working somehow? To diagnose, I can suggest you to grep and investigate logs from actions-runner-controller for lines containing runnerreplicaset_controller.go, e.g. kubectl logs -c manager deploy/controller-manager -n actions-runner-system | grep 'runnerreplicaset_controller.go'.
mumoshu
on 6 Jan 2021
There some error:
2021-01-06T00:51:22.882Z ERROR controllers.RunnerReplicaSet Failed to check if runner is busy
{"runner": "default/starcoin-runner-deployment-kgbb6", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/summerwind/actions-runner-controller/controllers.(*RunnerReplicaSetReconciler).Reconcile
/workspace/controllers/runnerreplicaset_controller.go:107
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2021-01-06T00:51:22.883Z ERROR controller-runtime.controller Reconciler error {"controller":
"runnerreplicaset", "request": "default/starcoin-runner-deployment-kgbb6", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
kubectl get runner
NAME ORGANIZATION REPOSITORY LABELS STATUS
starcoin-runner-deployment-kgbb6-78776 starcoinorg/starcoin Running
starcoin-runner-deployment-kgbb6-9w2bc starcoinorg/starcoin Running
starcoin-runner-deployment-kgbb6-cltr7 starcoinorg/starcoin Pending
starcoin-runner-deployment-kgbb6-ffzbz starcoinorg/starcoin Running
starcoin-runner-deployment-kgbb6-h4bh9 starcoinorg/starcoin Running
starcoin-runner-deployment-kgbb6-wr88j starcoinorg/starcoin Running
starcoin-runner-deployment-kgbb6-xfggn starcoinorg/starcoin Running
kubectl --context do get RunnerReplicaSet
NAME DESIRED CURRENT READY
starcoin-runner-deployment-kgbb6 1 5 4
@jolestar Thanks! The error message is definitely misleading - "runner": "default/starcoin-runner-deployment-kgbb6" should be fixed to "runnerreplicaset": "default/starcoin-runner-deployment-kgbb6", and the log message seem to not contain the actual runner name that failed.
Maybe the runner named starcoin-runner-deployment-kgbb6-cltr7 has never been registered to GitHub that prevented the controller to hang?
Would you mind browsing https://github.com/$USER/$REPO/settings/actions if the runners are for your personal project or https://github.com/$ORG/$REPO/settings/actions for organizational runners, and confirm that starcoin-runner-deployment-kgbb6-cltr7 is NOT registered, but all the other runners are registered?
mumoshu
on 6 Jan 2021
What does running kubectl logs starcoin-runner-deployment-kgbb6-cltr7 show you? Any error while the runner agent within the runner pod trying to register itself to GitHub?
mumoshu
on 6 Jan 2021
The starcoin-runner-deployment-kgbb6-cltr7 is pending, so no log output. I try to delete a runner, cltr7 is running, and output error "token expired".
Http response code: Unauthorized from 'POST https://api.github.com/actions/runner-registration'
{"message":"Token expired.","documentation_url":"https://docs.github.com/rest"}
Response status code does not indicate success: 401 (Unauthorized).
settings/actions page, all runner is offline:
jolestar
on 6 Jan 2021
I try to delete the RunnerReplicaSet
kubectl delete RunnerReplicaSet starcoin-runner-deployment-kgbb6
and all runners are rebuilt.
kubectl --context do get runner
NAME ORGANIZATION REPOSITORY LABELS STATUS
starcoin-runner-deployment-vt5bk-nhn4r starcoinorg/starcoin Running
starcoin-runner-deployment-vt5bk-qsg85 starcoinorg/starcoin Running
starcoin-runner-deployment-vt5bk-sk8n7 starcoinorg/starcoin Running
jolestar
on 6 Jan 2021
@jolestar Thanks for your help.
I might be a bit confused but the issue can be that our "runner controller" has a possible bug that results in leaving runner pods with expired registration token forever, which prevents the corresponding runner replicaset to work(?)
It's a bit involved, but it SHOULD NOT happen as our runner controller periodically (sync period) checks the registration token:
and once expired it replaces the token and the pod:
Perhaps it isn't working as expected or there's edge-case(s).
Out of curiosity, does creating new RunnerDeployment and decreasing the desired count immediately work? In other words, does your issue happen only when you scale-down after an hour or so?
mumoshu
on 6 Jan 2021
It's strange that it's now scale down. Let me watch it for a while longer.
kubectl --context do get RunnerReplicaSet
NAME DESIRED CURRENT READY
starcoin-runner-deployment-vt5bk 1 1 1
I remove the old offline runner qfql6 from the GitHub actions page.
jolestar
on 6 Jan 2021
Error again:
kubectl --namespace actions-runner-system logs controller-manager-5879594668-wp7mn manager
2021-01-06T04:21:19.494Z ERROR controllers.RunnerReplicaSet Failed to check if runner is busy {"runner": "default/starcoin-runner-deployment-vt5bk", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/summerwind/actions-runner-controller/controllers.(*RunnerReplicaSetReconciler).Reconcile
/workspace/controllers/runnerreplicaset_controller.go:107
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2021-01-06T04:21:19.494Z ERROR controller-runtime.controller Reconciler error {"controller": "runnerreplicaset", "request": "default/starcoin-runner-deployment-vt5bk", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2021-01-06T04:21:19.693Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "runner", "request": "default/starcoin-runner-deployment-vt5bk-2tvk8"}
kubectl logs starcoin-runner-deployment-vt5bk-2tvk8 runner
Http response code: Unauthorized from 'POST https://api.github.com/actions/runner-registration'
{"message":"Token expired.","documentation_url":"https://docs.github.com/rest"}
Response status code does not indicate success: 401 (Unauthorized).
@jolestar Thanks. I think we're close. Would you mind sharing me the result of kubectl get po -o yaml starcoin-runner-deployment-vt5bk-2tvk8(or those of any runner pods that failed like it), with the output of date command on your machine or controller pod? (
mumoshu
on 6 Jan 2021
There another error:
Starting Runner listener with startup type: service
Started listener process
Started running service
An error occurred: Not configured
Runner listener exited with error code 2
Runner listener exit with retryable error, re-launch runner in 5 seconds.
kubectl get po -o yaml starcoin-runner-deployment-68gdh-wwgzm
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-01-06T18:05:55Z"
labels:
pod-template-hash: 749fd4569f
runner-template-hash: 6d59d7cd4b
@jolestar Hey! Are you still using actions-runner-controller?
FYI, I've recently summarized how our controller can get stuck due to runners being unable to be registered for various reasons.
We're far from "fixing" all the root causes because they vary a lot but the universal fix can be #297 which is I'm working on currently.
mumoshu
on 9 Feb 2021
I try the actions-runner-controller of v0.18.2, this bug is resolved.
jolestar
on 8 Apr 2021
@jolestar Thanks for reporting! Glad to hear it worked.
mumoshu
on 8 Apr 2021
Related issues
hdoinfodation
路
4Comments
jlsan92
路
4Comments
stackdumper
路
6Comments
callum-tait-pbx
路
9Comments
jmmaloney4
路
6Comments
Most helpful comment
@jolestar Hey! Are you still using actions-runner-controller?
FYI, I've recently summarized how our controller can get stuck due to runners being unable to be registered for various reasons.
We're far from "fixing" all the root causes because they vary a lot but the universal fix can be #297 which is I'm working on currently.