Actions-runner-controller: Docker daemon Permission Denied when using Service containers in v0.9.4

[igorbrigadir]

Something changed between v0.9.3 and v0.9.4 - most likely #126 - that broke "Service containers" like these: https://github.com/actions/example-services/blob/main/.github/workflows/redis-service.yml

In github Actions logs when "Initializing Containers" i get:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

This happens when setting dockerdWithinRunnerContainer: true (or false).

In the commit before #126 it was working fine. Permission denied for docker usually means adding the runner user to docker group but this is already in the runner image i think? Not really sure where to fix this.

[igorbrigadir]

My mistake - it seems to be working. I was using my own custom runner image, and needed to make a bunch of changes to my Dockerfile, to make it more like https://github.com/summerwind/actions-runner-controller/blob/master/runner/Dockerfile.dindrunner as opposed to https://github.com/summerwind/actions-runner-controller/blob/master/runner/Dockerfile

[dylan-cyndx]

Hi @igorbrigadir - wanted to add that I'm experiencing the same exact issue on my end with v0.9.4. After upgrading, I've made no changes to dockerfiles or the actions workflow, and I am not using a custom runner image.

Workflows that succeed on v0.9.3 seem to fail on v0.9.4. Specifically, after running a variant of docker build -t my_svc:latest, I see the following printed in the actions log:

free(): invalid pointer
SIGABRT: abort
PC=0x7f0623b9cf47 m=0 sigcode=[...]
signal arrived during cgo execution
[...]
goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4afd50, 0xc42004bcc0, 0xc42004bce8)
    /usr/lib/go-1.8/src/runtime/cgocall.go:131 +0xe2 fp=0xc42004bc90 sp=0xc42004bc50
github.com/docker/docker-credential-helpers/secretservice._Cfunc_free(0xc73da0)
[...]
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=***&cachefrom=***&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=***&memory=0&memswap=0&networkmode=default&rm=1&session=***&shmsize=0&t=my_svc%3Alatest&target=&ulimits=null&version=1: dial unix /var/run/docker.sock: connect: permission denied

Reverting the installed version to v0.9.3 and rerunning the same failed workflow results in a success, so I'm also reasonably certain it has to do with the latest release.

If you can provide any troubleshooting steps as to how you resolved this issue (given we are not using a custom image), that would be greatly appreciated! Otherwise, I'm also happy to open a separate Github issue.

[igorbrigadir]

Does setting dockerdWithinRunnerContainer: false do anything for you? (In the yaml that defines your runners)

It could be an issue with the docker runner image?

I know you don't use a custom one but maybe it's worth experimenting with making one based on https://github.com/summerwind/actions-runner-controller/blob/master/runner/Dockerfile.dindrunner ? Those are the only things that come to mind to check first.

[mumoshu]

@igorbrigadir @dylan-cyndx Thanks for bringing this up! I've checked #126 and realized that this may be due to that we accidentally removed the RunAsGroup=0 setting from the runnercontainer. https://github.com/summerwind/actions-runner-controller/pull/126/files#diff-f4c239205beda2ef5d9d041809cd93cb32958b6b658fb0b2331c85b853777465L344

cc/ @Hi-Fi

[Hi-Fi]

I'd say it's more an issue with runner image that "locks" to single group (unlike summerwind/actions-runner-dind image)

Group removal was intentional. Reason easiest to explain with Docker:

docker container run --rm -it --entrypoint /usr/bin/id summerwind/actions-runner 
uid=1000(runner) gid=1000(runner) groups=1000(runner)

When id was set, results was:

docker container run --rm -it --entrypoint /usr/bin/id -u 1000:0 summerwind/actions-runner 
uid=1000(runner) gid=0(root) groups=0(root)

This worked fine with the separate dind container, but not with summerwind/actions-runner-dind where we need both sudo and docker groups (when gid defined, that's only one).

docker container run --rm -it --entrypoint /usr/bin/id summerwind/actions-runner-dind 
uid=1000(runner) gid=1000(runner) groups=1000(runner),27(sudo),1001(docker)

and

docker container run --rm -it --entrypoint /usr/bin/id -u 1000:0 summerwind/actions-runner-dind
uid=1000(runner) gid=0(root) groups=0(root)

Fix would then probably be to add more logic to put that gid 0 if running separate DinD container in pod.

Note that setting dockerdWithinRunnerContainer doesn't do much if no custom image (summerwind/actions-runner-dind) defined, too.

[mumoshu]

@Hi-Fi Thanks for clarifying!

Fix would then probably be to add more logic to put that gid 0 if running separate DinD container in pod.

Or just add the runner user to the docker group in our non-dind dockerfile, as you've done in the dind dockerfile?

[Hi-Fi]

@Hi-Fi Thanks for clarifying!

Fix would then probably be to add more logic to put that gid 0 if running separate DinD container in pod.

Or just add the runner user to the docker group in our non-dind dockerfile, as you've done in the dind dockerfile?

If adding user to docker -group, then the user directive should also be set just USER runner. As if gid is defined, that overrides other group memberships (which is why dind Dockerfile has only user defined, not group).

[mumoshu]

@Hi-Fi Gotcha- I'll try that. Thanks ☺️

[igorbrigadir]

Ok, i believe there's another separate error that's also related to this:

When i create a runner with this for example (using my own image)

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
spec:
  replicas: 6
  template:
    spec:
      organization: recsyslabs
      image: actions-runner:latest
      imagePullPolicy: Always
      imagePullSecrets:
        - name: regcred
      hostNetwork: true
      dockerdWithinRunnerContainer: true
      env: []

I get pods with 1 container (as expected).

But when a job is finished and the pod is reset / restarted - it suddenly comes up with 2 containers (runner and dind) inside the runner pod, as if dockerdWithinRunnerContainer was set to false.

[igorbrigadir]

Wait that last one might be my mistake again - i had fluxcd running and applying an out of date yaml that was missing dockerdWithinRunnerContainer: true !

[mumoshu]

@igorbrigadir Thanks for reporting back! Can I take it as you've successfully configured your runners with dockerdWithinRunnerContainer: true?

If you need another setup with dockerdWithinRunnerContainer: false(which is the default setting), please try v0.9.5 that includes #142 😃

[igorbrigadir]

Yep, i just tried 0.9.5 and it's all good. (I had a custom image, so i had to update to remove the user group in the same way too)