Actions-runner-controller: Failed to pull image "docker:dind". You have reached your pull rate limit.

[stackdumper]

Recently, Docker Hub introduced rate limits. As per November 18, it's limited to 250 unauthenticated pulls per 6 hours.

Is there a way to configure actions-runner-controller to pull docker:dind from a custom private registry like ECR or GCR?

Events:
  Type     Reason     Age                From                                  Message
  ----     ------     ----               ----                                  -------
  Normal   Scheduled  <unknown>          default-scheduler             Successfully assigned actions-runner-system/actions-runner-apiv2-actions-runner-deployment-wht4b-cjzdb to ip-xxx.ec2.internal
  Normal   Pulling    118s               kubelet, ip-xxx.ec2.internal  Pulling image "xxx.dkr.ecr.us-east-1.amazonaws.com/actions-runner:local.15"
  Normal   Pulled     81s                kubelet, ip-xxx.ec2.internal  Successfully pulled image "xxx.dkr.ecr.us-east-1.amazonaws.com/actions-runner:local.15"
  Normal   Created    77s                kubelet, ip-xxx.ec2.internal  Created container runner
  Normal   Started    77s                kubelet, ip-xxx.ec2.internal  Started container runner
  Normal   Pulling    37s (x3 over 77s)  kubelet, ip-xxx.ec2.internal  Pulling image "docker:dind"
  Warning  Failed     36s (x3 over 76s)  kubelet, ip-xxx.ec2.internal  Failed to pull image "docker:dind": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     36s (x3 over 76s)  kubelet, ip-xxx.ec2.internal  Error: ErrImagePull
  Normal   BackOff    12s (x5 over 76s)  kubelet, ip-xxx.ec2.internal  Back-off pulling image "docker:dind"
  Warning  Failed     12s (x5 over 76s)  kubelet, ip-xxx.ec2.internal  Error: ImagePullBackOff

[igorbrigadir]

It should work with something like this in the runner spec:

    spec:
      image: xxx.dkr.ecr.us-east-1.amazonaws.com/actions-runner:local.15
      imagePullPolicy: Always
      imagePullSecrets:
        - name: mydocker

you'll need to create the mydocker secret in the same namespace as your runners with

kubectl create secret docker-registry mydocker --docker-server=xxx.dkr.ecr.us-east-1.amazonaws.com --docker-username=something --docker-password=something --namespace mynamespace

[stackdumper]

Thanks for response. I'm already using this approach:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runner-deployment
spec:
  template:
    spec:
      image: xxx.dkr.ecr.us-east-1.amazonaws.com/actions-runner:local.15

However, when a new job is started, runner still tries to pull docker:dind from Docker Hub.

Events:
  Type     Reason     Age                From                                  Message
  ----     ------     ----               ----                                  -------
  (skipped)
  Normal   Started    77s                kubelet, ip-xxx.ec2.internal  Started container runner
  Normal   Pulling    37s (x3 over 77s)  kubelet, ip-xxx.ec2.internal  Pulling image "docker:dind"
  Warning  Failed     36s (x3 over 76s)  kubelet, ip-xxx.ec2.internal  Failed to pull image "docker:dind": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     36s (x3 over 76s)  kubelet, ip-xxx.ec2.internal  Error: ErrImagePull
  Normal   BackOff    12s (x5 over 76s)  kubelet, ip-xxx.ec2.internal  Back-off pulling image "docker:dind"
  Warning  Failed     12s (x5 over 76s)  kubelet, ip-xxx.ec2.internal  Error: ImagePullBackOff

[igorbrigadir]

Oh, that's weird - is this with specifying your own imagePullSecrets? it specifies the server, --docker-server=xxx.dkr.ecr.us-east-1.amazonaws.com so maybe that's where it's going wrong?

[stackdumper]

Actually I don't specify imagePullSecrets, instead granting permissions to pull images from a private ECR with EC2 instance IAM profiles. And everything works fine until the cluster's pull rate limit for Docker Hub is exceeded.

I will try to configure Kubernetes to pull all images from a private ECR by using --docker-server, thank you.

[mumoshu]

@stackdumper @igorbrigadir Thanks for bringing this up. Ah, so I believe you need to set --docker-image xxx.dkr.ecr.us-east-1.amazonaws.com/docker:dind on the actions-runner-controller(not runner).

https://github.com/summerwind/actions-runner-controller/blob/3f335ca628807682f06495097d8d2555415ea552/main.go#L77

If you're using kustomize for deployment, it should be added under args at https://github.com/summerwind/actions-runner-controller/blob/3f335ca628807682f06495097d8d2555415ea552/config/default/manager_auth_proxy_patch.yaml#L23-L25

[stackdumper]

@mumoshu Adding --docker-image to args of the controller-manager deployment did the trick . Thank you!