Actions-runner-controller: Unable to get dockerVolumeMounts working

[Puneeth-n]

Hi, I am trying to mount a AWS fsx volume to docker:dind image with the new dockerVolumeMounts feature and I am not sure if it is working as expected.

I puller a docker image from inside one runner and ried to do the same from another runner. The expectation was that it would not pull it again in the 2nd runner but it did.

the nodes are in the same AZ as the FSx volume and all the GHA are running on these nodes.

Chart version: 0.10.5
Controller: v0.18.2

Runner config

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: comtravo-github-actions-deployment
  namespace: ${kubernetes_namespace.ci.metadata[0].name}
spec:
  template:
    spec:
      nodeSelector:
        node.k8s.comtravo.com/workergroup-name: github-actions
      image: harbor/cache/comtravo/actions-runner:v2.277.1
      imagePullPolicy: Always
      repository: ${local.actions.git_repository}
      serviceAccountName: ${local.actions.service_account_name}
      securityContext:
        fsGroup: 1447
      dockerVolumeMounts:
      - name: docker-volume
        mountPath: /var/lib/docker
      volumes:
      - name: docker-volume
        persistentVolumeClaim:
          claimName: ${kubernetes_persistent_volume_claim.actions_docker_volume.metadata[0].name}
      resources:
        limits:
          memory: "4Gi"
        requests:
          memory: "256Mi"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: comtravo-github-actions-deployment-autoscaler
  namespace: ${kubernetes_namespace.ci.metadata[0].name}
spec:
  scaleTargetRef:
    name: comtravo-github-actions-deployment
  minReplicas: 4
  maxReplicas: 100
  metrics:
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    repositoryNames:
      - summerwind/actions-runner-controller
  scaleUpTriggers:
  - githubEvent:
      checkRun:
        types: ["created"]
        status: "queued"
    amount: 1
    duration: "1m"

k -n ci describe runner comtravo-github-actions-deployment-8f2gx-5bmhm

Name:         comtravo-github-actions-deployment-8f2gx-5bmhm
Namespace:    ci
Labels:       runner-deployment-name=comtravo-github-actions-deployment
              runner-template-hash=6959d947d9
Annotations:  <none>
API Version:  actions.summerwind.dev/v1alpha1
Kind:         Runner
Metadata:
  Creation Timestamp:  2021-04-13T14:35:10Z
  Finalizers:
    runner.actions.summerwind.dev
  Generate Name:  comtravo-github-actions-deployment-8f2gx-
  Generation:     1
  Managed Fields:
    API Version:  actions.summerwind.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
        f:generateName:
        f:labels:
          .:
          f:runner-deployment-name:
          f:runner-template-hash:
        f:ownerReferences:
      f:spec:
        .:
        f:dockerdContainerResources:
        f:image:
        f:imagePullPolicy:
        f:nodeSelector:
          .:
          f:node.k8s.comtravo.com/workergroup-name:
        f:repository:
        f:resources:
          .:
          f:limits:
            .:
            f:memory:
          f:requests:
            .:
            f:memory:
        f:securityContext:
          .:
          f:fsGroup:
        f:serviceAccountName:
        f:volumes:
      f:status:
        .:
        f:lastRegistrationCheckTime:
        f:phase:
        f:registration:
          .:
          f:expiresAt:
          f:repository:
          f:token:
    Manager:    manager
    Operation:  Update
    Time:       2021-04-13T15:07:16Z
  Owner References:
    API Version:           actions.summerwind.dev/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  RunnerReplicaSet
    Name:                  comtravo-github-actions-deployment-8f2gx
    UID:                   2492f02a-ee74-4777-9df9-9fb07d9b138f
  Resource Version:        69345080
  Self Link:               /apis/actions.summerwind.dev/v1alpha1/namespaces/ci/runners/comtravo-github-actions-deployment-8f2gx-5bmhm
  UID:                     5c7c3de8-15ba-41ee-80ea-a291c0cbada8
Spec:
  Dockerd Container Resources:
  Image:              harbor.infra.comtravo.com/cache/comtravo/actions-runner:v2.277.1
  Image Pull Policy:  Always
  Node Selector:
    node.k8s.comtravo.com/workergroup-name:  github-actions
  Repository:                                comtravo/ct-backend
  Resources:
    Limits:
      Memory:  4Gi
    Requests:
      Memory:  256Mi
  Security Context:
    Fs Group:            1447
  Service Account Name:  actions
  Volumes:
    Name:  docker-volume
    Persistent Volume Claim:
      Claim Name:  actions-docker-volume
Status:
  Last Registration Check Time:  2021-04-13T15:07:16Z
  Phase:                         Running
  Registration:
    Expires At:  2021-04-13T15:34:31Z
    Repository:  comtravo/ct-backend
    Token:       ASS5GHOQCZPOS6FVDRFG2YTAOW5APAVPNFXHG5DBNRWGC5DJN5XF62LEZYANUGERWFUW443UMFWGYYLUNFXW4X3UPFYGLN2JNZ2GKZ3SMF2GS33OJFXHG5DBNRWGC5DJN5XA
Events:
  Type    Reason                    Age                From               Message
  ----    ------                    ----               ----               -------
  Normal  RegistrationTokenUpdated  31m                runner-controller  Successfully update registration token
  Normal  PodCreated                19s (x2 over 31m)  runner-controller  Created pod 'comtravo-github-actions-deployment-8f2gx-5bmhm'

[mumoshu]

@Puneeth-n Hey! I have no experience in using FSx but is PVC backed by FSx is supposed to mount the "same volume" into multiple pods?

Also, sharing /var/lib/docker isn't how it's supposed to work.

The Docker daemon was explicitly designed to have exclusive access to /var/lib/docker. Nothing else should touch, poke, or tickle any of the Docker files hidden there.
https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

[Puneeth-n]

@mumoshu thanks for reminding me about this article. Then, what is the main purpose of dockerVolumeMounts? to have let's says have 50 persistent volumes for 50 runners? Aren't runners short lived? i.e, don't they die and a new one comes up after the action is done?

More importantly, how can i improve the docker pull and docker build performance? i'm trying to build huge docker containers with GHA and not happy with the docker build times

[mumoshu]

@Puneeth-n I think that's mainly for sharing more files between (1) the runner container that runs your job steps and (2)docker containers run within the dind container.

[mumoshu]

That's being said, I would appreciate it very much if you could share more use-cases with dockerVolumeMounts, if found any :smile:

[toast-gear]

It was added here https://github.com/summerwind/actions-runner-controller/pull/439 to resolve https://github.com/summerwind/actions-runner-controller/issues/435. Unforunately it's is another feature that has been added without any documentation by the author so it's not clear on how it is expected to be used. Those issues may help @Puneeth-n with figuring that out.

A PR to add docs would be greatly appreciated by yourself or @asoldino the original author.

[mumoshu]

Thanks @toast-gear! I haven't read @asoldino's original motivation very carefully

Motivation: Having a persistent volume mounted to /var/lib/docker on the dind sidecar can improve the performance of container jobs (layer caching).

When you used e.g. host volumes, this would work when you have only one runner pod per host. In a public cloud like AWS, it would imply that you may prefer combining smaller EC2 instances with the one-pod-per-host model.

But I think it would be preferable to avoid using PV just for making the docker builds faster. Probably you've better experience with e.g. using the nearest container image registry like ECR when you're in AWS with docker's --cache-from option, or even use some kind of object storage like S3 to backup/restore images and layers after/before the docker build.

[asoldino]

Hi @mumoshu, in my scenario we leverage private runners mainly because we need powerful machines to run container jobs based on very large images (~10s of Gb hosted on ACR/ECR) and complex compilation units. In this scenario, single-tenancy of runners is desirable. Initial benchmarks shown an improvement of ~5 minutes per build (on a 10 Gb image).
If using --cache-from means that a pristine runner has to download 10Gb image every time, I don't think it's solving the issue we have.
What I'm presenting is probably a corner case, albeit quite common where I'm coming from. This functionality really helped a lot of people already.

What do you think?

[mumoshu]

@asoldino Hey! Your scenario and the use of the feature seem completely valid. To be clear, you aren't concurrently writing to /var/lib/docker from multiple dockerd processes, right?

[asoldino]

Exactly, that's not supposed to happen

[mumoshu]

@asoldino Thanks for confirming!

@Puneeth-n Hey! I believe I wasn't clear. I only wanted to say that sharing /var/lib/docker from multiple concurrently-running containers is wrong. If you can ensure only one container is writing to /var/lib/docker, it should work fine. That's being said, if you'd like to share /var/lib/docker using a host volume, you will likely to need to set some pod anti affinity and/or big resource req/limits to avoid two or more runners pods concurrently scheduled onto the same host.

[mumoshu]

@Puneeth-n If you still need to use FSx, I think actions-runner-controller needs to be enhanced to enable the user to specify a PVC template rather than a PVC, like a K8s statefulset.

[Puneeth-n]

@mumoshu Thanks for clarifying.

[callum-tait-pbx]

@asoldino could you describe your setup a bit more for me please? You have RunnerDeployments in k8s cluster tied to a single really big k8s node which isn't shared with any other runners. Do your runners just have huge requests/limits?

[asoldino]

Sure:

• I have three node pools in my cluster (one for system pod, one for the platform pods - including the actions-runner-controller, one for the runners)
• The runners node pool has a node autoscaler active (using managed components from AKS)
• I make sure Kubernetes schedules the runners on the dedicated node pool e.g.

#...
kind: RunnerDeployment
spec:
  template:
    spec:
      nodeSelector:
        agentpool: runners
#...

• I make sure to cap the resources of one runner node by e.g. (for 8 cores 32 Gib Ram nodes)

#...
resources:
  limits:
    cpu: "4.0"
    memory: "16Gi"
dockerContainerResources:
  limits:
    cpu: "4.0"
    memory: "16Gi"
#...

• I have a HorizontalRunnerAutoscaler, e.g.

#...
kind: HorizontalRunnerAutoscaler
spec:
  scaleTargetRef:
    name: runners
#...

To recap: Resource request forces Kubernetes to schedule one pod per runner node, when the runner autoscaler kicks in then the node autoscaler provisions the extra nodes required and Kubernetes can eventually run the additional pod.

[Puneeth-n]

@asoldino are you using buildkit or using --cache-from to speedup docker builds?

[asoldino]

@Puneeth-n I'm not actively working on the workflows, I'm "just" a platform provider for my company. I can tell there are a few teams using --cache-from or buildkit, because most of the container jobs are usually normal jobs executed within a container instead of the runner directly. For us, it's much faster and easier to manage.