Actions-runner-controller: Passing IRSA to docker builds

[Puneeth-n]

I am trying to build docker images and in some docker images, I have some aws cli commands. I was wondering if there was any way of passing the IRSA to the docker daemon so that it could execute aws cli commands during docker build.

Please find below a sample Dockerfile

FROM python:3.8
...
RUN python -c "import nltk; nltk.download('punkt')"

ARG CI
ENV AWS_DEFAULT_REGION=eu-west-1
ENV AWS_REGION=eu-west-1
RUN if [ "true" = "$CI" ] ; then make s3_sync ; else echo "Not syncing with s3 because running outside CI" ; fi
COPY ./js/lib/passport /usr/src/app/js/lib/passport
...

[callum-tait-pbx]

You should just be able to map them into the container like this:

name: IRSA in container

on:
  workflow_dispatch:

jobs:
  sync:
    runs-on: [ 'self-hosted', 'Linux' ]
    container: 
      image: amazon/aws-cli
      env:
        AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
        AWS_ROLE_ARN: arn:aws:iam::%AWS_ACCOUNT_ID%:role/%YOUR_IRSA_ROLE%
      volumes:
        - /var/run/secrets/eks.amazonaws.com/serviceaccount/token:/var/run/secrets/eks.amazonaws.com/serviceaccount/token
    steps:
    - uses: actions/checkout@v2
    - name: aws
      run: |
        aws s3 ls

[Puneeth-n]

both the runner and dind have proper IRSA setup. my runner has awscli installed and manually running s3 sync works. when building the docker images, it does not seem to work. Since the docker container also has IRSA, shouldn't the dockerd running inside the docker container be able to execute aws commands?

PFB the config

Name:         foo-github-actions-deployment-jxr4z-zx2vl
Namespace:    ci
Priority:     0
Node:         ip-10-31-2-11.eu-west-1.compute.internal/10.31.2.11
Start Time:   Thu, 01 Apr 2021 11:53:01 +0200
Labels:       pod-template-hash=749fd4569f
              runner-deployment-name=foo-github-actions-deployment
              runner-template-hash=567488d8cf
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.31.2.233
IPs:
  IP:           10.31.2.233
Controlled By:  Runner/foo-github-actions-deployment-jxr4z-zx2vl
Containers:
  runner:
    Container ID:   docker://a5bb924cbb3655b944f6a25c9446d93e6524d6783d5cb88e0ad0990cc94ff0e3
    Image:          harbor.infra.foo.com/cache/foo/actions-runner:v2.277.1
    Image ID:       docker-pullable://harbor.infra.foo.com/cache/foo/actions-runner@sha256:bb40b3bf110a58d07236132348b093ec31aa5287b3eb852b086985423ccf1641
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 01 Apr 2021 11:53:04 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:     1m
      memory:  256Mi
    Environment:
      RUNNER_NAME:                  foo-github-actions-deployment-jxr4z-zx2vl
      RUNNER_ORG:
      RUNNER_REPO:                  foo/ct-backend
      RUNNER_ENTERPRISE:
      RUNNER_LABELS:
      RUNNER_GROUP:
      RUNNER_TOKEN:                 ASS5GHO6GCYULSHSKQTUYG3AMWRIJAVPNFXHG5DBNRWGC5DJN5XF62LEZYANUGERWFUW443UMFWGYYLUNFXW4X3UPFYGLN2JNZ2GKZ3SMF2GS33OJFXHG5DBNRWGC5DJN5XA
      DOCKERD_IN_RUNNER:            false
      GITHUB_URL:                   https://github.com/
      RUNNER_WORKDIR:               /runner/_work
      DOCKER_HOST:                  tcp://localhost:2376
      DOCKER_TLS_VERIFY:            1
      DOCKER_CERT_PATH:             /certs/client
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::01234567890:role/infra/actions
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /certs/client from certs-client (ro)
      /runner from runner (rw)
      /runner/_work from work (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from actions-token-lhns5 (ro)
  docker:
    Container ID:   docker://d30d51cd4dc8cf09204345f05348cfd5e8b8594c5675d49f8dfebde7fc58825c
    Image:          harbor.infra.foo.com/cache/library/docker:dind
    Image ID:       docker-pullable://harbor.infra.foo.com/cache/library/docker@sha256:5e1ed8b6e6db0e8507f40633f1f651084fd1b5445d83c37a66919c2a52d33fbf
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 01 Apr 2021 11:53:08 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      DOCKER_TLS_CERTDIR:           /certs
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::01234567890:role/infra/actions
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /certs/client from certs-client (rw)
      /runner from runner (rw)
      /runner/_work from work (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from actions-token-lhns5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  runner:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  work:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  certs-client:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  actions-token-lhns5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  actions-token-lhns5
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node.k8s.foo.com/workergroup-name=spot
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                               Message
  ----    ------     ----  ----                                               -------
  Normal  Scheduled  13m   default-scheduler                                  Successfully assigned ci/foo-github-actions-deployment-jxr4z-zx2vl to ip-10-31-2-11.eu-west-1.compute.internal
  Normal  Pulling    13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Pulling image "harbor.infra.foo.com/cache/foo/actions-runner:v2.277.1"
  Normal  Pulled     13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Successfully pulled image "harbor.infra.foo.com/cache/foo/actions-runner:v2.277.1"
  Normal  Created    13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Created container runner
  Normal  Started    13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Started container runner
  Normal  Pulled     13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Container image "harbor.infra.foo.com/cache/library/docker:dind" already present on machine
  Normal  Created    13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Created container docker
  Normal  Started    13m   kubelet, ip-10-31-2-11.eu-west-1.compute.internal  Started container docker

[callum-tait-pbx]

The credentials side of things are handled by the client side of docker, so where your docker CLI is running, the daemon doesn't get involved with the auth side of things. I've never tried to run AWS commands as part of a build step, I'm not sure how that would work tbh with the layers. You could try exporting your aws auth in your chosen way into the docker build Dockerfile and see if that works. If not I would suggest you do all of the AWS credential work on your runner and output the content to your build context location. Then have your Dockerfile COPY the content into the container, this would also massively help with keeping your image small.