Actions-runner-controller: Docker build failing with "failed to adjust OOM score for shim"

[sripberger]

My company uses summerwind/actions-runner-controller to handle self-hosted runners for our deployment workflow. Our main reason for doing this is that our database lives in a private-IP Cloud SQL instance, and we want knex database migrations to run as part of the workflow.

It's working fine for the most part, but suddenly (as of yesterday) we've started to see the docker build part of our workflow consistently fail with the following output:

Sending build context to Docker daemon  266.8MB

Step 1/14 : FROM node:12 as build
12: Pulling from library/node
22f9b9782fc3: Pulling fs layer
2c75c9e56a8a: Pulling fs layer
b31d19b99daa: Pulling fs layer
e000cbf49b08: Pulling fs layer
892def1a6f88: Pulling fs layer
0581e893c9eb: Pulling fs layer
92b6633afaae: Pulling fs layer
dc11bf172033: Pulling fs layer
f09b45a84f38: Pulling fs layer
e000cbf49b08: Waiting
892def1a6f88: Waiting
0581e893c9eb: Waiting
92b6633afaae: Waiting
dc11bf172033: Waiting
f09b45a84f38: Waiting
b31d19b99daa: Verifying Checksum
b31d19b99daa: Download complete
2c75c9e56a8a: Verifying Checksum
2c75c9e56a8a: Download complete
22f9b9782fc3: Verifying Checksum
22f9b9782fc3: Download complete
0581e893c9eb: Verifying Checksum
0581e893c9eb: Download complete
e000cbf49b08: Verifying Checksum
e000cbf49b08: Download complete
dc11bf172033: Verifying Checksum
dc11bf172033: Download complete
92b6633afaae: Verifying Checksum
92b6633afaae: Download complete
f09b45a84f38: Verifying Checksum
f09b45a84f38: Download complete
892def1a6f88: Verifying Checksum
892def1a6f88: Download complete
22f9b9782fc3: Pull complete
2c75c9e56a8a: Pull complete
b31d19b99daa: Pull complete
e000cbf49b08: Pull complete
892def1a6f88: Pull complete
0581e893c9eb: Pull complete
92b6633afaae: Pull complete
dc11bf172033: Pull complete
f09b45a84f38: Pull complete
Digest: sha256:05b0f50fc60d07a18cbe399d3c9d6942b2bd1e0202070efec217c541130dfe52
Status: Downloaded newer image for node:12
 ---> 18a989c5528a
Step 2/14 : ARG NPM_TOKEN
 ---> Running in b1c37961bf43
Removing intermediate container b1c37961bf43
 ---> e25c304f4026
Step 3/14 : WORKDIR /build
 ---> Running in 376b38e3b6d7
Removing intermediate container 376b38e3b6d7
 ---> 2dec829e69d6
Step 4/14 : COPY dist package.json package-lock.json ./
 ---> a442117e1482
Step 5/14 : RUN chown node:node /build
 ---> Running in f9168288134d
io.containerd.runc.v2: failed to adjust OOM score for shim: set shim OOM score: write /proc/299/oom_score_adj: invalid argument
: exit status 1: unknown
Error: Process completed with exit code 1.

We have two runners set up-- one deploys to a development environment, and the other deploys to production. The production runner continues to work, even though it was installed and set up in exactly the same way. The only differences between the two as are in GCP project names and whether the run on push to master or on release creation.

I've tried deleting the entire actions-runner-system namespace and re-installing everything and the problem won't go away. I'm sort of at my wits end with this and would appreciate some assistance. :)

Installed with:

kubectl apply -f https://github.com/summerwind/actions-runner-controller/releases/v0.13.1/download/actions-runner-controller.yaml

Runner manifest:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: development-runner
spec:
  repository: Batterii/api
  image: summerwind/actions-runner:v2.274.2
  labels:
    - vurvey-development-cluster
  env: []
  volumes:
    - name: secret-volume
      secret:
        secretName: deployer-api-service-account-key
  sidecarContainers:
    - name: cloud-sql-proxy
      image: gcr.io/cloudsql-docker/gce-proxy:1.17
      command:
        - "/cloud_sql_proxy"
        - "-ip_address_types=PRIVATE"
        - "-instances=vurvey-development:us-central1:vurvey-development-postgres=tcp:5432"
        - "-credential_file=/secrets/service-account-key.json"
      securityContext:
        runAsNonRoot: true
      volumeMounts:
        - name: secret-volume
          mountPath: /secrets/
          readOnly: true

Deployment workflow:

name: Deploy to api.vurvey.dev
on:
  push:
    branches:
    - master
jobs:
  deploy:
    runs-on: [self-hosted, vurvey-development-cluster]
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Set up node and npm
      uses: actions/setup-node@v1
      with:
        node-version: 12.x
        registry-url: https://registry.npmjs.org

    - name: Connect npm cache
      uses: actions/cache@v2
      env:
        cache-name: npm-cache
      with:
        # npm cache files are stored in `~/.npm` on Linux/macOS
        path: ~/.npm
        key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
        restore-keys: |
          ${{ runner.os }}-build-${{ env.cache-name }}-
          ${{ runner.os }}-build-
          ${{ runner.os }}-

    # Skipping post-install scripts so they can't steal NODE_AUTH_TOKEN.
    - name: Install dependencies
      run: npm ci --ignore-scripts
      env:
        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

    # Running post-install scripts, now without the NODE_AUTH_TOKEN set.
    - name: Run post-install scripts
      run: npm rebuild && npm run prepare --if-present

    - name: Build the dist directory
      run: npm run build

    - name: Build the docker image
      run: |-
        docker build . \
          --tag gcr.io/vurvey-development/batterii-api:$GITHUB_SHA \
          --build-arg NPM_TOKEN=${{ secrets.NPM_TOKEN }}

    - name: Set up gcloud
      uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        version: '290.0.1'
        project_id: vurvey-development
        service_account_key: ${{ secrets.DEV_DEPLOYER_KEY }}
        export_default_credentials: true

    - name: Configure glcoud as docker credential helper
      run: gcloud auth configure-docker

    - name: Push the docker image to the project's container registry
      run: docker push gcr.io/vurvey-development/batterii-api:$GITHUB_SHA

    - name: Set up kubectl
      uses: azure/setup-kubectl@v1
      with:
        version: 'v1.19.3'

    - name: Set up kustomize
      uses: imranismail/setup-kustomize@v1
      with:
        version: '3.1.0'

    - name: Get GKE credentials
      run: |-
        gcloud container clusters get-credentials "vurvey-development-cluster" \
          --zone "us-central1-c"

    - name: Run any new database migrations
      run: npm run migrate
      env:
        DB_USER: postgres
        DB_PASS: ${{ secrets.DEV_DB_PASSWORD }}
        DB_NAME: vurvey

    - name: Rollout GKE changes
      working-directory: gke/development
      run: |-
        kustomize edit set image API_IMAGE=gcr.io/vurvey-development/batterii-api:$GITHUB_SHA
        kustomize build . | kubectl apply -f -

    - name: Wait on rollout to finish in deployments.
      run: |-
        kubectl rollout status deployment api-deployment
        kubectl rollout status deployment mailer-deployment
        kubectl rollout status deployment image-transcoder-deployment
        kubectl rollout status deployment video-transcoder-deployment

Dockerfile:

FROM node:12 as build
ARG NPM_TOKEN
WORKDIR /build
COPY dist package.json package-lock.json ./
RUN chown node:node /build
USER node
RUN echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > .npmrc && \
    npm ci --prod && \
    rm -f .npmrc

FROM node:12
RUN apt-get update && apt-get install -y ffmpeg
WORKDIR /usr/local/src/vurvey-api/
COPY --from=build --chown=root:root /build ./
USER node
ENTRYPOINT ["node"]
CMD ["bin/batterii-api.js"]

[gregorygtseng]

For your runner, try adding cpu/memory requests/limits

[nkaplatt]

Just for anybody who comes across this issue.
Setting a resource limit worked. I'll be honest that I'm not 100% sure why, as by not having a limit I was under the impression that all resources could be consumed, but anyway that fixed the issue.

[sripberger]

Happy to hear of a fix, though I actually ended up replacing these runners with something more simple set up directly in a GCE machine. This works fine for our use case (for now) since we are only doing builds a couple times a day at most.

If and when we grow and need to scale more easily, this will definitely be the first solution I come back to, though. Thanks for the help! :+1:

[jwalters-gpsw]

For your runner, try adding cpu/memory requests/limits

How do you specify that in the Runner spec above? Thanks.