Kaniko: error removing var/run to make way for new symlink: unlinkat

[hadrien-toma]

Actual behavior
Can not build the image, was working and suddenly was not.

Expected behavior
Be able to build the image.

To Reproduce
File system:

├── context
│   ├── builder.ash
│   └── Dockerfile
└── index.bash

1 directory, 3 files

• Create an issue directory
• Create the following index.bash file under the issue directory:

#!/bin/bash

hereDir=`dirname $0 | while read a; do cd $a && pwd && break; done `

pids=""
rc=0

trap 'sigintTrap' 2

sigintTrap() {
    if [ "${pids}X" != "X" ]; then
        kill -9 ${pids}
    fi
    exit 2
}

minikube delete
minikube start

minikube mount ${hereDir}:/issue &
pids="${pids} $!"

minikube dashboard &
pids="${pids} $!"

cat <<EOF | kubectl apply --force -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko-job
spec:
  activeDeadlineSeconds: 1000
  template:
    metadata:
      name: kaniko-job
    spec:
      containers:
      - args:
        - -c
        - |
          /busybox/sh <<'EOF'

          /kaniko/executor \
            --context "/issue/context" \
            --no-push \
            --dockerfile "/issue/context/Dockerfile" \
            --insecure

          EOF
        command: [ "/busybox/sh" ]
        name: kaniko-job
        image: gcr.io/kaniko-project/executor:debug
        volumeMounts:
        - name: issue
          mountPath: /issue
      restartPolicy: OnFailure
      volumes:
      - name: issue
        hostPath:
          path: /issue
EOF

for pid in ${pids}; do
    wait ${pid} || let "rc=1"
done

if [ "${rc}" == "1" ]; then
    exit 1
fi

• Create a context directory in the issue directory
• Create the following Dockerfile file under the context directory:

FROM alpine:latest

COPY ./builder.ash /builder.ash

RUN chmod +x /builder.ash && /builder.ash ; rm -rf /builder.ash

• Create the following builder.ash file under the context directory:

#!/bin/ash

echo "Building..."

• Give execution permission to the index.bash script: chmod +x issue/index.bash
• Run the index.bash script: ./issue/index.bash
• I get the following in the pod:

INFO[0000] Resolved base name alpine:latest to alpine:latest 
INFO[0000] Resolved base name alpine:latest to alpine:latest 
INFO[0000] Retrieving image manifest alpine:latest      
INFO[0001] Retrieving image manifest alpine:latest      
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine:latest      
INFO[0003] Retrieving image manifest alpine:latest      
INFO[0004] Unpacking rootfs as cmd COPY ./builder.ash /builder.ash requires it. 
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_25_19_26_04.767922246: read-only file system

Additional Information
```

docker version && minikube version && kubectl version && uname -a

Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:29:52 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:28:22 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Linux mewtwo-ubuntu 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
```

[eduscrakozabrus]

I am have this error in gitlab ci

.gitlab-ci.yaml
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --cache=true --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME-$CI_COMMIT_SHA

INFO[0004] Unpacking rootfs as cmd ADD Gemfile* /app/ requires it.
60 error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_26_11_37_35.908221403: read-only file system

[eduscrakozabrus]

solution worked for me
change version of kaniko: gcr.io/kaniko-project/executor:debug-v0.15.0

[hadrien-toma]

Thank you @eduscrakozabrus, worked for me too :pray:. So is this a bug in the newest version?

[followsound]

I've also been having this issue - this solved it

[axyjo]

Looks like a duplicate of #1001

[bozzo]

I had the same issue, i've fixed it by adding --additional-whitelist /var/run/secrets with the gcr.io/kaniko-project/executor:debug image.

[cvgw]

Strong recommendation to everyone; use an explicitly tagged release of the kaniko image. Don't use latest in your critical systems. ~This change has not been released and is only affecting people using latest~ I was incorrect, latest is only built on release. This issue is only affecting people who are using the debug tag which is built on every commit to master. But still, use an explicit version!

[rokerkony]

@cvgw thx for the suggestion, it makes totally sense. It would be great to mention it in the readme as it always advises to use :latest

[tejal29]

closing this as dupe of #1001

[fischerman]

@tejal29 What is the solution to this problem? #1001 seems to be related to CloudBuild (which I have no knowledge about). This issue here seems to be related to Kubernetes. AFAIK whitelisting /var/run exposes the service account token in the image, which is a security thread. Not whitelisting breaks the build.

[bnouvelbmll]

Bug still present in debug-v0.17.0 - reverting to earlier version fix the build but not ideal.

'''
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_10_04_34.525459306: read-only file system
'''

[consideRatio]

I had the same issue, i've fixed it by adding --additional-whitelist /var/run/secrets with the gcr.io/kaniko-project/executor:debug image.

@bozzo, where is this parameter passed? To kaniko executor? It didn't recognize that flag. I'm confused about that.

[tomislater]

I have the same issue (0.17.0):

with --whitelist-var-run=false

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_12_59_06.412495393: read-only file system

0.16.0 works btw.

[bozzo]

@consideRatio It was an option added to the debug image. I don't know if it still present. btw, executor:debug-v0.17.0 worked for me with Gitlab Runners on Kubernetes, by running:

/kaniko/executor --context "$DOCKER_CONTEXT_PATH" --dockerfile "$DOCKER_DOCKERFILE_PATH/Dockerfile" --destination "$DOCKER_IMAGE" --cache --cache-dir="$CACHE_DIR" --verbosity info $DOCKER_BUILD_ARGS

[consideRatio]

@bozzo ah thanks for helping me out! I tried the following that does work for v0.16.0 on my gitlab-runner on k8s, but it fails for me with kaniko v0.17.0 with the error of this issue's title.

  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    # Kaniko reference:
    # https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#additional-flags
    - >
      /kaniko/executor
      --context $CI_PROJECT_DIR/images
      --dockerfile $CI_PROJECT_DIR/images/Dockerfile
      --destination $CI_REGISTRY_IMAGE:cpu-$CI_COMMIT_SHORT_SHA

Ignore /var/run when taking image snapshot. Set it to false to preserve /var/run/* in destination image. (Default true).

I think what happens in my case, is that the COPY statement in my Dockerfile triggers some similar logic that would require "whitelisting" but it does it outside the scope of a image snapshot situation, and hence the --whitelist-var-run flag isn't relevant.

 INFO[0004] Unpacking rootfs as cmd COPY base-notebook-assets/fix-permissions /usr/local/bin/fix-permissions requires it. 
 error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_14_26_54.673390439: read-only file system
 ERROR: Job failed: command terminated with exit code 1

ARG BASE_CONTAINER=ubuntu:bionic-20200112
FROM $BASE_CONTAINER

COPY base-notebook-assets/fix-permissions /usr/local/bin/fix-permissions

[cvgw]

Some info which will hopefully help clear up confusion:

tldr;
<= v0.16.0 = no flag
v0.16.0 <= debug < v0.17.0 = --additional-whitelist=/var/run
v0.17.0 = no flag

prior to v0.16.0 kaniko always whitelisted the path /var/run; meaning that /var/run and any sub paths were ignored.

between v0.16.0 and v0.17.0 introduced the --additional-whitelist flag and removed /var/run from the hardcoded whitelist. The default for --additional-whitelist was an empty list meaning that to get the same behavior as prior to v0.16.0 users needed to supply --additional-whitelist=/var/run. If you were using the debug tag during that time you may have seen the --additional-whitelist flag.

v0.17.0 removed the --additional-whitelist flag and added a new flag --whitelist-var-run. The default of the new flag --whitelist-var-run is true. This means that the default behavior in v0.17.0 (with respect to whitelisting /var/run) is the same as prior to v0.16.0

Updated, thanks @consideRatio for pointing out the mistake

[consideRatio]

Thank you for an excellent summary @cvgw!

For me, the --additional-whitelist within the debug-v0.16.0 image isn't recognized. It also didn't recognize the --whitelist-var-run=true or false option.

Using debug-v0.17.0 I crash directly without specifying any flags with this issue's titles as an error. It also crash like this if i specify --whitelist-var-run=true or false - a flag that it now recognizes.

---

The debug-v0.16.0 run that didn't fail due to this, later failed due to issues that i hope were resolved in debug-v0.17.0.

   ...
   File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
     mkdir(name, mode)
 PermissionError: [Errno 13] Permission denied: '/home/jovyan/.ipyvolume'

[tejal29]

hey folks, Sorry we had a bad release v0.17.0.
I fixed the /var/run issue and released a new v0.17.1 patch release.

[tejal29]

Bug still present in debug-v0.17.0 - reverting to earlier version fix the build but not ideal.

'''
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_10_04_34.525459306: read-only file system
'''

@bnouvelbmll This should now be fixed in v0.17.1 latest release pushed an hour ago.

[tejal29]

@tejal29 What is the solution to this problem? #1001 seems to be related to CloudBuild (which I have no knowledge about). This issue here seems to be related to Kubernetes. AFAIK whitelisting /var/run exposes the service account token in the image, which is a security thread. Not whitelisting breaks the build.

@fischerman Sorry for the inconvenience. We added the flag --whitelist-var-run which whitelisted /var/run by default however there was bug #1025 due to which it got added after extracting base image. We fixed the bug and now your build should work fine.

[tejal29]

I have the same issue (0.17.0):

with --whitelist-var-run=false

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_12_59_06.412495393: read-only file system

0.16.0 works btw.

@tomislater This is should be fixed now in v0.17.1. . We added the flag --whitelist-var-run which whitelisted /var/run by default however there was bug #1025 due to which it got added after extracting base image. We fixed the bug and now your build should work fine.

[consideRatio]

@tejal29 I wanted to confirm that v0.17.1 doesn't have the issue, and if i actively specify --whitelist-var-run=false I can get the same error again - confirming the flag's actively returning to the other behavior.

Thank you for your work on this! :heart: :tada:!

[olivierboudet]

it is fixed for me also with v0.17.1

[hadrien-toma]

Fixed for me too, I close the issue, thanks all :+1: :rocket: !