Kaniko: error building image: error building stage GCC

[denitol]

I try build gcc image with kaniko
and get error

INFO[0003] Unpacking rootfs as cmd RUN cmake -DCMAKE_BUILD_TYPE=Release requires it. 
error building image: error building stage: removing whiteout usr/bin/cmake/.wh..wh..opq: lstat /usr/bin/cmake/.wh..opq: operation not permitted

kaniko image : gcr.io/kaniko-project/executor:debug

stack:

k8s, gitlab, gitlab-runner(into k8s)

[donmccasland]

Are you using cacheing? Might be related to #742

[tejal29]

@denitol Can you provide more information on your docker file?

[z0mb1ek]

same problem

INFO[0030] RUN apt-get update && apt-get -y install wget build-essential git libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev
INFO[0030] Found cached layer, extracting to filesystem
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

[z0mb1ek]

my docker image

FROM nginx:1.17.8

COPY devops/nginx/ssl/certbundle.pem /etc/ssl/certbundle.pem

COPY devops/nginx/ssl/1.key /etc/ssl/1.key

COPY devops/nginx/nginx.conf /etc/nginx/nginx.conf

COPY devops/nginx/default.conf /etc/nginx/conf.d/default.conf

RUN apt-get update && apt-get -y install wget build-essential git libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev

RUN git clone https://github.com/kaltura/nginx-vod-module.git

RUN wget 'https://nginx.org/download/nginx-1.17.8.tar.gz' && tar xvzf nginx-1.17.8.tar.gz

WORKDIR nginx-1.17.8

RUN ./configure --add-dynamic-module=/nginx-vod-module \
    --with-file-aio \
    --with-threads \
    --with-cc-opt="-O3"

RUN ls -la /nginx-vod-module

[ribx]

We are also on gitlab/gitlab-runner + k8s.

I can reproduce this ~sometimes~ always (if cache is present) with this simple Dockerfile:

FROM nginx:1.17

RUN echo "server {listen 80; root /html/; \
          location /api/ {return 500;} \
          location /static/ {} \
          location / {try_files \$uri /index.html;}}" > /etc/nginx/conf.d/default.conf

COPY ./build /html

with this command:

/kaniko/executor
      --context "./"
      --dockerfile Dockerfile
      --destination "$REGISTRY/$GROUP/$PROJECT/frontend:$TAG" # maybe this is important?
      --cache=true

inside the image "gcr.io/kaniko-project/executor:debug" (sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3)
and also "gcr.io/kaniko-project/executor:debug-v0.16.0"

When the cache is cleared, the build works fine.

[Flauschbaellchen]

We're running into the same bug which is reproducable on every run as long as the cache exists.
Currently, we disabled the cache to be able to build our images again.

Can I help, e.g. do you need another Dockerfile example or debug output?
We're using gitlab-runner 12.7.1 on Debian 9 with Docker version 17.05.0-ce (build 89658be) and gcr.io/kaniko-project/executor:debug (025bd79d3e06)

[cvgw]

We're running into the same bug which is reproducable on every run as long as the cache exists.
Currently, we disabled the cache to be able to build our images again.

Can I help, e.g. do you need another Dockerfile example or debug output?
We're using gitlab-runner 12.7.1 on Debian 9 with Docker version 17.05.0-ce (build 89658be) and gcr.io/kaniko-project/executor:debug (025bd79d3e06)

You probably need to clear your cache so that it doesn't keep re-using the bad cached layer.

[Flauschbaellchen]

@cvgw Well, that's not a solution as within the next run a newly created cache would be saved which is also faulty.

[cvgw]

@cvgw Well, that's not a solution as within the next run a newly created cache would be saved which is also faulty.

Apologies if I'm misunderstanding you; my suggestion was to clear the cache when going from a known bad version of kaniko to a good version

E.G

image was previously built with kaniko v0.17.0 and cached
switch to kaniko v0.17.1
clear cache
build image with v0.17.1
at that point the cache is good to use for v0.17.1

[z0mb1ek]

Apologies if I'm misunderstanding you; my suggestion was to clear the cache when going from a known bad version of kaniko to a good version

E.G

image was previously built with kaniko v0.17.0 and cached
switch to kaniko v0.17.1
clear cache
build image with v0.17.1
at that point the cache is good to use for v0.17.1

try this, not working too

[Flauschbaellchen]

With kaniko 0.24.0 this problem still persists.

First run without cache is successful, but on the second run, with the cached layer, it fails:

INFO[0013] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: error removing /usr/local/lib/python2.7/site-packages/.wh.setuptools-41.4.0.dist-info to make way for new file.: fstatat /usr/local/lib/python2.7/site-packages/.wh.setuptools-41.4.0.dist-info: operation not permitted

[Laski]

I was thinking: would it be possible to write (perhaps outside kaniko) some script that retires the job without cache if the first one fails? Probably using the --cleanup flag

[Narretz]

Can confirm that this is still an issue with Kaniko 1.2.0 (and Gitlab Runner 12.8.0 + kubernetes executor)

[gajus]

Is there a workaround?

[Keramblock]

Same

[deepakkt-a]

Can confirm that this is still an issue with Kaniko 1.2.0 (and Gitlab Runner 12.8.0 + kubernetes executor)

And with v1.3