Kaniko: error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

[purificant]

Actual behavior
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

Expected behavior
Build finishes without error.

To Reproduce
Steps to reproduce the behavior:

1. Enable Kaniko and set version:

gcloud config set builds/use_kaniko True
gcloud config set builds/kaniko_image gcr.io/kaniko-project/executor:v0.17.1

1. Start build:

gcloud builds submit --tag gcr.io/project_id/test_image

1. Build is finished.
2. Start build again.
3. Consistently see the same error message for all subsequent builds.

Additional Information
Dockerfile:

FROM nginx:1.17.8-alpine
LABEL name="test_image"
RUN rm -rf /etc/nginx/conf.d/* && rm -rf /usr/share/nginx/html

Kaniko image:

Pulling image: gcr.io/kaniko-project/executor:v0.17.1
v0.17.1: Pulling from kaniko-project/executor
Digest: sha256:565d31516f9bb91763dcf8e23ee161144fd4e27624b257674136c71559ce4493

Triage Notes for the Maintainers

| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |

• - [ ]

|
| Please check if the build works in docker but not in kaniko |

• - [x]

|
| Please check if this error is seen when you use --cache flag |

• - [x]

|
| Please check if your dockerfile is a multistage dockerfile |

• - [ ]

|

[mcfedr]

Got something that looks similar to this

Step #0 - "fpm/prod": error building image: error building stage: failed to take snapshot: unable to add file /usr/share/doc/gcc-6/.wh.README.Bugs to layered map: error creating hash for /usr/share/doc/gcc-6/.wh.README.Bugs: lstat /usr/share/doc/gcc-6/.wh.README.Bugs: no such file or directory

Rollback to 0.16.0 and not seeing this.

[cvgw]

This should be fixed now, can you give tag a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79 a try?

[Andor]

@cvgw build with gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 still fails for me.

My Dockerfile:

FROM fedora:31
RUN yum install zstd xz bash tar findutils coreutils buildah skopeo rsync -y

I build in GitLab inside docker container with overlayfs, runner in GCP.

My command: /kaniko/executor --context $CONTEXT --cache --cache-ttl 1440h --destination $DOCKER_IMAGE

[Andor]

Works fine when I dropped the cache by deleting $DOCKER_IMAGE/cache tags.

[mcfedr]

@cvgw the version a1af057f997316bfb1c4d2d82719d78481a02a79 that you linked solved the issue I had,

but - seems to break switching users, the HOME env var is not changed.

This simple docker file illustrates the issue,

FROM alpine
RUN adduser -S -D myuser

RUN echo $HOME
RUN whoami

USER myuser

RUN echo $HOME
RUN whoami

If you run it with latest (or docker build), HOME correctly updates to /home/myuser, with a1af057... it is still /root.

[mcfedr]

Looks like this was fixed by https://github.com/GoogleContainerTools/kaniko/pull/995 but seems to have broken again, might be good to merge https://github.com/GoogleContainerTools/kaniko/pull/1072

[cvgw]

Bummer, thanks for the report @mcfedr. Will look into it

[cvgw]

@cvgw the version a1af057 that you linked solved the issue I had,

but - seems to break switching users, the HOME env var is not changed.

This simple docker file illustrates the issue,

FROM alpine
RUN adduser -S -D myuser

RUN echo $HOME
RUN whoami

USER myuser

RUN echo $HOME
RUN whoami

If you run it with latest (or docker build), HOME correctly updates to /home/myuser, with a1af057... it is still /root.

cc @tejal29, we should fix this before the next release so there isn't another regression. Also should figure out why the integration tests didn't pick this up.

[mcfedr]

Bummer, thanks for the report @mcfedr. Will look into it

I'm sure this doesnt really need saying, but it is something that makes using kaniko difficult, the stability being... a little all over the place. But you do a good job of responding quickly, so thanks for that, and the general experience, of fast builds, is great.

[cvgw]

@cvgw the version a1af057 that you linked solved the issue I had,

but - seems to break switching users, the HOME env var is not changed.

This simple docker file illustrates the issue,

FROM alpine
RUN adduser -S -D myuser

RUN echo $HOME
RUN whoami

USER myuser

RUN echo $HOME
RUN whoami

If you run it with latest (or docker build), HOME correctly updates to /home/myuser, with a1af057... it is still /root.

~I'm not able to repro this bug using tag a1af057f997316bfb1c4d2d82719d78481a02a79~

Dockerfile

FROM alpine
RUN adduser -S -D myuser

USER myuser

Docker command

docker run \
  -v $LOCAL_CONTEXT:/workspace
  gcr.io/kaniko-project/executor:a1af057f997316bfb1c4d2d82719d78481a02a79 \
  --context dir:///workspace/ \
  --dockerfile Dockerfile \
  --destination $DESTINATION \
  -v info

Test

$ docker run --rm $DESTINATION /bin/sh -c 'echo $HOME'
=> /home/myuser

~@mcfedr am I missing something from the repo steps or interpreting the test result wrong?~

This is the same problem as #1082 $HOME is not set correctly during the build, but is correct in the final image.

[kroshilin]

This should be fixed now, can you give tag a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79 a try?

This helped me, thanks. I had to clear (corrupted?) cache before retry.

[3qdev]

@cvgw It works for me, thanks. When do you plan the next release. I need this bugfix.

[cvgw]

@cvgw It works for me, thanks. When do you plan the next release. I need this bugfix.

cc @tejal29 for release schedule

[tejal29]

Verified, following on gcr.io/tejal-test/executor:debug

FROM nginx:1.17.8-alpine
LABEL name="test_image"
RUN rm -rf /etc/nginx/conf.d/* && rm -rf /usr/share/nginx/html

and

FROM alpine
RUN adduser -S -D myuser

RUN echo $HOME
RUN whoami

USER myuser

RUN echo $HOME
RUN whoami

shows the current home dir

tejaldesai@@kaniko (master)$ docker run -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test_1078 --tarPath=t.tar --no-push
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
INFO[0001] Retrieving image manifest alpine             
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest alpine             
INFO[0002] Retrieving image manifest alpine             
INFO[0003] Unpacking rootfs as cmd RUN adduser -S -D myuser requires it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] RUN adduser -S -D myuser                     
INFO[0003] cmd: /bin/sh                                 
INFO[0003] args: [-c adduser -S -D myuser]              
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] RUN echo $HOME                               
INFO[0003] cmd: /bin/sh                                 
INFO[0003] args: [-c echo $HOME]                        
/root
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0003] RUN whoami                                   
INFO[0003] cmd: /bin/sh                                 
INFO[0003] args: [-c whoami]                            
root
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0003] USER myuser                                  
INFO[0003] cmd: USER                                    
INFO[0003] RUN echo $HOME                               
INFO[0003] cmd: /bin/sh                                 
INFO[0003] args: [-c echo $HOME]                        
/home/myuser
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0003] RUN whoami                                   
INFO[0003] cmd: /bin/sh                                 
INFO[0003] args: [-c whoami]                            
myuser
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] Resolving paths                              
INFO[0003] No files were changed, appending empty layer to config. No layer added to image. 
tejaldesai@@kaniko (master)$ 

[purificant]

Thank you for the fix!
Works for me in v0.18.0 for fresh builds. Had to clear caches created by v0.17.1