Kaniko: error building image: error building stage: mkdir /usr/share/bug/systemd: not a directory

[JordanGoasdoue]

Actual behavior
A clear and concise description of what the bug is.

The error is :

INFO[0001] Unpacking rootfs as cmd RUN echo $HOME requires it. 
error building image: error building stage: mkdir /usr/share/bug/systemd: not a directory

This is this Dockerfile which is used to have this error :

FROM eu.gcr.io/xxxxxxxxxxxxx/jessie:v0.0.1-dist-kaniko
USER www-data
RUN echo $HOME

Our home made jessie:v0.0.1-dist-kaniko jessie docker image use debian:jessie-slim as FROM.
systemd deb package is in version 215 on debian:jessie-slim, and eu.gcr.io/xxxxxxxxxxxxx/jessie:v0.0.1-dist-kaniko add as debian distribution a home made one, with inside systemd_230 deb package version.
On the systemd_215, this is a file : /usr/share/bug/systemd

On the systemd_230, this is now a directory with inside files :

/usr/share/bug/systemd/control
/usr/share/bug/systemd/script 

The error appears during the apt-get -y -q dist-upgrade command while upgrading the systemd deb packages from version 215 to version 230
Here i suppose then the error is kaniko doesn't succeed to create the new directory /usr/share/bug/systemd/ with inside files control and script

Expected behavior
A clear and concise description of what you expected to happen.

To pass to the next step which is taking snapshot.

To Reproduce
Steps to reproduce the behavior:

1. Create new Dockerfile for instance tagged jessie:v0.0.1-dist-kaniko with FROM debian:jessie-slim, which contains systemd_215 deb packages
2. On this image, Add distribution with inside systemd_230 (or newest version) deb packages
3. On this image, install new version packages with command apt-get -y -q dist-upgrade
4. Create new Dockerfile for instance tagged jessie:v0.0.1-test-kaniko which will be :

FROM eu.gcr.io/xxxxxxxxxxxxx/jessie:v0.0.1-dist-kaniko
USER www-data
RUN echo $HOME

1. Build this jessie:v0.0.1-test-kaniko image using Kaniko
2. Expect the error :

INFO[0001] Unpacking rootfs as cmd RUN echo $HOME requires it. 
error building image: error building stage: mkdir /usr/share/bug/systemd: not a directory

Triage Notes for the Maintainers

| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |

• - [ ]

|
| Please check if the build works in docker but not in kaniko |

• - [x]

|
| Please check if this error is seen when you use --cache flag |

• - [ ]

|
| Please check if your dockerfile is a multistage dockerfile |

• - [ ]

|

[cvgw]

@rfjg can you please share a Dockerfile to build your base image jessie:v0.0.1-dist-kaniko

[JordanGoasdoue]

With pleasure, I have succeeded to reproduce the same error with this Dockerfile while calling it with the other I already mentionned on my first post :

FROM debian:jessie-slim

RUN set -x \
  && apt-get -y update \
  && echo '\nAPT::Install-Recommends "false";\nAPT::Install-Suggests "false";\n' >> /etc/apt/apt.conf \
  && echo "deb [check-valid-until=no] http://archive.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list \
  && apt-get -o Acquire::Check-Valid-Until=false -y update \
  && apt-get -y -q dist-upgrade \
  && apt-get -y -t jessie-backports install systemd \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

Thanks :+1:

[cvgw]

@tejal29 @samos123 I found some additional info here

When kaniko goes to extract /usr/share/bug that directory already exists on the filesystem (part of kaniko's image?) so os.MkDirAll actually does nothing (if you look at the implementation it just returns nil). But when kaniko goes to extract /usr/share/bug/systemd (which is a directory in the case of the example dockerfile) it finds that /usr/share/bug/systemd already exists in the local file system. BUT this time it is not a directory. In this case, /usr/share/bug/systemd is a regular file so os.MkdirAll returns an error not a directory.

I think this requires a philosophical decision about what kaniko should do when a path in the image being extracted matches a path on the local filesystem BUT they are different types, have different permissions, or different uid/gid.

I would suggest that we delete the path from the existing filesystem and replace it with that one being extracted from the image.

Some logging I added that helps illustrate the situation.

TRAC[0004] creating dir /usr/share/bug
INFO[0004] /usr/share/bug mode drwxr-xr-x uid 0 gid 0
INFO[0004] path /usr/share/bug file apt
INFO[0004] path /usr/share/bug file gnupg
INFO[0004] path /usr/share/bug file procps
INFO[0004] path /usr/share/bug file systemd
INFO[0004] path /usr/share/bug file udev
TRAC[0004] directory path /usr/share/bug exists and is a directory
TRAC[0004] creating dir /usr/share/bug/systemd
INFO[0004] /usr/share/bug mode drwxr-xr-x uid 0 gid 0
ERRO[0004] read dir path /usr/share/bug/systemd readdirent: not a directory
TRAC[0004] directory path /usr/share/bug/systemd exists and is a regular file
DEBU[0004] Whiting out /usr/share/bug/systemd/.wh..wh..opq
error building image: error building stage: failed to get filesystem from image: removing whiteout usr/share/bug/systemd/.wh..wh..opq: unlinkat /usr/share/bug/systemd/.wh..opq: not a directory

[tejal29]

I would suggest that we delete the path from the existing filesystem and replace it with that one being extracted from the image.

+1 to this. I think we should remove what existed before and replace it.

[tejal29]

@tejal29 @samos123 I found some additional info here

When kaniko goes to extract /usr/share/bug that directory already exists on the filesystem (part of kaniko's image?) so os.MkDirAll actually does nothing (if you look at the implementation it just returns nil). But when kaniko goes to extract /usr/share/bug/systemd (which is a directory in the case of the example dockerfile) it finds that /usr/share/bug/systemd already exists in the local file system. BUT this time it is not a directory. In this case, /usr/share/bug/systemd is a regular file so os.MkdirAll returns an error not a directory.

I think this requires a philosophical decision about what kaniko should do when a path in the image being extracted matches a path on the local filesystem BUT they are different types, have different permissions, or different uid/gid.

I would suggest that we delete the path from the existing filesystem and replace it with that one being extracted from the image.

Some logging I added that helps illustrate the situation.

TRAC[0004] creating dir /usr/share/bug
INFO[0004] /usr/share/bug mode drwxr-xr-x uid 0 gid 0
INFO[0004] path /usr/share/bug file apt
INFO[0004] path /usr/share/bug file gnupg
INFO[0004] path /usr/share/bug file procps
INFO[0004] path /usr/share/bug file systemd
INFO[0004] path /usr/share/bug file udev
TRAC[0004] directory path /usr/share/bug exists and is a directory
TRAC[0004] creating dir /usr/share/bug/systemd
INFO[0004] /usr/share/bug mode drwxr-xr-x uid 0 gid 0
ERRO[0004] read dir path /usr/share/bug/systemd readdirent: not a directory
TRAC[0004] directory path /usr/share/bug/systemd exists and is a regular file
DEBU[0004] Whiting out /usr/share/bug/systemd/.wh..wh..opq
error building image: error building stage: failed to get filesystem from image: removing whiteout usr/share/bug/systemd/.wh..wh..opq: unlinkat /usr/share/bug/systemd/.wh..opq: not a directory

This issue was fixed in Pr #1147

[tejal29]

@JordanGoasdoue Looks this issue is now fixed on master.

tejaldesai@@~$  docker run -it --entrypoint /busybox/sh -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/example:/workspace -v /usr/local/google/home/tejaldesai/workspace/kaniko/out:/kaniko gcr.io/kaniko-project/executor:debug-edge

/ # /kaniko/executor -f Dockerfile --context=dir://workspace --destination=

INFO[0000] Retrieving image manifest debian:jessie-slim 
INFO[0001] Retrieving image manifest debian:jessie-slim 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest debian:jessie-slim 
INFO[0002] Retrieving image manifest debian:jessie-slim 
INFO[0003] Executing 0 build triggers                   
INFO[0003] Unpacking rootfs as cmd RUN set -x   && apt-get -y update   && echo '\nAPT::Install-Recommends "false";\nAPT::Install-Suggests "false";\n' >> /etc/apt/apt.conf   && echo "deb [check-valid-until=no] http://archive.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list   && apt-get -o Acquire::Check-Valid-Until=false -y update   && apt-get -y -q dist-upgrade   && apt-get -y -t jessie-backports install systemd   && apt-get clean   && rm -rf /var/lib/apt/lists/* requires it. 
INFO[0004] RUN set -x   && apt-get -y update   && echo '\nAPT::Install-Recommends "false";\nAPT::Install-Suggests "false";\n' >> /etc/apt/apt.conf   && echo "deb [check-valid-until=no] http://archive.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list   && apt-get -o Acquire::Check-Valid-Until=false -y update   && apt-get -y -q dist-upgrade   && apt-get -y -t jessie-backports install systemd   && apt-get clean   && rm -rf /var/lib/apt/lists/* 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0005] Resolving 5430 paths                         
INFO[0005] cmd: /bin/sh                                 
INFO[0005] args: [-c set -x   && apt-get -y update   && echo '\nAPT::Install-Recommends "false";\nAPT::Install-Suggests "false";\n' >> /etc/apt/apt.conf   && echo "deb [check-valid-until=no] http://archive.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list   && apt-get -o Acquire::Check-Valid-Until=false -y update   && apt-get -y -q dist-upgrade   && apt-get -y -t jessie-backports install systemd   && apt-get clean   && rm -rf /var/lib/apt/lists/*] 
INFO[0005] Running: [/bin/sh -c set -x   && apt-get -y update   && echo '\nAPT::Install-Recommends "false";\nAPT::Install-Suggests "false";\n' >> /etc/apt/apt.conf   && echo "deb [check-valid-until=no] http://archive.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list   && apt-get -o Acquire::Check-Valid-Until=false -y update   && apt-get -y -q dist-upgrade   && apt-get -y -t jessie-backports install systemd   && apt-get clean   && rm -rf /var/lib/apt/lists/*] 
+ apt-get -y update

...
+ apt-get clean
+ rm -rf /var/lib/apt/lists/archive.debian.org_debian_dists_jessie-backports_InRelease /var/lib/apt/lists/archive.debian.org_debian_dists_jessie-backports_main_binary-amd64_Packages.gz /var/lib/apt/lists/deb.debian.org_debian_dists_jessie-updates_InRelease /var/lib/apt/lists/deb.debian.org_debian_dists_jessie-updates_main_binary-amd64_Packages.gz /var/lib/apt/lists/deb.debian.org_debian_dists_jessie_Release /var/lib/apt/lists/deb.debian.org_debian_dists_jessie_Release.gpg /var/lib/apt/lists/deb.debian.org_debian_dists_jessie_main_binary-amd64_Packages.gz /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_jessie_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_jessie_updates_main_binary-amd64_Packages.gz
INFO[0071] Taking snapshot of full filesystem...        
INFO[0071] Resolving 5484 paths  

I also verified the image built could be pulled correctly.

docker pull gcr.io/tejal-test/test1
Using default tag: latest
latest: Pulling from tejal-test/test1
370e9f631e97: Pull complete 
9c271a17c613: Pull complete 
Digest: sha256:07bd6d5c89602325bdb7e895607bb8a40b00344f06946fb05f310b9dca083eea
Status: Downloaded newer image for gcr.io/tejal-test/test1:latest
gcr.io/tejal-test/test1:latest

I am going to close this now!

[zaventh]

Is this really fixed? I am running in to a similar issue:

error building image: error building stage: failed to get filesystem from image: mkdir /var/spool/mail: file exists

I have built Kaniko in an Alpine base image where /var/spool/mail is actually a symlink, not a directory.

bash-5.0# ls -la /var/spool/mail
lrwxrwxrwx 1 root root 9 Oct 21 09:23 /var/spool/mail -> /var/mail

However when building a centos image and running useradd, it tries to mkdir /var/spool/mail which fails with the aforementioned error.

I was able to work around it for now by deleting the symlink in the base image and making it a regular directory, but this thread suggests I shouldn't have to do that any longer. What are the rules exactly?