Kaniko: Missing files in Kaniko-built image

Created on 12 Feb 2020 · 12Comments · Source: GoogleContainerTools/kaniko

Actual behavior
When building from Red Hat's UBI 8 and installing nginx a library is missing in the resulting image preventing docker from pulling it.

$ docker pull registry.example.com/nginx:v0.2
v0.2: Pulling from it/docker-images/nginx
03e56b46bf0b: Already exists
3a13cc2f5d65: Already exists
d42cbf90e64a: Pull complete
b4427cf0b409: Extracting [==================================================>] 
 22.93MB/22.93MB
97d00bd6afb4: Download complete
failed to register layer: Error processing tar file(exit status 1): open /lib64/libtirpc.so.3.0.0: no such file or directory

Expected behavior
The files should be present

To Reproduce

$ docker run --rm -v $(pwd):/workspace gcr.io/kaniko-project/executor:debug

Additional Information

$ docker run --rm gcr.io/kaniko-project/executor:debug version  
Kaniko version :  v0.17.1

$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:26:10 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:24:42 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Build Context:

Dockerfile
```
FROM registry.access.redhat.com/ubi8/ubi-minimal

# Copy nginx repo
COPY nginx.repo /etc/yum.repos.d/nginx.repo

# TODO: Add --nodocs or --setopt=tsflags=nodocs once
# https://bugzilla.redhat.com/show_bug.cgi?id=1769831
# is closed
RUN microdnf install nginx \
&& microdnf clean all

EXPOSE 8080/tcp

VOLUME /usr/share/nginx/html

# Run nginx
CMD ["/usr/sbin/nginx", "-g", "daemon off;"]
```

nginx.repo
[nginx-stable] name=nginx stable repo baseurl=http://nginx.org/packages/rhel/$releasever/$basearch/ gpgcheck=1 enabled=1 gpgkey=https://nginx.org/keys/nginx_signing.key module_hotfixes=true

Kaniko Image (fully qualified with digest)
gcr.io/kaniko-project/executor:debug@sha256:025bd79d3e0699b5f59142b03f7e66916980bd0e32653b9c7e21b561d4e538c3

Triage Notes for the Maintainers

| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |
- - [ ]
|
| Please check if the build works in docker but not in kaniko |
- - [x]
|
| Please check if this error is seen when you use --cache flag |
- - [ ]
|
| Please check if your dockerfile is a multistage dockerfile |
- - [ ]
|

arefilesystems in progress kinbug prioritp0 regression

Source

HaehnleinMar

👍3

Most helpful comment

Thanks @HaehnleinMar Sorry for the regression. Please use v0.16.0 untill we fix this issue.

tejal29 on 13 Feb 2020

👍2 👎1

All 12 comments

Could be related to:

1039

1038

1024

1028

I had a working build by moving from AmazonLinux:2018.03 to AmazonLinux:2, but pip3 was not working with that image. Yesterday I fixed the issue with pip3, and the images didn't work anymore (build successful, but pulling images doesn't work).

It's always something different, sometimes it is git, or zip, or java ... tried different approaches including not uninstalling installed software.

stephanf on 12 Feb 2020

Also getting this on a go build with 0.17.1.

error building image: error building stage: failed to take snapshot: unable to add file /root/.cache/go-build/f6/.wh.f6274bb3f6af4a08a241f2da8b1723712d7eb3e2038fa9bf9fc2f5958addf420-a to layered map: error creating hash for /root/.cache/go-build/f6/.wh.f6274bb3f6af4a08a241f2da8b1723712d7eb3e2038fa9bf9fc2f5958addf420-a: lstat /root/.cache/go-build/f6/.wh.f6274bb3f6af4a08a241f2da8b1723712d7eb3e2038fa9bf9fc2f5958addf420-a: no such file or directory

tbonfort on 12 Feb 2020

Thanks @HaehnleinMar Sorry for the regression. Please use v0.16.0 untill we fix this issue.

tejal29 on 13 Feb 2020

👍2 👎1

Thank you @tejal29 for that suggestion. Using v0.16.0 it works like a charm :+1:

HaehnleinMar on 13 Feb 2020

guys I need an equivalent version of v0.16.0 for executor:debug. What would that be?

binnythomas-1989 on 18 Feb 2020

@binnythomas-1989 please use the tag v0.16.0-debug

tejal29 on 18 Feb 2020

👎1

debug-v0.16.0

stephanf on 18 Feb 2020

I believe this is the same issue as #1039 as is fixed in a1af057f997316bfb1c4d2d82719d78481a02a79.

Tags a1af057f997316bfb1c4d2d82719d78481a02a79 and debug-a1af057f997316bfb1c4d2d82719d78481a02a79 have the new code

cvgw on 25 Feb 2020

I verfied this on latest image gcr.io/tejal-test/executor

docker run -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test_1045 '

Complete.
INFO[0014] Taking snapshot of full filesystem...        
INFO[0014] Resolving paths                              
INFO[0016] EXPOSE 8080/tcp                              
INFO[0016] cmd: EXPOSE                                  
INFO[0016] Adding exposed port: 8080/tcp                
INFO[0016] VOLUME /usr/share/nginx/html                 
INFO[0016] cmd: VOLUME                                  
INFO[0016] adding volume /usr/share/nginx/html to whitelist 
INFO[0016] CMD ["/usr/sbin/nginx", "-g", "daemon off;"] 
tejaldesai@@kaniko (r-v0.18.0)$

tejaldesai@@kaniko (r-v0.18.0)$ docker pull gcr.io/tejal-test/test_1045
Using default tag: latest
latest: Pulling from tejal-test/test_1045
0a63a759fe25: Pull complete 
9d6c79b335fa: Pull complete 
8975e1c4bc3c: Pull complete 
43d749462e9b: Pull complete 
Digest: sha256:ddc9bc6d6d819618ff871e1b04414ab36088d14c3dbf292b0fa39dac24038b9f
Status: Downloaded newer image for gcr.io/tejal-test/test_1045:latest
gcr.io/tejal-test/test_1045:latest

tejal29 on 6 Mar 2020

Facing the same issue with build image usingdebug-v0.19.0 executor, but inside running container:

Unable to find image '<somename>' locally
latest: Pulling from <somename>
ac9208207ada: Pull complete
4f3f99a61312: Pull complete
fd34dc44f01c: Pull complete
f4efdfd25de7: Pull complete
2fcdbdc9c4c7: Pull complete
5a4f61ccd968: Pull complete
4c85c679d9e1: Pull complete
9c1482cd1d98: Pull complete
f37097ffe8b0: Pull complete
e0ff42de1c82: Pull complete
c344f69900f3: Pull complete
b67f438365c2: Pull complete
0e551847c7df: Pull complete
3fc55cba6164: Pull complete
Digest: sha256:e81612d107dc045d0736b9c9f750d42cec3d97cee50d7edcd8068832709f1375
Status: Downloaded newer image for <somename>
[root@86f72346de6b devops]#
[root@86f72346de6b devops]# yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   libelf.so.1: cannot open shared object file: No such file or directory

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.5 (default, Oct 30 2018, 23:45:53)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq


[root@86f72346de6b devops]#

Workaround - run ldconfig

gmmephisto on 14 Apr 2020

And another fail on pulling image:

$ docker run -ti --rm <somename>
Unable to find image '<somename>' locally
ac9208207ada: Already exists
4f3f99a61312: Already exists
fd34dc44f01c: Already exists
f4efdfd25de7: Already exists
2fcdbdc9c4c7: Already exists
5a4f61ccd968: Already exists
4c85c679d9e1: Already exists
9c1482cd1d98: Already exists
f37097ffe8b0: Already exists
06e1d006bd86: Pull complete
c599036dfe85: Pull complete
1094881658c8: Pull complete
655e5f9566d6: Pull complete
8306954b60d9: Pull complete
d545b9e2c7a9: Extracting [==================================================>]  151.7MB/151.7MB
docker: failed to register layer: Error processing tar file(exit status 1): file exists.
See 'docker run --help'.

Was successfully build with existing cache. I'll try to figure out how to stable reproduce this error.

gmmephisto on 14 Apr 2020

Same issue with debug-v0.19.0