Kaniko: Impossible to build a dockerfile with continuumio/miniconda3:4.7.12

[WildMattock]

I'm trying to build the following dockerfile in kaniko:
Dockerfile.txt

The command is:
/kaniko/executor --dockerfile=./Dockerfile --context=./ --destination=registry.container-registry.svc.cluster.local:5000/templates/test1:0.0.1

Follows the log with the error:

---

INFO[0030] Resolved base name continuumio/miniconda3:4.7.12 to continuumio/miniconda3:4.7.12
INFO[0030] Resolved base name continuumio/miniconda3:4.7.12 to continuumio/miniconda3:4.7.12
INFO[0030] Retrieving image manifest continuumio/miniconda3:4.7.12
INFO[0061] Image continuumio/miniconda3:4.7.12 not found in cache
INFO[0061] Retrieving image manifest continuumio/miniconda3:4.7.12
INFO[0082] Built cross stage deps: map[]
INFO[0082] Retrieving image manifest continuumio/miniconda3:4.7.12
INFO[0092] Image continuumio/miniconda3:4.7.12 not found in cache
INFO[0092] Retrieving image manifest continuumio/miniconda3:4.7.12
INFO[0103] Unpacking rootfs as cmd RUN set -x && apt-get update && apt-get install --no-install-recommends --no-install-suggests -y libpq-dev build-essential && rm -rf /var/lib/apt/lists/* requires it.
INFO[0114] Taking snapshot of full filesystem...
INFO[0150] ENTRYPOINT [ "/bin/bash", "-c" ]
INFO[0150] EXPOSE 5000
INFO[0150] cmd: EXPOSE
INFO[0150] Adding exposed port: 5000/tcp
INFO[0150] RUN set -x && apt-get update && apt-get install --no-install-recommends --no-install-suggests -y libpq-dev build-essential && rm -rf /var/lib/apt/lists/*
INFO[0150] cmd: /bin/sh
INFO[0150] args: [-c set -x && apt-get update && apt-get install --no-install-recommends --no-install-suggests -y libpq-dev build-essential && rm -rf /var/lib/apt/lists/*]

• apt-get update
  Hit:1 http://deb.debian.org/debian buster InRelease
  Hit:2 http://deb.debian.org/debian buster-updates InRelease
  Hit:3 http://security.debian.org/debian-security buster/updates InRelease
  Reading package lists...
  E: Release file for http://deb.debian.org/debian/dists/buster-updates/InRelease is expired (invalid since 69d 2h 16min 4s). Updates for this repository will not be applied.
  E: Release file for http://security.debian.org/debian-security/dists/buster/updates/InRelease is expired (invalid since 69d 1h 8min 37s). Updates for this repository will not be applied.
  error building image: error building stage: failed to execute command: waiting for process to exit: exit status 100

---

I've tried to build it with docker and it's ok.

Do you have any idea why this behavior?

Thanks in advance.

[WildMattock]

Meanwile I found a workaround:
In the dockerfile, placing the command [rm -rf /var/lib/apt/lists/*] before the command [apt-get update] it works.

Docker build works better in this situation because every RUN starts from a clean situation (no caching previous layers).
I think Kaniko should do the same.

[WanzenBug]

I also encountered this issue. After some debugging I noticed that apt-get update only "Hits" the urls, so no refreshing the content. Normally I would expect out-of-date lists to be fetched. However it seems apt bases this decision on the modification date of the fetched list.

So during kaniko build I get:

INFO[0041] args: [-c stat /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease] 
  File: /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease
  Size: 65372       Blocks: 128        IO Block: 4096   regular file
Device: 5ah/90d Inode: 1675188     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-01-14 11:04:33.420231465 +0000
Modify: 2020-01-14 11:04:26.281185105 +0000
Change: 2020-01-14 11:04:26.281185105 +0000
 Birth: -

However the same file in docker is

stat /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease
  File: /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease
  Size: 65372       Blocks: 128        IO Block: 4096   regular file
Device: 5ah/90d Inode: 2795291     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-12-02 23:51:32.000000000 +0000
Modify: 2019-12-02 23:51:32.000000000 +0000
Change: 2020-01-07 11:30:37.872005263 +0000
 Birth: -

So in kaniko my output is

INFO[0044] args: [-c apt-get update]                    
Hit:1 http://deb.debian.org/debian buster InRelease                                                 
Hit:2 http://deb.debian.org/debian buster-updates InRelease                                         
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
Reading package lists... Done
E: Release file for http://deb.debian.org/debian/dists/buster-updates/InRelease is expired (invalid since 34d 8h 36min 34s). Updates for this repository will not be applied.
E: Release file for http://security.debian.org/debian-security/dists/buster/updates/InRelease is expired (invalid since 35d 11h 13min 8s). Updates for this repository will not be applied.

and in docker (notice that its says Get instead of Hit):

$ apt-get update
Hit:1 http://deb.debian.org/debian buster InRelease                                                                    
Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]                                 
Get:3 http://security-cdn.debian.org/debian-security buster/updates InRelease [65.4 kB] 
Get:4 https://apt-archive.apps.openshift.grasp-cloud.com/repository/apt bionic/main all Packages [56.6 kB]
Get:5 https://apt-archive.apps.openshift.grasp-cloud.com/repository/apt bionic/main amd64 Packages [128 kB]
Get:6 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [171 kB]

[cvgw]

I also encountered this issue. After some debugging I noticed that apt-get update only "Hits" the urls, so no refreshing the content. Normally I would expect out-of-date lists to be fetched. However it seems apt bases this decision on the modification date of the fetched list.

So during kaniko build I get:

INFO[0041] args: [-c stat /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease] 
  File: /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease
  Size: 65372         Blocks: 128        IO Block: 4096   regular file
Device: 5ah/90d   Inode: 1675188     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-01-14 11:04:33.420231465 +0000
Modify: 2020-01-14 11:04:26.281185105 +0000
Change: 2020-01-14 11:04:26.281185105 +0000
 Birth: -

However the same file in docker is

stat /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease
  File: /var/lib/apt/lists/security.debian.org_debian-security_dists_buster_updates_InRelease
  Size: 65372         Blocks: 128        IO Block: 4096   regular file
Device: 5ah/90d   Inode: 2795291     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-12-02 23:51:32.000000000 +0000
Modify: 2019-12-02 23:51:32.000000000 +0000
Change: 2020-01-07 11:30:37.872005263 +0000
 Birth: -

So in kaniko my output is

INFO[0044] args: [-c apt-get update]                    
Hit:1 http://deb.debian.org/debian buster InRelease                                                 
Hit:2 http://deb.debian.org/debian buster-updates InRelease                                         
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
Reading package lists... Done
E: Release file for http://deb.debian.org/debian/dists/buster-updates/InRelease is expired (invalid since 34d 8h 36min 34s). Updates for this repository will not be applied.
E: Release file for http://security.debian.org/debian-security/dists/buster/updates/InRelease is expired (invalid since 35d 11h 13min 8s). Updates for this repository will not be applied.

and in docker (notice that its says Get instead of Hit):

$ apt-get update
Hit:1 http://deb.debian.org/debian buster InRelease                                                                    
Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]                                 
Get:3 http://security-cdn.debian.org/debian-security buster/updates InRelease [65.4 kB] 
Get:4 https://apt-archive.apps.openshift.grasp-cloud.com/repository/apt bionic/main all Packages [56.6 kB]
Get:5 https://apt-archive.apps.openshift.grasp-cloud.com/repository/apt bionic/main amd64 Packages [128 kB]
Get:6 http://security-cdn.debian.org/debian-security buster/updates/main amd64 Packages [171 kB]

I wonder if this could be related to https://github.com/GoogleContainerTools/kaniko/issues/793

[cvgw]

The example dockerfile included here references local files, the contents of which I'm not aware of.

Can someone supply an example file that does not depend on local files or provides instructions for generating those local files?

[WildMattock]

The example dockerfile included here references local files, the contents of which I'm not aware of.

Can someone supply an example file that does not depend on local files or provides instructions for generating those local files?

I attached the dockerfile with COPY command as example but you can consider the following:

FROM continuumio/miniconda3:4.7.12
ENTRYPOINT [ "/bin/bash", "-c" ]
EXPOSE 5000
RUN set -x && apt-get update && apt-get install --no-install-recommends --no-install-suggests -y libpq-dev build-essential && rm -rf /var/lib/apt/lists/*
CMD [ "/bin/bash", "-c" ]

[cvgw]

Thanks @WildMattock I'm able to repro with that.

@WanzenBug Thanks for sharing that investigation. It lead me right to the problem (or at least one of them)

When kaniko extracts a file from a base image (or a cached image) it is not setting atime or mtime on the file.

https://github.com/GoogleContainerTools/kaniko/blob/00a01f9f290b8df7620adbc199d03954eed312ba/pkg/util/fs_util.go#L245

By adding in the correct code for setting mtime and atime I am able to successfully build the supplied dockerfile. I'll open a PR.