Buildkit: [master] Cache issue, digest should be identical

[crazy-max]

Ref. https://github.com/docker/build-push-action/runs/1228146265?check_suite_focus=true#step:13:9
cc. @tonistiigi

Dockerfile

FROM alpine

RUN echo "Hello world!"

Local registry

docker run -d --name registry -p 5000:5000 registry:2

Repro script

#!/bin/sh

BUILDER_NAME=buildkit_repro
LOCAL_CACHE_DIR=/tmp/.buildx-cache
IIDFILE1=/tmp/digest1
IIDFILE2=/tmp/digest2

rm -rf $LOCAL_CACHE_DIR $IIDFILE1 $IIDFILE2
docker buildx rm ${BUILDER_NAME} || true

docker buildx create --name ${BUILDER_NAME} \
  --driver docker-container \
  --driver-opt network=host \
  --driver-opt "image=moby/buildkit:v0.8-beta" \
  --buildkitd-flags "--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host"

docker buildx inspect --bootstrap ${BUILDER_NAME}

#[+] Building 3.3s (1/1) FINISHED
# => [internal] booting buildkit                                                                                                                                                                                      
# => => pulling image moby/buildkit:v0.8-beta                                                                                                                                                                          
# => => creating container buildx_buildkit_buildkit_repro0                                                                                                                                                             
#Name:   buildkit_repro
#Driver: docker-container
#
#Nodes:
#Name:      buildkit_repro0
#Endpoint:  unix:///var/run/docker.sock
#Status:    running
#Flags:     --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
#Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

docker buildx build --progress plain \
  --builder ${BUILDER_NAME} \
  --tag localhost:5000/name/app:latest \
  --platform linux/amd64,linux/arm64 \
  --cache-from type=local,src=${LOCAL_CACHE_DIR} \
  --cache-to type=local,dest=${LOCAL_CACHE_DIR} \
  --iidfile ${IIDFILE1} \
  --file ./Dockerfile --push .

#WARN[0000] invalid non-bool value for BUILDX_NO_DEFAULT_LOAD:
#WARN[0000] local cache import at /tmp/.buildx-cache not found due to err: could not read /tmp/.buildx-cache/index.json: open /tmp/.buildx-cache/index.json: no such file or directory
##1 [internal] load build definition from Dockerfile
##1 transferring dockerfile: 77B 0.0s done
##1 DONE 0.1s
#
##2 [internal] load .dockerignore
##2 transferring context: 2B 0.0s done
##2 DONE 0.1s
#
##4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
##4 DONE 2.0s
#
##3 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
##3 DONE 2.0s
#
##5 [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:18551807089175890...
##5 resolve docker.io/library/alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s done
##5 DONE 0.1s
#
##7 [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:18551807089175890...
##7 resolve docker.io/library/alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s done
##7 sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 0B / 2.71MB 0.2s
##7 sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 1.05MB / 2.71MB 0.3s
##7 sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 2.10MB / 2.71MB 0.5s
##7 ...
#
##5 [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:18551807089175890...
##5 sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.80MB / 2.80MB 0.5s done
##5 extracting sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 0.1s done
##5 DONE 0.7s
#
##7 [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:18551807089175890...
##7 sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 2.71MB / 2.71MB 0.5s done
##7 extracting sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd
##7 extracting sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 0.1s done
##7 DONE 0.6s
#
##6 [linux/amd64 2/2] RUN echo "Hello world!"
##6 0.783 Hello world!
##6 DONE 0.8s
#
##8 [linux/arm64 2/2] RUN echo "Hello world!"
##8 0.754 Hello world!
##8 DONE 0.8s
#
##9 exporting to image
##9 exporting layers
##9 exporting layers 0.2s done
##9 exporting manifest sha256:f808cf05ac62f6761689fab6c97fd2ad3416ea9c4348cec7740aed601ff7d71e 0.0s done
##9 exporting config sha256:936d7905d55cbee132d84867409f70f770fdf65d8f4218f3f8808a0f2d6fb09f 0.0s done
##9 exporting manifest sha256:63429b119a387b71d2c984e70a394e715a87eadc39a99fe3f2064cb0a45c8ae1 0.0s done
##9 exporting config sha256:e839aeeb4af56adc264913fad713941921c52e7fa4b57c42e3c8d86a5e233e3c 0.0s done
##9 exporting manifest list sha256:ddd15800a4e00050dbfd5a23eb851a1cd3d9451786910181c9a8955cde3fb946 0.0s done
##9 pushing layers 0.1s done
##9 pushing manifest for localhost:5000/name/app:latest 0.1s done
##9 DONE 0.5s
#
##10 exporting cache
##10 preparing build cache for export 0.0s done
##10 writing layer sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 0.1s done
##10 writing layer sha256:d05c82b7372d59e4b82319e8b6a6e73a0140dd93c347deea6994cc6811c2d392 done
##10 writing layer sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 0.0s done
##10 writing config sha256:b5cef521cfa2b1a1bcfcace83f7fa80c07715398dac68636791b396289d5c9c8 done
##10 writing manifest sha256:ee5cfa03d78b67cef19118d544510e6a4cfdc8d9ef5f587a3b5ee1f9a91044e7 done
##10 DONE 0.1s

docker buildx prune --builder ${BUILDER_NAME} -a -f

#ID                                                                      RECLAIMABLE     SIZE    LAST ACCESSED
#oc5mj4iwp28go9qlkenvvvc1e                                               true            8.32kB
#n89qpp684bc0gjx7goxgd8e51                                               true            8.32kB
#9fz0fgn5tp1zamdp0dkt0jn87*                                              true            4.10kB
#ode8aggkdncmqmqup1mcos7di*                                              true            4.14kB
#k7dc7jbl3stq459cdi4q6usih                                               true            8.76MB
#0n94ofzpwdhh5k2k43cqm3vvp                                               true            8.42MB
#Total:  17.20MB

docker buildx build --progress plain \
  --builder ${BUILDER_NAME} \
  --tag localhost:5000/name/app:latest \
  --platform linux/amd64,linux/arm64 \
  --cache-from type=local,src=${LOCAL_CACHE_DIR} \
  --cache-to type=local,dest=${LOCAL_CACHE_DIR} \
  --iidfile ${IIDFILE2} \
  --file ./Dockerfile --push .

#WARN[0000] invalid non-bool value for BUILDX_NO_DEFAULT_LOAD:
##2 [internal] load build definition from Dockerfile
##2 transferring dockerfile: 77B done
##2 DONE 0.1s
#
##1 [internal] load .dockerignore
##1 transferring context: 2B done
##1 DONE 0.1s
#
##4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
##4 DONE 0.7s
#
##3 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
##3 DONE 0.8s
#
##7 importing cache manifest from local:13915614070024948304
##7 DONE 0.0s
#
##8 [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:18551807089175890...
##8 resolve docker.io/library/alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s done
##8 DONE 0.1s
#
##5 [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:18551807089175890...
##5 resolve docker.io/library/alpine@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.1s done
##5 DONE 0.1s
#
##6 [linux/amd64 2/2] RUN echo "Hello world!"
##6 CACHED
#
##9 [linux/arm64 2/2] RUN echo "Hello world!"
##9 CACHED
#
##10 exporting to image
##10 exporting layers done
##10 exporting manifest sha256:f808cf05ac62f6761689fab6c97fd2ad3416ea9c4348cec7740aed601ff7d71e 0.0s done
##10 exporting config sha256:936d7905d55cbee132d84867409f70f770fdf65d8f4218f3f8808a0f2d6fb09f
##10 exporting config sha256:936d7905d55cbee132d84867409f70f770fdf65d8f4218f3f8808a0f2d6fb09f 0.0s done
##10 exporting manifest sha256:c3625799fdae70211596a4f9d9367f60d98f7dce462a6f345d01af763b45a5e6 0.0s done
##10 exporting config sha256:d9566c240f8107087aa450f8cab37da5316f6d0a5bd2c140836556e3c40bac7a 0.0s done
##10 exporting manifest list sha256:6ce940e49c0f76ecd59cdbd0ace4a07a49c30515e56f7acc2559d466da5ab059 0.0s done
##10 pushing layers 0.0s done
##10 pushing manifest for localhost:5000/name/app:latest
##10 pushing manifest for localhost:5000/name/app:latest 0.1s done
##10 DONE 0.2s
#
##11 exporting cache
##11 preparing build cache for export done
##11 writing layer sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd done
##11 writing layer sha256:d05c82b7372d59e4b82319e8b6a6e73a0140dd93c347deea6994cc6811c2d392 done
##11 writing layer sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c done
##11 writing config sha256:b5cef521cfa2b1a1bcfcace83f7fa80c07715398dac68636791b396289d5c9c8 done
##11 writing manifest sha256:ee5cfa03d78b67cef19118d544510e6a4cfdc8d9ef5f587a3b5ee1f9a91044e7 done
##11 DONE 0.0s

cat ${IIDFILE1} && echo ""
cat ${IIDFILE2} && echo ""

#sha256:ddd15800a4e00050dbfd5a23eb851a1cd3d9451786910181c9a8955cde3fb946
#sha256:6ce940e49c0f76ecd59cdbd0ace4a07a49c30515e56f7acc2559d466da5ab059

if [ "$(cat ${IIDFILE1})" != "$(cat ${IIDFILE2})" ]; then
  >&2 echo "ERROR: Digests should be identical"
  exit 1
fi

#ERROR: Digests should be identical

[crazy-max]

Works fine with --driver-opt "image=moby/buildkit:buildx-stable-1"

#sha256:dd26cc4c475f15fe446f874b76bb668fe4939b7561627d159bbd6afe3a5d1054
#sha256:dd26cc4c475f15fe446f874b76bb668fe4939b7561627d159bbd6afe3a5d1054

[tonistiigi]

@crazy-max Thanks for a good repro.

I bisected this to #1645 that was quite unexpected. @AkihiroSuda Any ideas?

[AkihiroSuda]

This continuity change might be potentially related?

https://github.com/moby/buildkit/pull/1645/files#diff-d4d8c24a8c3e02973685287a9d9e60b7

[tonistiigi]

@AkihiroSuda Yes, the difference seems to be that with differ changes the layer created with RUN echo is now same for both platforms. This means that initially the snapshots are different with different timestamps. The difference between the 2 digests is that timestamp for the shared layer is not preserved and taken from the same layer on another platform.

Some of these cases can not be prevented, at least by current design where creation timestamp does not define uniqueness for the layer. If you already have partial previous cache locally with different timestamps, that would be used instead. But in this case the base layer is different so there might be something we can do.

[tonistiigi]

Looked more into this and there is no way to fix this without making changes to the remote cache json format. I can fix the duplicate timestamp issue in the manifest list for this case but on importing the way blob is found for a record in the cache config is by the content digest so if there are 2 descriptors in the manifest list with the same digest only one of them would be loaded anyway.

https://github.com/moby/buildkit/blob/master/cache/remotecache/v1/doc.go#L18 ideas?

[AkihiroSuda]

Can't we rollback the behavior of continuity?

[tonistiigi]

@AkihiroSuda The continuity behavior looks technically correct. There shouldn't be changes between these layers. The only thing that is different is the creation time.

[tonistiigi]

https://github.com/moby/buildkit/pull/1739 fixes this specific case by creating an actual empty layer for this RUN command

[crazy-max]

@tonistiigi This is fixed, I no longer reproduce the issue with moby/buildkit:master.