Buildx: Docker desktop for mac `buildx` confuses layers between architectures

[securisec]

• [x] I have tried with the latest version of my channel (Stable or Edge)
• [ ] I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

buildx build images should be using the right layers for different architectures

Actual behavior

buildx is confusing layers between architectures even if the final builds are passing.

Information

Docker for Mac: 10.15.5
Docker version: Docker version 19.03.12, build 48a66213fe

Steps to reproduce the behavior

• docker buildx build --platform linux/arm64 -t mytag:devarm .
• Upload to hub.docker
• Download to pi. Everything works so far in a pi
• build a regular x64 image using docker build -t mytag:dev .
• x64 works fine.
• Make some file changes and rerun buildx
• Upload to hub.docker
• Download new image to pi. Now when trying to run in a pi, it will throw the exec error indicating wrong architecture
• do docker system prune -a in Mac and then rebuild the arm image with the above specified command.
• Upload to hub.docker
• Download again to pi. now it will work again in pi

Based on a test case:
Dockerfile:

FROM golang:1.14.2-alpine AS server
WORKDIR /server
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o test

FROM alpine:3.9.6
WORKDIR /app
COPY --from=server /server/test .
CMD ["/app/test"]

• First i built the x64 image and pushed to docker hub with dev tag.

✖ docker build -t securisec/test:dev .
Sending build context to Docker daemon  4.096kB
Step 1/8 : FROM golang:1.14.2-alpine AS server
1.14.2-alpine: Pulling from library/golang
cbdbe7a5bc2a: Already exists
408f87550127: Pull complete
fe522b08c979: Pull complete
618fff1cf170: Pull complete
0d8b518583db: Pull complete
Digest: sha256:9b3ad7928626126b72b916609ad081cfb6c0149f6e60cef7fc1e9e15a0d1e973
Status: Downloaded newer image for golang:1.14.2-alpine
 ---> dda4232b2bd5
Step 2/8 : WORKDIR /server
 ---> Running in 812d83680d54
Removing intermediate container 812d83680d54
 ---> 796f94f11c1f
Step 3/8 : COPY . .
 ---> fd91c2de467a
Step 4/8 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o test
 ---> Running in df3e89043fe0
Removing intermediate container df3e89043fe0
 ---> e50846f77c77
Step 5/8 : FROM alpine:3.9.6
3.9.6: Pulling from library/alpine
31603596830f: Pull complete
Digest: sha256:414e0518bb9228d35e4cd5165567fb91d26c6a214e9c95899e1e056fcd349011
Status: Downloaded newer image for alpine:3.9.6
 ---> 78a2ce922f86
Step 6/8 : WORKDIR /app
 ---> Running in 71028b47eab1
Removing intermediate container 71028b47eab1
 ---> f132b619680e
Step 7/8 : COPY --from=server /server/test .
 ---> 801e148986c4
Step 8/8 : CMD ["/app/test"]
 ---> Running in a14235433091
Removing intermediate container a14235433091
 ---> 22f132854b69
Successfully built 22f132854b69
Successfully tagged securisec/test:dev
/tmp/test   19s
❯❯ docker push securisec/test:dev
The push refers to repository [docker.io/securisec/test]
3ccfef4aa468: Pushed
cd4f73738e5f: Pushed
89ae5c4ee501: Mounted from securisec/save
dev: digest: sha256:46ec9e57c9a1c902f77675ee1e65683e96f105329ce43abf32ef74852e4d209c size: 946

• Then i built the arm image with the tag devarm and pushed to hub.

❯❯ docker buildx build --platform linux/arm64 -t securisec/test:devarm .
[+] Building 3.5s (13/13) FINISHED
 => [internal] load build definition from Dockerfile                                                     0.0s
 => => transferring dockerfile: 252B                                                                     0.0s
 => [internal] load .dockerignore                                                                        0.0s
 => => transferring context: 2B                                                                          0.0s
 => [internal] load metadata for docker.io/library/alpine:3.9.6                                          0.0s
 => [internal] load metadata for docker.io/library/golang:1.14.2-alpine                                  0.0s
 => [server 1/4] FROM docker.io/library/golang:1.14.2-alpine                                             0.0s
 => => resolve docker.io/library/golang:1.14.2-alpine                                                    0.0s
 => [internal] load build context                                                                        0.0s
 => => transferring context: 421B                                                                        0.0s
 => [stage-1 1/3] FROM docker.io/library/alpine:3.9.6                                                    0.0s
 => => resolve docker.io/library/alpine:3.9.6                                                            0.0s
 => [stage-1 2/3] WORKDIR /app                                                                           0.0s
 => [server 2/4] WORKDIR /server                                                                         0.0s
 => [server 3/4] COPY . .                                                                                0.0s
 => [server 4/4] RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o test                     3.3s
 => [stage-1 3/3] COPY --from=server /server/test .                                                      0.1s
 => exporting to image                                                                                   0.0s
 => => exporting layers                                                                                  0.0s
 => => writing image sha256:0158c16c64fb86955a03a28580a59ffce40b8d3dc1304eff814e0353552d01f3             0.0s
 => => naming to docker.io/securisec/test:devarm                                                         0.0s
/tmp/test   3s
❯❯ docker push securisec/test:devarm
The push refers to repository [docker.io/securisec/test]
a46a31b336a7: Pushed
21428ffaff35: Pushed
89ae5c4ee501: Layer already exists
devarm: digest: sha256:646b019918f8390943959f58663e32c0d1f4c6c41d8d2826bfe1923c9bd26e6f size: 946
/tmp/test   6s

This will not work on pi with the devarm tag:

Linux ubuntu 5.4.0-1012-raspi #12-Ubuntu SMP Wed May 27 04:08:35 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
➜  ~ docker pull securisec/test:devarm
devarm: Pulling from securisec/test
31603596830f: Already exists
a63d3a549d9f: Pull complete
2f70b035da12: Pull complete
Digest: sha256:646b019918f8390943959f58663e32c0d1f4c6c41d8d2826bfe1923c9bd26e6f
Status: Downloaded newer image for securisec/test:devarm
docker.io/securisec/test:devarm
➜  ~ docker run --rm  securisec/test:devarm
standard_init_linux.go:211: exec user process caused "exec format error"

• Run prune system:
• Now run buildx without building x64 first.
• Push to hub (see how size is now different)

• Download new image in pi
• Now it will work

➜  ~ docker pull securisec/test:devarm
devarm: Pulling from securisec/test
941f399634ec: Already exists
891542077819: Pull complete
389fdabbcebc: Pull complete
Digest: sha256:2bf709df743dbdb7b8cf67d4986d8c5c53c3d26dc04529549b115ed427e5a718
Status: Downloaded newer image for securisec/test:devarm
docker.io/securisec/test:devarm
➜  ~ docker run --rm  securisec/test:devarm
test

Code in test app:

package main

import "fmt"

func main() {
    fmt.Println("test")
}

Thoughts

• Even though I can specify different architectures to buildx, it should distinguish between layers when using buildx and build separately.
• The size of the two architectures should be different (and they are when building after prune), but if rebuilding arm without prune, it is the same size as x64

[tonistiigi]

This is expected. If you replace a single arch image with another single arch image in hub it replaces it. If you want multi-arch image that can run in both you need to specify both platforms in --platform or push to separate paths and merge later with buildx imagetools create

[securisec]

This is expected. If you replace a single arch image with another single arch image in hub it replaces it. If you want multi-arch image that can run in both you need to specify both platforms in --platform or push to separate paths and merge later with buildx imagetools create

I am not replacing the same tag on docker up. One tag is dev and the other is devarm. My x64 images are built as part of autobuild under the dev tag, while the arm images are pushed to hub using the devarm tag.

[tonistiigi]

You need to provide a reproducer we can try then to explain what is wrong there. In the steps you provided earlier:

docker buildx build --platform linux/arm64 -t mytag .
..
build a regular x64 image using docker build -t mytag .

[securisec]

Ah, I will update the OP indicating that i am using different tags.

[securisec]

@tonistiigi i updated the post with full step by step with output and Dockerfile

[tonistiigi]

Ok, the first "mistake" you are doing is that you are not using buildkit/buildx when you build your first image. The effect of that is that your base images, eg. golang:1.14.2-alpine will be pulled and stored in your docker images (x86 only). Now this means that when you are doing FROM golang:1.14.2-alpine with buildx Docker driver or DOCKER_BUILDKIT=1 docker build we see that there is a local image and never check the registry. This is to support the use cases when you need to override the base image with your local version.

To fix this case master has been updated in https://github.com/moby/moby/pull/40629 to also compare the platform of the local image and trigger a pull if there is a mismatch.

To avoid this, make sure your docker images doesn't contain specific versions of your base images that you don't want to use. If you are using buildkit/buildx the base images will not go to docker images when you just use them in a pull. You can also use --pull to make sure images are checked from the registry. In buildx you can also use the container driver, this keeps the state completely separate from docker daemon and maybe more importantly for you allows you to build your case to just a single multi-arch image with --platform linux/amd64,linux/arm64

[fahedouch]

I would add for @tonistiigi comment that you can force the pull of an arm image within you Dockerfile (without a need to check your driver images)

```
FROM --platform=linux/arm64 golang:1.14.2-alpine AS server
WORKDIR /server
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o test

FROM alpine:3.9.6
WORKDIR /app
COPY --from=server /server/test .
CMD ["/app/test"]
````

[tonistiigi]

@fahedouch This doesn't actually cover the local images case (pre moby/moby#40629). Before this PR if you have a local image and building without --pull, that image is always used. And because it is local it is already for a specific architecture.

[securisec]

I have started using --pull to make sure I dont have the issues anymore. @tonistiigi should I close this issue, or leave it open for future enhancement tracking?