Buildx: Where did the built multi-platform image go?

[Rory-Z]

I really like buildx, I want to use it in my code, but I have a issue.
I executed the docker buildx build --platform=linux/amd64,linux/386 -f . --output=type=image command and output the following information.

 => exporting to image                                                                                                                                                                                          0.1s
 => => exporting layers                                                                                                                                                                                         0.0s
 => => exporting manifest sha256:f70f46db5ab5126060072198f9fe4056240cf5ab9f0819a60bc141501d5b1198                                                                                                               0.0s
 => => exporting config sha256:af6012ceb069e31c852bb3e509eaab5cfdc1fca82d336e1a014cc7672989bcf6                                                                                                                 0.0s
 => => exporting manifest sha256:b71684386f12acf835cf00fe1c1de7be104535d6debca08cf7789430c1d53456                                                                                                               0.0s
 => => exporting config sha256:0847da7ba34f27312e9e16eb015a309b290322c5d99ee64d56a172a75c906423                                                                                                                 0.0s
 => => exporting manifest list sha256:4eb73fed7ba678c004b851cefcf2c8d9e5b60ce8bfceb3df09f32c08fbdd0296                                                                                                          0.0s

But I can't find my image. Where did it go?

[tonistiigi]

Add push=true to the output or use --output type=registry to push the image to registry during build to access it.

[Rory-Z]

@tonistiigi For some reason I don't want to push the image to the registry. Is there any other way?

[tonistiigi]

There are other outputs as well. https://github.com/docker/buildx#-o---outputpath-typetypekeyvalue I pointed you to registry because you are building a multi-platform image, therefore I assumed you need to distribute it to multiple machines.

[Rory-Z]

@tonistiigi I tried to use --output=tar, but it outputs a directory structure instead of a tarball. What should I do?

$ tree tar -L 2
tar
├── linux_386
│   ├── bin
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── media
│   ├── mnt
│   ├── opt
│   ├── proc
│   ├── root
│   ├── run
│   ├── sbin
│   ├── srv
│   ├── sys
│   ├── tmp
│   ├── usr
│   └── var
└── linux_amd64
    ├── bin
    ├── dev
    ├── etc
    ├── home
    ├── lib
    ├── media
    ├── mnt
    ├── opt
    ├── proc
    ├── root
    ├── run
    ├── sbin
    ├── srv
    ├── sys
    ├── tmp
    ├── usr
    └── var

[tonistiigi]

What way do you wish to access the image?

[Rory-Z]

I want to be able to access multi-platform built-in images directly in the docker images. If not, I want to be able to export the image to a file, similar to the work of docker save. I guess --output = tar is equivalent todocker build && docker save, is that the case?

[tonistiigi]

Docker does not support multi-platform images locally atm. Local image extracted in docker can only be for a single platform that the current node is based on. --output type=oci gives you the oci transport tarball with layers for all subimages.

[tonistiigi]

I guess --output = tar is equivalent to docker build && docker save, is that the case?

No, that would be --output type=docker (with the limitations listed above)

[Rory-Z]

I use --output=oci , but I don't see how it differs from --output=tar

$ tree oci/ -L 2
oci/
├── linux_386
│   ├── bin
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── media
│   ├── mnt
│   ├── opt
│   ├── proc
│   ├── root
│   ├── run
│   ├── sbin
│   ├── srv
│   ├── sys
│   ├── tmp
│   ├── usr
│   └── var
└── linux_amd64
    ├── bin
    ├── dev
    ├── etc
    ├── home
    ├── lib
    ├── media
    ├── mnt
    ├── opt
    ├── proc
    ├── root
    ├── run
    ├── sbin
    ├── srv
    ├── sys
    ├── tmp
    ├── usr
    └── var

[tonistiigi]

Post full commands of what you are running.

[Rory-Z]

docker buildx build --platform=linux/amd64,linux/386 -t emqx/emqx:test -f deploy/docker/Dockerfile . --output=oci

[tonistiigi]

--output type=oci https://github.com/docker/buildx#-o---outputpath-typetypekeyvalue

[wohali]

Hi @tonistiigi , I'm having the same issue, and may be trying to solve the same problem as @zhanghongtong .

Our goal is to export each of the separate built images locally to the docker daemon, and validate them locally (using https://github.com/multiarch/qemu-user-static and various test cases) before pushing them to the registry.

With --output type=oci I'm getting:

failed to solve: rpc error: code = Unknown desc = oci exporter cannot export named image

Is this possible at all without this kind of approach (given our Dockerfile is under a subdirectory named $VERSION):

for ARCH in amd64 arm64v8 ppc64le; do
  from="$(awk '$1 == toupper("FROM") { print $2 }' $VERSION/Dockerfile)"
  docker pull "$ARCH/$from"
  docker tag "$ARCH/$from" "$from"
  docker build -t apache/couchdb:$arch-$VERSION $VERSION
done

allowing access to each separate platform image as apache/couchdb:$arch-$version locally for validation, then assembling the manifest and pushing later?

Building all of the images at once, and bringing them over one at a time for testing would be acceptable, too.

[DannyBoyKN]

What would be the approach pushing the results of the docker buildx build (images + manifets) to a local registry, eg. Nexus ?
The --push argument allows only pushing to the Docker Hub.
It tried --output=type=registry,ref=localhost:5000 but that is not recognized, still requiring authentication at the Docker Hub:

failed to solve: rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed

Or is there some other solution using intermediate folders or archives ?

[tonistiigi]

localhost:5000 is not a valid image ref. it is probably interpreted as a hub image.

[DannyBoyKN]

You mean ref= exists for --output=type=registry ? This is not documented.
I was thinking of adding the registry destination URL, similar as if just giving the --push argument (shorthand for --output=type=registry, see documentation) which is using docker.io by default. localhost:5000 is the address of my local registry
The image ref. should be provided/added by the buildx build command.

[jl-massey]

@DannyBoyKN have you tried just tagging your image as if you were going to push to you r local repo? E.g.
docker buildx build -t localhost:5000/marchpkg:latest --platform linux/amd64,linux/ppc64le . --push

This is how docker knows the host. Admittedly, I tried to do this for a multi-arch image I'm trying to build, but I'm not getting far enough to push, yet. Good luck.

[DannyBoyKN]

Honestly, I don't remember if I tried this, I think I did ...

Unfortunately, I'm stuck, too!
The build process already fails when downloading because of the DNS. buildx uses by default google (8.8.8.8 and 8.8.4.4) which is not available behind my firewall. Giving the local DNS in the daemon.json file stops with 'connection refused'

I'll try further ...

[DannyBoyKN]

Well, just retried - was sure I did it already - with this Dockerfile:

FROM gcc:4.9
COPY main.c /usr/src/myapp/
WORKDIR /usr/src/myapp
RUN uname -m

and then

>$ docker buildx create --use --driver docker-container --name multiarch
...
>$  docker buildx build --platform linux/amd64,linux/arm64/v8,linux/arm/v7 --tag localhost:5000/multiarch:test --push  .

the error is

...
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request:
Head http://localhost:5000/v2/multiarch/blobs/sha256:39f6cc0761da5c1bc61d59c5cbe9188f22bc173d6f1038d6cccf1292f0b79594:
dial tcp localhost:5000: connect: connection refused

[tonistiigi]

If you are pushing to localhost from a container driver you need to use host networking for the container https://github.com/docker/buildx#--driver-opt-options . Custom dns can be set with buildkitd config file.

[barcus]

i need also to test my image and push them later

for arch in amd64 arm64 arm  ; do 
    docker buildx build \
    --platform $arch \
    --output type=docker \
    --tag me/myimage:${version}-${arch} \
    $version/
done

It works well ❤️

[DannyBoyKN]

@tonistiigi

Using --driver-opt network=host indeed works for pushing to localhost. But that's only for my local testing. The aim is to get them pushed to our Nexus registry, but still don't get the DNS configured.
With the "_buildkitd config file_" you mean in ~/.docker/config.json ?
Whatever DNS is set there it is correctly propagated into the conatiners /etc/resolv.conf. I did several DNS settings with and without network=host and didn't succeed ... don't have the exact error at hand at the moment ....

@barcus

That's interesting and tagging, pushing and running with localhost:5000/gcc-4.9:${arch} --push worked:

FROM gcc:4.9
RUN uname -m

$ docker run --rm localhost:5000/gcc-4.9:arm uname -a
Unable to find image 'localhost:5000/gcc-4.9:arm' locally
arm: Pulling from gcc-4.9
e925dd4ffa2a: Pull complete 
c9bfbf7dfc78: Pull complete 
015138dd660d: Pull complete 
d88b2b5023e5: Pull complete 
4d0d77a38079: Pull complete 
996bfab2b29c: Pull complete 
d27243b445c7: Pull complete 
2f949e025be6: Pull complete 
d55a5da9fec4: Pull complete 
3976cacabfa7: Pull complete 
Digest: sha256:b8dcfe0a3bbf2dbcb49a5117d8dee8fd412da31663a8c9be745eb6909bebf4d2
Status: Downloaded newer image for localhost:5000/gcc-4.9:arm
Linux c07606921642 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 armv7l GNU/Linux

[indraneelpatil]

@DannyBoyKN were you able to figure this out? I have the same problem which is a multiplatform build using buildx but I am not able to specify the --push flag because I am trying to push to a private nexus registry

This is my command :
docker -D buildx build --platform linux/arm64,linux/amd64 -t private.repo.com/nav_2_0:multi_support_image --push .

[DannyBoyKN]

Unfortunately not. I had not time so far to dig into howto provide the correct DNS information as @tonistiigi pointed out above.

[ballerburg9005]

Why is this issue closed? How can you do a multi-arch build with buildx now, and save the image, without pushing it to any registry?

[barcus]

@ballerburg9005 i use both :
1 - use buildx with --output "type=docker,dest=myimage.tar"
2 - use buildx with --output "type=oci,dest=myimage.tar"

for arch in amd64 arm64 arm  ; do 
    docker buildx build \
    --platform $arch \
    --output "type=docker,push=false,name=me/myimage:mytag-$arch,dest=myimage.tar" \
    $path_to_dockerfile/
done

Documentation can help as well :)

[ozbillwang]

Not sure why docker buildx bulid doesn't save a copy to local image as default. After run it, I don't see it locally.

So what's option to save the image as local image, then I can docker image |grep <image> directly?

[barcus]

@ozbillwang --output "type=docker,name=${app_name}:${tag}