Buildx: Value too large for defined data type

[jdrouet]

I'm trying to build a Rust project for amd64, armv7 and armv8 locally, but when I do, I get the following error:

 > [linux/arm/v7 4/5] RUN cargo fetch:
#16 0.502     Updating crates.io index
#16 0.956 warning: spurious network error (2 tries remaining): could not read directory '/usr/local/cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
#16 1.378 warning: spurious network error (1 tries remaining): could not read directory '/usr/local/cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
#16 1.808 error: failed to get `actix-http` as a dependency of package `docker-bug-value-too-long v0.1.0 (/code)`
#16 1.808
#16 1.808 Caused by:
#16 1.809   failed to fetch `https://github.com/rust-lang/crates.io-index`
#16 1.809
#16 1.809 Caused by:
#16 1.809   could not read directory '/usr/local/cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c cargo fetch]: buildkit-runc did not terminate successfully
make: *** [Makefile:5: build] Error 1

I put a repository here for you to reproduce it.

I don't really know if it's buildx related or buildkit related (most probably buildkit). The image builds well for armv7 on an raspberry pi, but not from my linux machine.

[ro-kue]

Same here. I ran your example repo according to the instructions and it fails for me with the exact same error. I figured out that it seems to be a problem with 32bit systems on a 64bit host whilst doing research. Something about larger inodes so I guess cargo's first file system operation just blows up.

[mentos1386]

I have the same issue. Attaching my Dockerfile. I tried to build it for armv7 on my amd64 machine. Anyone knows of any workaround?

FROM rust:1 as build
WORKDIR /usr/src/starship
ARG VERSION=0.47.0
RUN git clone https://github.com/starship/starship.git . \
&&  git checkout tags/v$VERSION
RUN cargo build --release
RUN strip target/release/starship
FROM scratch AS binaries
COPY --from=build /usr/src/starship/target/$TARGETPLATFORM/release/starship /

[jdrouet]

You can take a look here. I do a fetch using the build platform arch and then copy it in the target platform.

[nicobo]

I've the exact same error but in my builder image : I've tried rust:buster, debian:latest, python:3, ubuntu... to no avail...

Any idea I'm blocked ? (Or is there a catch in @jdrouet 's solution that I've missed? )

I cannot use a JFS volume as shown in the following link because the error happens at build time :
https://itsze.ro/blog/2020/11/29/cross-compile-for-raspberry-pi-with-docker.html

[jdrouet]

@nicobo in my previous message I give this link in which I use multi stage builds to fix the problem ;)

[nicobo]

@jdrouet yes I'm using multistage like in your Dockerfile but the problem happens in the build stage so I don't even reach the step where I could COPY --from...

I'm not using --platform=$BUILDPLATFORM (I let docker select the right arch), does that make a difference ?

[jdrouet]

if you don't use --platform=$BUILDPLATFORM it will use the target platform. By specifying --platform=$BUILDPLATFORM you force it to build with your computer's platform and you won't have the bug from qemu.

[bbernhard]

@jdrouet Thanks for the example Dockerfile - worked like a charm!

Not really related to buildx, but does anyone know how to pass the BUILDPLATFORM to the Dockerfile when building the Dockerfile with docker-compose? Up to now, I've always used docker-compose build when I wanted to quickly build a docker image on my machine and only used buildx for building images for multiple architectures.

But with the --platform=$BUILDPLATFORM workaround from above, docker-compose build always fails with: failed to parse platform : "" is an invalid component of "": platform specifier component must match "^[A-Za-z0-9_-]+$": invalid argument.

I already tried: docker-compose build --build-arg "BUILDPLATFORM=linux/amd64" but unfortunately that doesn't work either - same error.

It's not a big deal for me, as I can use buildx also for my local builds, but I was just wondering if anyone has an idea?

edit: I figured it out. COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build works for me (just make sure that your docker-compose version is not too old)

[nicobo]

if you don't use --platform=$BUILDPLATFORM it will use the target platform. By specifying --platform=$BUILDPLATFORM you force it to build with your computer's platform and you won't have the bug from qemu.

Right, I managed to build it thanks to your example !
I'm not a Rust developer : I'm trying to compile someone else's Rust project ; so I didn't figured out that it was a Rust-specific solution.

As I understand it now (if useful to anyone) :

1. since the error happens when running cargo fetch, this step needs to be executed inside the working container (cargo vendor will do it)
2. then copy the generated/downloaded files (vendor, .cargo) into the target image
3. finally finish the build inside the target image in offline mode (cargo build ... --offline), using the previously cached files

I have "vendor" files that look like architecture-specific (e.g. winapi-i686-pc-windows-gnu) so I will see if it works at runtime when I can test a container.
I wonder then if Rust would simply allow us to fully cross-compile for all targets, then just copy the generated files into the final image?

My process is a little bit different since I first clone the code of an existing project from github, so I don't need to do cargo init. The final Dockerfile looks like this and it builds fine : https://gist.github.com/nicobo/06530acc4fd82497878d45a93348ae3c

Thanks @jdrouet !

[bbernhard]

@nicobo That's basically what I am doing now. Always building the Rust dependencies when building my docker image took too long, so precompiled them with cross and injected the right shared object into my docker build.

see https://github.com/bbernhard/signal-cli-rest-api/tree/master/ext/libraries/zkgroup

and https://github.com/bbernhard/signal-cli-rest-api/blob/master/Dockerfile#L18

[nicobo]

@bbernhard awesome this is _libzkgroup_ I needed to build ! I noticed your image on Docker Hub but I've missed the cross-compiled binaries... from now on I can directly use yours !

[bbernhard]

@bbernhard awesome this is _libzkgroup_ I needed to build ! I noticed your image on Docker Hub but I've missed the cross-compiled binaries... from now on I can directly use yours !

what a coincidence :grinning: - glad it's helpful to you! :)

[nicobo]

Hello.

I am now reaching another level since this error arises when building a Python program, which depends on cryptography, a library which in turn builds with Rust... See : pyca/cryptography#5771

Let me explain the Python way : on some architectures (here linux/arm/v7) there is no "Python Wheel" package for this dependency so the pip install command builds it from source... using Rust... triggering again a Value too large... error !

The problem is that I don't think I can alter the way this dependency builds when triggered by a pip install (the cargo build command is out of hand), and anyway I'm getting tired to implement patches making my Dockerfiles hard to maintain...

Any news ? Is there any link to an original issue on qemu or anything ?

I've disabled rust in the cryptography build for now but this trick will be removed from their next release...