Compose: Docker-Compose build stores a temporary file in /tmp and ignores Docker's TMPDIR

[maciejjaskowski]

While performing
docker-compose ... build I was getting consistently errors:

chematica@deb01:/opt/sas/chematica/zmq$ docker-compose -f docker-compose.common.yml -f docker-compose.local.yml build
Building rabbitmq
Traceback (most recent call last):
 File "<string>", line 3, in <module>
 File "compose/cli/main.py", line 62, in main
 File "compose/cli/main.py", line 114, in perform_command
 File "compose/cli/main.py", line 220, in build
 File "compose/project.py", line 300, in build
 File "compose/service.py", line 727, in build
 File "site-packages/docker/api/build.py", line 55, in build
 File "site-packages/docker/utils/utils.py", line 95, in tar
 File "tarfile.py", line 2014, in add
 File "tarfile.py", line 2043, in addfile
 File "tarfile.py", line 274, in copyfileobj
IOError: [Errno 28] No space left on device
docker-compose returned -1

On the root device indeed not much space was left but docker itself never complained for TMPDIR points to another device.

Turns out that docker-compose writes in my case a ~1GB tmp file to /tmp (ignoring TMPDIR variable) with filenames like:
tmpePRwRB
tmpjkvgDF
before proceeding with actual image building.

[shin-]

That's odd, we use the python tempfile functions that should account for TMPDIR values.

[maciejjaskowski]

Oh, now I see!

Let me rephrase my error report:

I was erronously assuming that TMPDIR setup for docker daemon (as run via systemd) also holds for docker compose.

I think it would be more intuitive this way, as docker-compose is part of docker ecosystem.

It is somewhat unintuitive that when running docker-compose some tmp files are stored in one place and others (created by docker daemon) somewhere else. It is reasonable though.

What do you think?

[shin-]

I understand there can be some confusion, but I'm not sure how docker-compose would be able to access the environment variables you used for the docker daemon? They're two different processes isolated from each other, that in many cases don't even live on the same machine.

[maciejjaskowski]

Yeah, I can totally see that :-)

How about a warning message explaining a possible confusion when "No space left on device" happens?

I think the main confusion came from the fact that I knew Docker creates temporary files using TMPDIR and somehow neglected the possibility that Docker Compose does that on it's own, too.
I kept investigating why does Docker Daemon behave differently when run from Docker Compose.

Not sure you can actually do sth about it. Maybe it's enough that now that I filed this report it will become google'able ;)

[Zechtitus]

running into this myself.

Is the fix to just export TMPDIR=/some_new_path_with_room and then redo the docker-compose build?

[maciejjaskowski]

Yes, that should work.
In our case we found somewhat more robust to remove the default /tmp directory and create a symlink pointing to a directory on another, spacious hard drive.

[michaelbaudino]

I'm hitting the same issue and I can confirm that using TMPDIR=$(pwd) docker-compose build does the trick (assuming you have space wherever you're running this command, indeed).

Also, it's worth noting that this huge temp file is actually a tarball of the current directory. So it's going to be as big as your current directory (which is a pain when, like me, you happen to have a 20GB database dump in the current directory :scream: and when your /tmp is an in-memory tmpfs, thus explicitely limited to a few GBs :scream::scream::scream:).

I hope it can help.

Any idea why is Docker Compose doing this? Docker doesn't (_e.g._ building the same directory with docker build . works perfectly).

[shin-]

@michaelbaudino Maybe docker does it in a more optimized way? Older versions also didn't always interpret the .dockerignore file correctly, so maybe your database dump was getting included when it wasn't supposed to be (this is fixed in 1.20.1). Either way, this is absolutely necessary as the context needs to be sent to the engine as a tarball for building.

[michaelbaudino]

@shin- Indeed, thanks for your clarification :tada:

I added *.dump and DockerCompose stopped adding this 20GB file to the tarball it's sending to the engine as context.

I no longer need to prepend every docker-compose build command with TMPDIR=$(pwd) \o/

[michaelbaudino]

@shin- Indeed, thanks for your clarification :tada:

I added *.dump and DockerCompose stopped adding this 20GB file to the tarball it's sending to the engine as context.

I no longer need to prepend every docker-compose build command with TMPDIR=$(pwd) \o/