Nixpkgs: Build fails with 8GB space on /tmp - No space left on device

[davidak]

Issue description

I'm trying to rebuild with staging branch.

[root@imac:~]$  df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        802M     0  802M   0% /dev
tmpfs           7.9G  2.1M  7.9G   1% /dev/shm
tmpfs           4.0G  7.0M  4.0G   1% /run
tmpfs           7.9G  360K  7.9G   1% /run/wrappers
/dev/sda4       458G  134G  300G  31% /
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs           7.9G   17M  7.9G   1% /tmp
/dev/sda1       197M  174M   24M  89% /boot
tmpfs           1.6G   88K  1.6G   1% /run/user/1000
tmpfs           1.6G  4.0K  1.6G   1% /run/user/78

But the rebuild fails:

[ 48%] Building HexagonGenDAGISel.inc...
  CXX      menuattributeiter.lo
varobj.c:2539:1: fatal error: error writing to /build/ccE65Is0.s: No space left on device
 }
 ^
/build/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp:243:1: fatal error: error writing to /build/ccUCp0u7.s: No space left on device
 }
 ^
compilation terminated.
compilation terminated.
make[2]: *** [Makefile:1622: varobj.o] Error 1
make[2]: Leaving directory '/build/gdb-8.2.1/gdb'
make[1]: *** [Makefile:8825: all-gdb] Error 2
make[1]: Leaving directory '/build/gdb-8.2.1'
make: *** [Makefile:850: all] Error 2
make[2]: *** [lib/ExecutionEngine/CMakeFiles/LLVMExecutionEngine.dir/build.make:89: lib/ExecutionEngine/CMakeFiles/LLVMExecutionEngine.dir/GDBRegistrationListener.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:5574: lib/ExecutionEngine/CMakeFiles/LLVMExecutionEngine.dir/all] Error 2
make: *** [Makefile:152: all] Error 2
builder for '/nix/store/l6zb0p9x0qh61c2z01vjnpbhv0p0q93l-gdb-8.2.1.drv' failed with exit code 2
note: build failure may have been caused by lack of free disk space
building '/nix/store/kgfvxx6g7vhbihb9wpz9j2ha5zjlywh1-gstreamer-0.10.36.drv'...
cannot build derivation '/nix/store/5717l1jaw5lgavps9h2gd3g6ppbw631s-rustc-1.31.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9wd6hx622jlg109crglkwcq6dzn7psh4-librsvg-2.44.11.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/j8h056pxlalrgxyyd6b2pmd8l74wv46h-gst-plugins-bad-1.14.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ma311sr4vz5bb9kn7ljd49mfwqzhdsjg-wine-3.0.4.drv': 1 dependencies couldn't be built
building '/nix/store/zkpdcw0yyn94hd92qz222j052r45jpay-gts-0.7.6.drv'...
cannot build derivation '/nix/store/wcckb16wd1sc3i41d761fqwd8dsv3f1l-playonlinux-4.3.4.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vnkwq8rkgwqpzgkfsr5kxfndw0hmkys6-system-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/gqd0jr4ljpszh8c63wcbk6bvwqx0r302-nixos-system-imac-19.03.git.319ebef.drv': 1 dependencies couldn't be built
error: build of '/nix/store/gqd0jr4ljpszh8c63wcbk6bvwqx0r302-nixos-system-imac-19.03.git.319ebef.drv' failed

Steps to reproduce

boot.tmpOnTmpfs = true;

rebuild on staging, building all the things

Workaround

mount -o remount,size=15G /tmp

Technical details

• system: "x86_64-linux"
• host os: Linux 4.19.16, NixOS, 19.03.git.8291376 (Koi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2
• channels(root): "nixos-18.09.1829.0396345b794, nixos-hardware, nixos-unstable-19.03pre164715.201d739b0ff"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[jtojnar]

Perhaps the issue is that /tmp gets filled? I thing that is where /build is mounted to.

[7c6f434c]

/build/llvm/lib/ExecutionEngine/GDBRegistrationListener.cpp:243:1: fatal error: error writing to /build/ccUCp0u7.s: No space left on device

/build is a path in build sandbox, and it is located on /tmp, and 8GiB not being enough for an LLVM build is not completely unimaginable.

[davidak]

@jtojnar good idea!

After starting the build it is already 27% filled!

[jtojnar]

Indeed

https://github.com/NixOS/nix/blob/800cd55ab74fffe9a1d212c2b8caf4024ebd98b6/src/libstore/build.cc#L1863

https://github.com/NixOS/nix/blob/522cebdef427bb8aad0f3ca140deff2e8bb88017/src/libutil/util.cc#L426

[memberbetty]

An optional attribute buildSpaceRequired with an example value of "16G" (compatible with GNU dd units) for packages that require a lot of space to build would help save wasted builds in the future.

[davidak]

Yes, it's actually /tmp that is filled up. It goes up to 95% and then the build fails.

It would be helpful if it fails directly with a meaningful error message like @memberbetty suggested.

tar: firefox-60.4.0/layout/reftests/w3c-css/received/css-writing-modes/support/swatch-olive.png: Cannot write: No space left on device
tar: firefox-60.4.0/layout/reftests/w3c-css/received/css-writing-modes/support/mplus-1p-regular.woff: Cannot write: No space left on device
builder for '/nix/store/6qj2wsgnpp85ip00mnmcp68bihx7asrg-spidermonkey-38.8.0.drv' failed with exit code 2
note: build failure may have been caused by lack of free disk space

[vcunat]

Meaningful... in the sense that nix should print it's about /tmp?

[davidak]

@vcunat something like "PACKAGE can't get built because 16GB space is needed in /tmp while only 7.9 GB is available"

[vcunat]

Relying on all packages to be correctly annotated by this won't be easy. BTW, firefox in particular also needs lots of RAM. On i686 it won't build anymore due to not fitting some processes into 32-bit address space.

[worldofpeace]

@davidak Are you using boot.tmpOnTmpfs = true;?

[davidak]

@worldofpeace yes.

I don't understand why the size of the tmpfs is 7.9G. I have 16 GB RAM.

And the systemd unit don't sets it:

[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=mode=1777,strictatime,nosuid,nodev

(Get path with systemctl status tmp.mount)

[vcunat]

If you don't specify anything, tmpfs gets 50% of RAM. You can add e.g. size=12G to options. (I use that live: mount -o remount,size=12G.)

[worldofpeace]

I feel like this now naturally drifts into having https://github.com/NixOS/nixpkgs/issues/23912

[davidak]

Relying on all packages to be correctly annotated by this won't be easy. BTW, firefox in particular also needs lots of RAM

@vcunat Yes, it will get outdated quickly. Maybe hydra can capture the usage before deleting the build directory and saves as statistics. Then we could update the packages automatically before every stable release.

I have disabled huge desktop programs like chromium, firefox and libreoffice for now, since i mostly want to test the pantheon desktop.

[vcunat]

Yes, theoretically, but I don't think it will be very reliable anyway. If you run two builds at once, for example (not a rare thing IMO).

[davidak]

@vcunat but they don't use the same directory, right?

Just du before deleting, should be quiet easy.

[davidak]

I installed NixOS on a thinkpad with 256 MB RAM once. It shouldn't start building firefox there. That would just be a waste of time and bad UX!

[vcunat]

I meant the detection if there's enough free space before the build starts.

[jtojnar]

By the way, to avoid having to rebuild the world when you want to use something from staging, it is often possible to checkout certain evaluation that was already build by hydra (one that has low number queued/grey jobs):

https://hydra.nixos.org/jobset/nixpkgs/staging

For example I can try the first one with only 257 queued jobs and not many failures

I make sure the pantheon changes are there:

https://github.com/NixOS/nixpkgs/blob/bd62ac6b88c6690294a6c257f9fe5dd8bcbb2b00/nixos/modules/services/x11/desktop-managers/pantheon.nix

and then I try to build from that commit. Thankfully, none of the failures are affecting us so Nix will just fetch the build derivations.

[davidak]

@jtojnar oh, thanks! i didn't know that staging is also build since there is no channel.

Do you know the workflow with staging and staging-next? What is merged into what and when? (especially when will staging get merged in master which is hopefully less broken?) I didn't found that documented.

[worldofpeace]

Do you know the workflow with staging and staging-next?
I didn't found that documented.

Best documentation we have currently is the RFC.

[vcunat]

Now there's also a habit opening a PR for each staging-next merge, so we get a thread to discuss stabilization for the particular iteration. Current one: https://github.com/NixOS/nixpkgs/pull/54306

[memberbetty]

@vcunat One can just add all the concurrent builds and compare that to the amount of space available. The error message then would be "you are trying to build package A(1GB) and B(16GB). Your available space is 8GB. This means that building is not possible. Please upgrade hardware or optimize the build.".

[mirh]

Why can't you just have some variable/setting to use a disk folder rather than tmpfs?
It seems stupid to kill building in these situations, as if somehow using the latter wasn't simply out of habit.

[vcunat]

The FS for /tmp is choice of your system, and you can configure it any way you want. I believe nix (typically daemon) respects the $TMPDIR variable, so you can even separate that from the rest of the system.

[mirh]

Right, it would make sense to have a free space check in either scenario.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[davidak]

This is still an issue.

[avanov]

Just got it reproduced with 8GB RAM allocated for my VirtualBox builder.