Nix: Single-user install should not use /run/user/<pid> for build directories.

[ambrop72]

I just performed a single-user install of Nix in Fedora 30 using the install script. Tried building something that extracts a big source file and got "No space left on device" errors even though there is plenty of space available on the root filesystem. It turns out that Nix is putting build directories into /run/user/<pid>, which is a tmpfs with only a few hundred MB available in my case.

I could work around this by setting TMPDIR=/tmp, but I think this behavior is inappropriate as a default.

[edolstra]

Well, /tmp is also a tmpfs on most distributions, so that wouldn't be any better.

[ambrop72]

You are right, on Fedora 30 /tmp is also tmpfs. However, it does not have a size limit:

[ambrop72]

Sorry, accidentally clicked close...
On Fedora 30 I see:

tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=605528k,mode=700,uid=1000,gid=1000)

So /tmp is a tmpfs with no size limit but /run/user/<pid> is a tmpfs with a 600MB limit. So using /tmp is better. Anyway I think the right choice then is /var/tmp.

[edolstra]

Yes, /var/tmp solves that issue, but unfortunately it's also persistent across reboots, which is typically not what you want for Nix's temporary files...

[ambrop72]

By the way, /run/user/pid gets used because XDG_RUNTIME_DIR is set to that. This is in fact undocumented in the Nix manual.

[cleverca22]

https://github.com/NixOS/nix/blob/7c6391ddc730519a632cc0ee526c94a04812d871/src/nix-build/nix-build.cc#L375-L389

[ambrop72]

I think that at least Nix should not be picking up XDG_RUNTIME_DIR but should end up using /tmp in common setups. Using /run/user/pid is the worst because of the size limit.

[edolstra]

No, XDG_RUNTIME_DIR is usually much better than /tmp because you don't have to worry about /tmp races.

[ambrop72]

The issue is really about the problem that in a default installation of Nix on a common distribution, builds that end up with more than 600MB in the build dir will fail. This seems like a big problem to me that should be fixed. Any ideas for making that case not fail by default?

[matthewbauer]

For reference I think https://github.com/NixOS/nix/pull/2728 fixes this

[catern]

Hmm, I didn't realize the impact of #2728 on single-user installs trying to do builds. I agree with @edolstra that sandboxed builds should try to avoid using /tmp if at all possible, and prioritizing XDG_RUNTIME_DIR in that scenario makes sense.

IMO the right fix here is to use nix-daemon instead of the single-user install, then you can configure TMPDIR/XDG_RUNTIME_DIR appropriately for nix-daemon so that it doesn't use /run/user for sandboxed builds.

In #2728 I just wanted to make nix-shell stop setting TMPDIR (it replaces TMPDIR with XDG_RUNTIME_DIR if TMPDIR is unset), which I think is correct because a build inside nix-shell already isn't sandboxed, and keeping these environment variables intact is more correct for the user's configuration.

[ambrop72]

IMO the right fix here is to use nix-daemon instead of the single-user install, then you can configure TMPDIR/XDG_RUNTIME_DIR appropriately for nix-daemon so that it doesn't use /run/user for sandboxed builds.

A single-user install is what a new user does when following the "Get Nix" button on https://nixos.org/nix/. Stumbling upon this problem will detract new users from further use of Nix. Once this issue is experienced, it can be quite hard, especially for a new user, to figure out what is happening, and how to fix it.

If we stop properly supporting single-user installs, the instructions on "Get Nix" should say to do a multi-user install (--daemon argument to the install script).

[ambrop72]

Sandboxing was mentioned, but I don't see how sandboxing has anything to do with single/multi-user or which temp directory we use. Actually as far as I can tell, sandboxing is enabled by default also for single-user installs if the kernel can do it (which is great).

[ambrop72]

Could someone explain what exactly is the problem with using /tmp? @edolstra mentioned races but I don't think there is any real problem, since one is supposed to create entries in /tmpusing O_CREAT|O_EXCL - if there was a race and something else created it, we detect it and can retry.

[TravisWhitaker]

This failure mode (/run/user/<pid> out of space) has roadblocked many new Nix users at my company; it'd be great to have better behavior here, but it's not obvious to me what the best behavior across non-NixOS distros should be.

[knuton]

For the record, the XDG base directory spec has this to say on XDG_RUNTIME_DIR:

Applications should use this directory for communication and synchronization purposes and should not place larger files in it, since it might reside in runtime memory and cannot necessarily be swapped out to disk.

[zimbatm]

Another approach would be to dedicate a folder inside of the store dir. Something like /nix/build/<drvHash>-<drvName>. We have a better guarantee that the store dir has space available, and we control the layout which allows us to cleanup old build directories more easily.

If you want an example of another software that does that:

[root@x1:/var/lib/docker]# ls -la
total 68
drwx--x--x 15 root root  4096 Aug 14 11:52 .
drwxr-xr-x 41 root root  4096 Aug 19 10:06 ..
drwx------  2 root root  4096 Mar  9 11:43 builder
drwx------  4 root root  4096 Mar  9 11:43 buildkit
drwx------  3 root root  4096 Mar  9 11:43 containerd
drwx------  2 root root  4096 Aug 14 11:52 containers
drwx------  3 root root  4096 Mar  9 11:43 image
drwxr-x---  3 root root  4096 Mar  9 11:43 network
drwx------ 35 root root 12288 Aug 14 11:52 overlay2
drwx------  4 root root  4096 Mar  9 11:43 plugins
drwx------  2 root root  4096 Aug 14 11:52 runtimes
drwx------  2 root root  4096 Mar  9 11:43 swarm
drwx------  2 root root  4096 Aug 14 11:52 tmp
drwx------  2 root root  4096 Mar  9 11:43 trust
drwx------  2 root root  4096 Mar  9 11:43 volumes

[fzakaria]

I'm also seeing this; curious what others mitigate this with ?

In nix-shell should I unset XDG_RUNTIME_DIR or forcibly set export TMPDIR=/tmp