Nixpkgs: nixos-rebuild boot fails when xorg-server build script tries to open directories under /var/tmp

[peterhoeg]

Issue description

nixos-rebuild boot fails to run due to this:

1. systemd will create directories under /var/tmp which no user has access to for services with PrivateTmp=true set.
2. https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/x11/xorg/overrides.nix#L462 replaces the "compiled" directory for the xorg-server package with a link to /var/tmp.
3. the build script will try to descend into sub directories and fail due to 1.

This is the error:

cannot open `/nix/store/gv307dkp3r5hzj6jmqq2nxlsj2l31i90-xorg-server-1.18.3/share/X11/xkb/compiled/[e-mail]-CgnwMR':
Permission denied at /nix/store/9g4wc31j7a2xp22xpgwr0qssfxahxdzl-builder.pl line 58.

If x/xkb needs to dynamically compile keymaps (I'm guessing that is what that $out/share/X11/xkb/compiled directory is for), shouldn't we instead give it its own directory under
/var/lib/xkb or similar instead?

Cc: @groxxda

Technical details

• NixOS: 17.03.git.014efa9M (Gorilla)
• Nix: nix-env (Nix) 1.11.4
• nixpkgs: 17.03.git.e09f343M

[groxxda]

Maybe relevant for #19629

@peterhoeg can you please confirm my assumptions

• this only happens without sandboxing
• you have no special settings for the xkb stuff

I'm not convinced this is a xkb specific problem, it may as well be a bug in builder.pl.
At least the path that it tries to access looks very suspicious..

[peterhoeg]

@groxxda

1. it happens _with_ and _without_ useSandbox = true;
2. it happens _with_ and _without_ setting xkbOptions

[groxxda]

@peterhoeg any idea why this failure does not show up on hydra?

[dezgeg]

Probably it happens only when systemPackages (or maybe something else that is being buildEnv'd) contains more than one package having a share/X11/xkb directory.

[dezgeg]

I wonder if that symlink is needed anymore? Such directory doesn't exist on my Arch Linux installation for instance (edit: nvm, they just override the location to /var/lib/xkb). The code which added that is pretty old: https://github.com/NixOS/nixpkgs/commit/9d156931631d1de6581793ce9b0ec30833597df3

[peterhoeg]

Now this is getting odd.

1. Everything was working for the longest time.
2. { nix.useSandbox = true; } all along except for
3. I started making some changes to the KDE packages, which caused massive rebuilds. I disabled useSandbox in the hope that I could speed up the compilation (that basically the previously compiled stuff wouldn't be thrown away every time)
4. My machine was now booted with a generation that was built with useSandbox disabled.

And this is where the problems started.

Checking /etc/nix/nix.conf it would show that useSandbox was disabled, but trying to build with (and without) useSandbox enabled would give the error I mentioned.

Stupid me had run nix-collect-garbage -d earlier as the drive was getting VERY full.

So in under to work around it, I did mount -o remount,rw /nix/store and edited /etc/nix/nix.conf to enable useSandbox but keep in mind that in my configuration it was still enabled all along and after a reboot nixos-rebuild works again.

Why would /etc/nix/nix.conf override? Is it because nix-daemon was already started without useSandbox so that takes priority?

[dezgeg]

Why would /etc/nix/nix.conf override? Is it because nix-daemon was already started without useSandbox so that takes priority?

Yes, if you modify nix.conf you generally need to restart nix-daemon.

BTW, a safer way to do that is to NOT remount /nix/store rw but instead, make /etc/nix/nix.conf not a symlink but a normal file.

[peterhoeg]

Let's say I boot the machine using a generation built with useSandbox = false.

If I then enable useSandbox, I would imagine that the new generation is built in a sandbox as I would otherwise have to do it twice because my first round would not be. Then when rebooting nix.conf would tell me that useSandbox is true but it wasn't true when the generation was built.

[joelmo]

I have the same problem. Everything under /var/tmp gets copied to /nix/store/...-xorg-server-1.18.4/share/X11/xkb/compiled.

Maybe it can be fixed by being more specific here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/x11/xorg/overrides.nix#L462

Say ln -s /var/tmp/systemd-private-* $out...?

[peterhoeg]

@joelmo, does it work with nix.useSandbox = true;?

@groxxda, any thoughts on using /var/lib/xkb or similar?

[stapelberg]

I ran into the same problem. Setting nix.useSandbox = true fixes the problem for me.

[vcunat]

@joelmo: does it really get copied for you? I see only a symlink in the code.

[joelmo]

Sorry /nix/store/...-xorg-server-1.18.4/share/X11/xkb/compiled is a symlink. I haven't tried useSandbox but that would probably solve my problem.

[dezgeg]

I think we can just do this:

diff --git a/pkgs/servers/x11/xorg/overrides.nix b/pkgs/servers/x11/xorg/overrides.nix
index 930132b..60a7354 100644
--- a/pkgs/servers/x11/xorg/overrides.nix
+++ b/pkgs/servers/x11/xorg/overrides.nix
@@ -509,7 +509,6 @@ in
         '';
         postInstall = ''
           rm -fr $out/share/X11/xkb/compiled
-          ln -s /var/tmp $out/share/X11/xkb/compiled

           cp -rT ${darwinOtherX}/bin $out/bin
           rm -f $out/bin/X

That should just fall back to /tmp/ as per the code:

 48         /*
 49          * If XKM_OUTPUT_DIR specifies a path without a leading slash, it is
 50          * relative to the top-level XKB configuration directory.
 51          * Making the server write to a subdirectory of that directory
 52          * requires some work in the general case (install procedure
 53          * has to create links to /var or somesuch on many machines),
 54          * so we just compile into /usr/tmp for now.
 55          */
 56 #ifndef XKM_OUTPUT_DIR
 57 #define XKM_OUTPUT_DIR  "compiled/"
 58 #endif
 59 
...
 73 
 74 static void
 75 OutputDirectory(char *outdir, size_t size)
 76 {
 77 #ifndef WIN32
 78     /* Can we write an xkm and then open it too? */
 79     if (access(XKM_OUTPUT_DIR, W_OK | X_OK) == 0 &&
 80         (strlen(XKM_OUTPUT_DIR) < size)) {
 81         (void) strcpy(outdir, XKM_OUTPUT_DIR);
 82     }
 83     else
 84 #else
 85     if (strlen(Win32TempDir()) + 1 < size) {
 86         (void) strcpy(outdir, Win32TempDir());
 87         (void) strcat(outdir, "\\");
 88     }
 89     else
 90 #endif
 91     if (strlen("/tmp/") < size) {
 92         (void) strcpy(outdir, "/tmp/");
 93     }
 94 }

Or would an explicit patch to that function be better?

(Arguably though, buildEnv should be fixed to not follow non-store symlinks.)

[vcunat]

Sounds OK to me.

[sorpaas]

Still the same error after applying patch from @dezgeg. Running 16.09. Any suggestions?

[joelmo]

Do someone know what functionality ln -s /var/tmp $out/share/X11/xkb/compiled have? I can remove that and don't notice anything different, running kde5.

[sorpaas]

Just fixed the problem (http://git.that.world/nixpkgs.git/commit/?h=infra&id=02d2655fb9e9c8220c761484f1b0c6861af376d3). Turns out there's 2 more ln-s to be removed.

[changlinli]

@sorpaas It looks like your fix works for me

[luispedro]

This is still failing for me. Does it need to be backported for 16.09?

[vcunat]

@luispedro: I believe no fix has been pushed to the official repo yet.

[aszlig]

Apart from the oddity that compiled is a symlink to /var/tmp and gets into the output path, I think we should also address this in buildEnv so that it doesn't try to dereference symlinks that are outside of the store.

[roxma]

Same problem here, I'm kind of scratching my head now.
Looks like useSandbox = true doesn't work for me.

[sheenobu]

@roxma I had to do useSandbox = true, then nixos-rebuild switch (without KDE or KDM enabled) , then check /etc/nix/nix.conf to ensure sandbox was enabled, then systemctl restart nix-daemon, finally you can enable KDE and re-run nixos-rebuild switch.

[roxma]

@sheenobu
Thanks, now It's working!

[joelmo]

Another way to go around this error is to remove everything in /var/tmp.

[gnidorah]

@vcunat Will it go into nixos-16.09 branch of nixpkgs-channels? Thanks.

[vcunat]

@gnidorah, I planned to wait a few days so it gets more testing.

[vcunat]

Picked to 16.09. It's a bit later than I anticipated, and moreover channels are stalled due to Hydra's current setup.