Nixpkgs: grsecurity: Out-of-box experience

[NeQuissimus]

Issue description

After switching to the grsecurity kernel with desktop profile, not very many of my daily applications seem to work anymore.

Most importantly, I have to reboot and choose a prior generation to get off the kernel. nixos-rebuild switch will fail every time.

Chromium refuses because it cannot properly sandbox.

All of these issues may be user errors, I don't know. But @joachifm had asked me to open this as a bug as the OOB experience is quite poor.

Steps to reproduce

• Switch kernelPackages = pkgs.linuxPackages_latest; to kernelPackages = pkgs.linuxPackages_grsec_desktop_4_5; in configuration.nix, nixos-rebuild switch, reboot.
• nix-env -i <ANYTHING> or change the configuration (i.e. switch back to the normal "latest" kernel) and nixos-rebuild switch

Both will have similar output to this:

[Sat 16/05/07 14:53 UTC][pts/0][x86_64/linux-gnu/4.5.3-grsec][5.2]
<root@nixus:~>
zsh 1241 # nixos-rebuild switch
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/d28r2ck82j4f8wm999wwqfli41mzdpdv-etc-nixos.conf.drv
  /nix/store/z571d4zz6k7zl1ybz6mg37z3qznbgk6c-unit-systemd-sysctl.service.drv
  /nix/store/48y2vgy5c8jynq5v5cwxm7w47mkjwhbn-system-units.drv
  /nix/store/0hgxcb1vkylh6hxv6fbqyb8h2hhrpihr-etc.drv
  /nix/store/25yw44zlik8libs2azhiww18qhpqkq8q-nixos-system-nixus-16.09pre82794.e936f7d.drv
building path(s) ‘/nix/store/8zjwnq7j0w892g5djsn7asd55yixw95z-etc-nixos.conf’
error: while setting up the build environment: cannot unmount real root filesystem: Operation not permitted

The journal has more details:

May 07 14:54:09 nixus kernel: grsec: use of CAP_SYS_ADMIN in chroot denied for /nix/store/bg1h0brg33pzs8r72iv6cwiml8kv66s5-nix-1.11.2/bin/nix-daemon[nix-daemon:1292] uid/euid:0/0 gid/egid:0/0, parent /nix/store/bg1h0brg33pzs8r72iv6cwiml8k

Chromium prints this into the journal, just like Dropbox.

May 07 14:50:46 nixus kernel: grsec: denied resource overstep by requesting 26 for RLIMIT_NICE against limit 0 for /nix/store/788kcc4ygxs5qnggrapmprbjr59lpcza-chromium-49.0.2623.110/libexec/chromium/chromium[chromium:1105]

Technical details

• System: 16.09pre81686.f1675d9 (Flounder)
• Nix version: nix-env (Nix) 1.11.2
• Nixpkgs version: "16.09pre82794.e936f7d"

[joachifm]

Thank you! This is very useful.

[joachifm]

I have a fix for this queued up, but it significantly weakens chroot protections, so I need to think about this a bit more. I'm leaning towards introducing a grsecurity-nixos variant that includes nixos-rebuild support + the grsecurity path patch (which also weakens hardening compared to vanilla grsecurity).

[NeQuissimus]

Awesome, let me know if you need me to provide more info or test anything

[joachifm]

I've pushed a stop-gap change that ought to address the main part of the issue here.

I think the resource overstep message is related to auditing; my guess is that grsec isn't blocking anything here but simply reporting on it.

[NeQuissimus]

Chromium did not even start when these messages were reported. When I have a chance, I will try again and see if there were other messages in the journal.

[joachifm]

Chromium did not even start when these messages were reported

Ah, I see ... I've been running chromium on grsec for a while without problems (not on NixOS, though); will look into this further; could be missing PaX markings.

[joachifm]

To avoid wasted time: there's probably not much point in further testing until https://github.com/NixOS/nixpkgs/issues/15492 has been resolved

[joachifm]

The current grsec work is at https://github.com/joachifm/nixpkgs/tree/grsec-ng

[joachifm]

Now we're hopefully 2 mass rebuilds away from a working grsec desktop; once paxmarks have been fixed and I've had the opportunity to ensure that chromium, firefox et al actually work, I'll furnish my proposal for a new grsec interface.

[NeQuissimus]

awesome :) I will give this a test-spin as soon as I can

[joachifm]

Okay, so I've merged the new stuff. Chromium still fails, though ... will look more into that when I get the chance, pretty sure it's just a missing paxmark somewhere.

[NeQuissimus]

Awesome, I will give this a try next weekend.

[NeQuissimus]

So I tried this out, on the latest master code (as of Thursday).
I still cannot get off the grsec kernel without a reboot, or do any other nixos-rebuild switch for that matter

[Thu 16/06/16 22:20 UTC][pts/0][x86_64/linux-gnu/4.5.7-grsec][5.2]
<root@nixus:~>
zsh 17 # nixos-rebuild switch -I nixpkgs=/home/nequi/dev/nix/nixpkgs
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/0xjpwhs3c3rrl8cn9v3j2qvyajq6sqh2-sbt-0.13.11.drv
  /nix/store/5xgh0xq918gyr2b4jknv9ch4ap5wlayl-kernel-modules-shrunk.drv
  /nix/store/11pbp9yqdf0qz426zknxvbn4cvqxlram-stage-1-init.sh.drv
  /nix/store/i2hg4xjp6dfi58c7m9sgwy5kykihin7g-minecraft-2015.07.24.drv
  /nix/store/ln8b7jdb9v3ha5188cisnwi5vn07iv89-scala-2.11.8.drv
  /nix/store/mz2cvywsq6zl0rlwikafj0gwgzw8s3gs-nixos-version.drv
  /nix/store/95g4ngzhcvmrciv93r69c7d130vk0ny1-system-path.drv
  /nix/store/ayilv2j4an7lzy48mbbmfgkfdc1w2fck-dbus-conf.drv
  /nix/store/7l4fpqlc0pks43q85xy0y66d4dqiv1rm-unit-dbus.service.drv
  /nix/store/g0vbclmjl6drz5vwfl0vph0zvsk61bkl-unit-polkit.service.drv
  /nix/store/yscjvrny6bg0jha48d3hbg33h2635di7-unit-cpufreq.service.drv
  /nix/store/2b9kf6kpdy32j9rmly85ny2hwqaxs52i-system-units.drv
  /nix/store/g58k24wrnn5396sygy6yf2ymwjpd5a6m-issue.drv
  /nix/store/i6p00mav3i492pj2668f778mhkkrp13i-initrd.drv
  /nix/store/m2rvagh3hxxmdwr53m5j623vilfmgf0q-firmware.drv
  /nix/store/n7kn6lwc5h2n2dvw4nswa4k1w2d57ff8-etc-nixos.conf.drv
  /nix/store/pzg2l7w1c2a0aiaf3ajm4vvxxamaprhs-etc-os-release.drv
  /nix/store/klfimmqi7g2fnhqskf8qd6ddjk5mg4h5-etc.drv
  /nix/store/ykfcj1v6szxlvni3q27hcdv6x3xrc64m-nixos-system-nixus-16.09.git.f7ab8f8.drv
these paths will be fetched (74.05 MiB download, 82.32 MiB unpacked):
  /nix/store/gx06q77wmicpiv1y7rvmn79x6c1wn39d-scala-2.11.8.tgz
  /nix/store/ilqv4l931c8p9lbmfysmr5jb4d02572g-linux-4.6.2
  /nix/store/jxf9bwfq84v5lb1iafhcpyacsav85s2s-kernel-modules
fetching path ‘/nix/store/ilqv4l931c8p9lbmfysmr5jb4d02572g-linux-4.6.2’...
building path(s) ‘/nix/store/wy1454pf3mm51zy83x236hdn7724lbmp-etc-os-release’
killing process 1483
error: while setting up the build environment: cannot unmount real root filesystem: Operation not permitted

[joachifm]

Hm, that doesn't make too much sense, the grsecurity module disables the requisite chroot restrictions. Now, if you just set the kernel and nothing else, that'll fail, by design --- is that what you did?

[joachifm]

To do what the module does manually, you'd need to do

sysctl kernel.grsecurity.chroot_caps=0
sysctl kernel.grsecurity.chroot_deny_mount=0

and so on.

[NeQuissimus]

Ah, my bad, I must have buggered something up.

Setting the module to enabled and pulling in the kernel works just fine now.
This is awesome. Chromium is unhappy but Firefox works

[joachifm]

Great. Thank you for testing. I hope to sort out chromium eventually ...

[NeQuissimus]

Note that because Chromium has these sandbox issues with grsec, apps based on Chromium (i.e. Atom editor) will also not work.