Nixpkgs: RFC: Harden(ed) NixOS

[thoughtpolice]

(Spurred by some conversation in #7212, so I decided to write down my thoughts.)

Currently, NixOS and Nixpkgs have a lot of areas in which the security story can improve. While I think we're well positioned to take care of many of these, it's going to take some work, and others we'll need to reach consensus on. (The Good News is that many other people have gone down this road, so we have some good things to learn from)

As a rough starting point, I have identified roughly 5 major areas to look for future enhancements, with my personal suggestions on what we might want to do.

Binary hardening

As inspired by #7212, currently, NixOS doesn't default to enabling any hardening capabilities in GCC. There are an array of things we can do here to mitigate exploitability, that we can either enforce as a GCC spec file, or as part of NIX_CFLAGS/gcc-wrapper. Here are a few of them:

• Static compile-time enhancements via -D_FORTIFY_SOURCE=2 to improve protections for various built in string/buffer functions.
• Position-independent-executables to help defend against things like ROP-based code exploitation attacks via -pie -fPIC (PIEs are essentially just shared-objects-in-disguise much like transformers are robots-in-disguise).
• -fno-strict-overflow protects against the compiler optimizing away arithmetic overflow tests, which can happen when it gets particularly aggressive about undefined behavior.
• Stack protection via -fstack-protector*.
  
  
  
  • The default -fstack-protector protects functions with calls to things like alloca or ones that allocate more than 8 bytes of stack data. This buffer size can be controlled via --ssp-buffer-size, for example --ssp-buffer-size=1 triggers on _any_ stack allocations..
  
  
  • GCC 4.9 also supports -fstack-protector-strong, which implies more coverage than -fstack-protector by default.
  
  
  • There is also -fstack-protector-all which does this for _all_ functions. I am pretty sure this implies --ssp-buffer-size=1, but I'm not 100%.
  
  
• PLT/GOT protection: we need -z relro to mark dynamic relocations by the dynamic linker as read-only after ld.so resolves them. However, we also need -z now in order to force non-lazy resolution of relocations so the linker resolves them all at startup: otherwise they are performed on demand, and not marked as read-only until resolved (by which point an attacker could have used an arbitrary write to overwrite the relocation).

Suggestion: default all expressions to -fstack-protector-all --ssp-buffer-size=1 -D_FORTIFY_SOURCE=2 -pie -fPIC -z relro -z now -fno-strict-overflow as part of cc/ld. In other words, the whole gamut. For example, mkDerivation could support disableHardeningOptions = true; to disable options in specific expressions.

Implications: actually, fairly large in theory. These all in conjunction can have a significant penalty to things like startup time (due to resolving dynamic symbols up front), or execution speed. In particular, -fstack-protect-all is going to be costly (I wouldn't say beyond utility, but probably a good 20% speed loss at least). Also, PIE is going to really, really hurt performance on i686-linux, because PIE steals a register from the pathetic 32bit register set. Just PIE alone will have a significant impact on 32bit users, and -fstack-protect-all will add insult to injury.

Really, only benchmarking these things will tell us. I'd estimate the hit for these doesn't matter for a significant amount of software (common things like systemd, coreutils, anything that is setuid, etc). But a lot of things that are sort of nebulous middle grounds can be discussed on a maintainer-by-maintainer basis, I suppose (e.g. networking services are both performance _and_ security critical, so we may want something better here).

Binary determinism

This is issue #2281. I've had this on my queue to finish integrating for a while now; with my new machine this can hopefully be a reality soon enough... Unfortunately as we didn't make the cut for GSoC 2015, there won't be any sponsored work on this. There is still the issue of GCC PGO determinism too, which does not have a clear consensus, but the large majority of the work is elsewhere.

Suggestion: I get off my ass and merge this.

Service hardening

Currently, very few of our NixOS services try to take advantage of any security or isolation features that can be offered e.g. by systemd. Basic examples that are probably worthy of spreading around the tree:

• PrivateTmp, since a lot of services are probably fine with their own /tmp mount. (unless they do something weird like use it for cross service IPC, which is now handled by /run).
• DevicePolicy, which can restrict access to /dev (lots of them could get by with e.g. "closed")
• InaccessibleDirectories, ReadOnlyDirectories, and ReadWriteDirectories (for example, some would never need access to /home, while others like tarsnap could always restrict themselves to ReadOnlyDirectories=/ with a specific ReadWriteDirectories for the cache).
• NoNewPrivileges, which automatically sets prctl(PR_SET_NO_NEW_PRIVS) for daemons. Again, useful for services like tarsnap or logging services (but care is needed if e.g. things invoke setuid programs like ping).

Suggestion: We begin enhancing services with these and encouraging maintainers of modules to do the same. Honestly this can probably be done pretty easily and fairly incrementally by maintainers or any interested newcomers.

We should pay attention to upstream systemd units or units from other distributions here too, since they'll have figured out some of this, too.

Kernel enhancements

Kernel security enhancements mostly come from one thing and one thing only (IMO): grsecurity. The good news is that grsecurity support mostly works with my module, and of course it's possible to go out of band with your own custom linuxPackages.

One thing is that we don't currently offer prebuilt grsecurity packages. Hydra actually builds them, but there's no way for the module to automagically select the right one. This should be fixed. Futhermore, the binary builds and module need some clean up (e.g. RANDSTACK is actually completely useless for pre-built binaries since the random offset can be known by any attacker, so RANDSTACK should always imply a local kernel build.)

Also, my grsecurity module could use a bit of work. Unfortunately it's split up in several places due to needing build/module support, but I do think this could be cleaned up/refactored a bit.

There are other things to consider, too. For example, there are kernel patches floating around to do various other things; it would be really nice, for example, if we could have a patch to randomize the MAC address assigned by the kernel - this would be good for my laptop and could be controlled by a boot switch for the kernel.

Suggestion: Well, it mostly works I guess.

MAC/RBAC

NixOS currently doesn't offer any form of policy enforcement in the place of MAC systems. There are a lot to choose from, but it basically comes down to AppArmor vs SELinux. Right now there's nobody really supporting either, so in lieu of this, the support that _does_ exist is geared towards AppArmor. AppArmor isn't as expressive as SELinux, but it's a hell of a lot simpler and the policies are far easier to maintain. I think this is pretty important to get people to write policies.

The good news is the actual infrastructure is there: apparmor is packaged and works. There are even NixOS modules for it, but nothing really uses it. This is pretty easy though: we can begin clipping apparmor policies from upstream packages and places like Ubuntu.

Suggestion: We should just start writing policies, and enforce AppArmor by default on NixOS. This is the best way to ensure people keep using it, IMO.

Sidenote: gradm

The story is better for grsecurity users: you get gradm which is a totally badass RBAC system. The unfortunate news is that last time I tried it, gradm was buggy on my 3.4 kernel, and I didn't take the time to find a way to reconcile things like wanting NixOS modules to declare RBAC policies with the self-learning mode. In practice it may just be best to instead have services that activate/deactivate the learning mode via systemd, or just do nothing at all and expect users to enforce it themselves.

---

Small-form categorized TODO list

• Binary hardening
  
  
  
  • [ ] Enhance gcc to support transparent hardening.
  
  
  • Option 1: Add these things to NIX_CFLAGS, might work OK.
  
  
  • Option 2: Hack GCC. Hardened Gentoo does _not_ use spec files, but there is an overlay that patches GCC to automatically add the aforementioned options automatically. This does not imply -D_FORTIFY_SOURCE=2 as far as I can tell, but that one _can_ be added to NIX_CFLAGS most likely or gcc-wrapper.
  
  
  • Note: -D_FORTIFY_SOURCE=2 is, IIRC, only available at -O2 or above, so _that_ can be annoying if we don't deal with it in gcc-wrapper. Or we could just patch the stupid warnings it emits out of GCC, too.
  
  
  • [ ] Implement RELRO/NOW binding (-z relro & -z now), as well as -fno-strict-overflow.
  
  
  • [ ] Enable PIE. NOTE: Large 32bit performance implication.
  
  
  • [ ] Decide on stack protector settings. NOTE: Performance implications.
  
  
  • [ ] Default to hardened binaries and extend mkDerivation to allow opt-out.
  
  
• Binary determinism
  
  
  
  • [ ] Merge #2281 upstream and make sure system_tarball_pc remains deterministic.
  
  
  • [ ] Q: What to do about GCC PGO?
  
  
• Service hardening
  
  
  
  • [ ] Enhance services, make them harder, better, faster, stronger (and hopefully more isolated).
  
  
  • [ ] We should cross reference our services with upstream ones; many times an upstream will have a better one, or a distro will (such as ArchLinux).
  
  
• Kernel enhancements (grsec)
  
  
  
  • [ ] Disable RANDSTACK on pre-built images (useless).
  
  
  • [ ] Cleanup layout; split between nixpkgs, build-support, nixos. Can probably be refactored.
  
  
  • [ ] Make the grsecurity module serve prebuilt images.
  
  
• MAC/RBAC
  
  
  
  • [ ] Cut over to AppArmor 2.9 by default.
  
  
  • [ ] Enable by default on NixOS.
  
  
  • [ ] Enhance policy coverage for services.
  
  
  • [ ] (gradm) Make sure it actually works.
  
  
  • [ ] (gradm) Decide on learning mode support (or none if we decide to give up here).
  
  

CC: @wizeman @domenkozar @peti @edolstra

[vcunat]

From previous discussions, some of the flags (with negligible performance disadvantages) should be easily accepted to be in the defaults, perhaps even without an explicit option to disable them. Some regular distros also do have some of these for all packages IIRC.

Side note: I've seen that generating non-deterministic binaries is also used to improve security. I've seen that elsewhere, but wiki also mentions randomization of control flow https://en.wikipedia.org/wiki/Binary_hardening. (I'm no security guy, but I expect such techniques aren't available with unmodified toolchain, and non-determinism is a bit difficult with nix anyway.)

[copumpkin]

:+1: :+1: :+1:

@thoughtpolice in addition to all your excellent points, I've also wondered about reliable "auditability" of the system. The recent work on https://github.com/NixOS/nixpkgs/issues/7092 helps figure out what configuration.nix a particular system configuration came from, but I think we could do a lot better (to prevent people from tampering with persistent records) with e.g., ZFS and snapshots. Think what attackers who successfully break into a NixOS system might do to cover their tracks and what we might do to at least detect their evil deeds :smile:

@vcunat I think 99% of the security benefits you'd get from generating nondeterministic binaries are attained by ASLR on deterministic binaries. That is, unless your threat model involves an attacker using tooling that assumes binaries are laid out in a specific manner (and doesn't otherwise interpret ELF or your object format), what matters is runtime memory layout, not static binary layout. That's not to say that other aspects of binary hardening aren't applicable. If we have no source for e.g., Spotify, there might be instances in which the binary can be modified and made more secure, but that seems like a much less common case for a distro that is almost 100% source-based.

[wizeman]

FWIW, I am in general agreement with @thoughtpolice's suggestions.

I would add that it should be a no-brainer to allow the implementation of the security enhancements that have already been done by any of the mainstream, general-purpose distros that rely on a large community of contributors (e.g. Debian, Ubuntu, Arch Linux, non-hardened Gentoo, ...). The last time I looked, Ubuntu in particular implemented a significant number of these enhancements. These have already been battle-tested and shouldn't have a significant impact in maintaining most packages.

Also, I think it would be good to prioritize the enhancements @arno01 has been working on in #7212 and related PRs (although, I guess we should try applying them to most of NixOS instead of on a package-by-package basis, assuming that's what we want). This is obviously a priority to him and it can be frustrating and discouraging if we don't accept his contributions (or at least, help / guide him) for a long time. The same applies to any other already-implemented contributions / contributors.

[ttuegel]

In particular, -fstack-protect-all is going to be costly (I wouldn't say beyond utility, but probably a good 20% speed loss at least).

I assume this has already been thought of, but: if there will be that kind of penalty, we need to allow per-package opt-out. We package numerical computing libraries for which that kind of penalty is unacceptable, whatever the security implications. The same thing goes for PIE. I'm not asking for an unhardened system, just some variable for stdenv.mkDerivation to tell it not to do those things.

[thoughtpolice]

Some thoughts: Yes, I fully envision this will be controllable via a flag for mkDerivation to opt out of hardening, because there are legitimate cases where it's worth it.

The granularity of these options is up for debate; for example RELRO/BIND_NOW protection can probably _always_ be enforced. -D_FORTIFY_SOURCE=2 is already a default on many Linux distributions and will have very little impact on things like numerical code, so it can also always be enforced.

Disabling PIE and stack protection will be the biggest wins for most applications. Note that PIE is very efficient on x86_64 in comparison to i386; in fact, stack protection will be the biggest loss on 64bit, and PIE should have little effect. It's only 32bit where they both hurt.

Numerical notes: note that PIE is only going to hurt _applications_ - this is a very important distinction for numerical libraries, because PIE is basically just a dynamically shared object disguised as ELF. Meaning, if you already had a numerical library on 32bit machines that offered a .so, and linked against it, you already lost performance, and PIE won't change that very much for extant applications (assuming the hot spots are actually in numeric libraries.) What this means is that libraries are probably OK, but things that are inlined directly into the application source may suffer. So, I don't think PIE will really have a giant impact here, but I could be wrong.

However, it's not clear to me how high the demand for 32bit numerical libraries are anyway in this scenario anyway - almost anyone seriously using them is going to opt for a 64bit instruction set which will offer wider register sets and more features. Finally, things like dynamically linked copies of ffmpeg (a performance sensitive library) are _already_ suffering from the performance loss implied by shared objects on 32bit systems (but this is mitigated because they will hand roll their register allocation and assembly for core hot code spots, making this less relevant) - and considering ffmpeg has had over 1000 bugs brute forced out of it and you use it to watch videos with VLC, paying the hit of PIE on the main application + stack protection is probably worth it. I think the maintainer should decide ultimately, I guess.

I think where this is _really_ going to be a bigger impact is things like databases, which aren't hand written assembly, but a very large performance sensitive code base.

I suppose for this we'll have to extend the Hardened GCC patches if we decide to go that route, because then gcc-wrapper needs the option to turn some components off in the driver (probably the most robust solution, but more heavy weight than modifying gcc-wrapper - the Hardened Gentoo overlay patches simply unilaterally enable all protections, and I don't _think_ offer an option to turn it off, so everything on the HG overlay is fully protected I believe.)

Thanks for the feedback, let me know if you think this sounds all wrong. :)

[vcunat]

Actually, is it likely that someone would want to run 32-bit stuff on a security-sensitive machine? (Numerical-intensive stuff also doesn't sound like going well with the register limitations inherent to standard i686.)

[arno01]

I think we need something like smart-flexi-hardening. It should be relatively easy to enable hardening global-wise or per-package-wise including the levels of hardening, for example: L0 {disabled}, L1 {performance, e.g. fstack-protector}, L2 {trade off between L2 and L3, e.g. fstack-protector-strong}, L3 {full hardening, e.g. fstack-protector-all} ... where it should be relatively easy for the user to distinguish the impact (from all: performance, security and compatibility sides) between different hardening options.
This way it will be easy for anyone to switch it On/Off whilst knowing what are the implications. And by default it could be L1 with clear indication in the default /etc/nixos/configuration.nix

[Phreedom]

On Tuesday, April 07, 2015 14:10:20 Austin Seipp wrote:

Some thoughts: Yes, I fully envision this will be controllable via a flag
for mkDerivation to opt out of hardening, because there are legitimate
cases where it's worth it.
+1 for this. Implementing flags is a prequisite to finding out what the default
flag values should be and what packages are "special".

[edolstra]

Binary hardening: this doesn't sound desirable if the performance impact is that bad. Could be an option for network-facing services.

Service hardening: :+1:

Kernel enhancements: I'm reluctant to depend too much on non-standard kernel patches, since they can create a situation where we can't update our kernel to the latest upstream version because the patches no longer apply. E.g. the discussion at https://github.com/NixOS/nixpkgs/pull/6858 seems to imply that grsecurity does not support 3.18 kernels, meaning we can't use that (LTS) branch.

Enabling AppArmor by default: Sounds good.

[thoughtpolice]

I think we need something like smart-flexi-hardening. It should be relatively easy to enable hardening global-wise or per-package-wise including the levels of hardening, for example: L0 {disabled}, L1 {performance, e.g. fstack-protector}, L2 {trade off between L2 and L3, e.g. fstack-protector-strong}, L3 {full hardening, e.g. fstack-protector-all} ... where it should be relatively easy for the user to distinguish the impact (from all: performance, security and compatibility sides) between different hardening options.

The problem is that this does not scale nicely for users - they really want binary packages, and every single one of these different flags implies a complete rebuild of pretty much the entire transitive closure of your system environment, because it will imply hash changes all the way through the graph your system closure. Selecting appropriate defaults for users is extremely important so they get A) good improvements with B) minimal friction and pain. The current default is essentially equivalent to your L0, which I find unoptimal. :)

In general I'm OK with leaving judgement to maintainers of packages/services about what hardening steps are necessary, but I'm much more hesitant about leaving that decision up to users, because the trade offs are much less clear than they may seem.

Binary hardening: this doesn't sound desirable if the performance impact is that bad. Could be an option for network-facing services.

The impact is really only PIE/stack protection. The others are very cheap and increase security with few - if any - complications. Again, I think the correct thing to do is to have services that absolutely need the extra performance opt-out of these two features. Just looking at my system closure for example, there are a _lot_ of applications for which this would only increase security with few downsides - systemd, coreutils, tcpdump/libpcap, etc etc. In fact, I'd say network services are the ones that want to opt _out_, not in, precisely because they're performance sensitive and tend to have a large amount of security scrutiny anyway.

I also like to keep in mind _almost every other distribution enforces many of these enhancements anyway_. This performance impact may _seem_ bad, but it's a loss that a very high majority of distribution maintainers see as appropriate. In fact, PIE is the only truly controversial one.

An alternative would be to instead default on -fstack-protector, which will greatly reduce the impact of stack protection. In fact, I do believe most other distros actually just use -fstack-protector, precisely because the impact is much lower than the -all variant.

PIE is something special on its own, and perhaps worth not enabling by default on 32bit (which is a bit weird and inconsistent), but on 64bit, should absolutely be enforced IMO as it can significantly complicate code-reuse exploits and RIP-relative addressing is cheap in comparison to stealing a base register.

Kernel enhancements: I'm reluctant to depend too much on non-standard kernel patches, since they can create a situation where we can't update our kernel to the latest upstream version because the patches no longer apply. E.g. the discussion at #6858 seems to imply that grsecurity does not support 3.18 kernels, meaning we can't use that (LTS) branch.

Just to be clear: I do not ever plan on making NixOS require things like grsecurity* - which, on its own, would be a bigger discussion than this IMO - which is an extensive and massive patchset. (Something like a MAC randomization patch is, OTOH, about 20 lines of code), and it also slightly complicates the LTS story as I indicated in that other thread. The support we have now would need extensive polishing in any case.

• I would like it, however, if NixOS was a kickass distro for _using_ grsecurity with good support. You may be surprised at how much of a selling point this is to some people, on its own.

[copumpkin]

Anyone have thoughts on my ongoing audit point above? Being secure is fine and dandy, but it's also nice to have reasonable confidence that your security measures are actually in effect :smile:

[arno01]

@thoughtpolice :+1:
I think the best what we could do is to get (Binary && Service) to the security level of the current Debian's (feel free to read as Fedora, Ubuntu, pick any recent Linux distribution) security level.
Simple as that, noticing my nginx and postfix were both running without any protection (not even basic stack smashing protection), I, to be honest, got pretty scared. It is not impossible to get/find a 0-day exploit and all just gets screwed without having at least just mentioned basic and pretty painless protections... Seeing what I have seen, I personally go for the PIE as well.

We all know that buffer overflows (not only) are common in the C and C++ languages and every qualified security expert knows that each and every protection mechanism -- matters. Having just one single weak point can lead to a system compromise.

[joachifm]

Regarding AppArmor profiles: it's probably worth looking into re-using profiles provded by the apparmor-profiles package. In a lot of cases, it's simply a matter of patching the binary name.

[joachifm]

RBAC can be implemented in AppArmor via apparmor-pam, which would benefit everybody not just grsec users. AppArmor also has what looks to be a fairly comprehensive learning mode (all of that is currently broken on NixOS, though). All-in-all, it seems most cost-effective to focus on AppArmor for RBAC and automatic learning and rely on grsec mostly for PaX et al.

[thoughtpolice]

As an update, I have a branch (not yet published) which:

• Begins some hardening for services via systemd, and also increases some systemd dependencies so we can do things like have apparmor and libseccomp support.
• Dramatically simplifies the kernel maintenance and drops some old kernels and patches. By the time this is merged, we will probably only carry A) the 4.0 mainline kernel and B) the 3.18 stable kernel. We will continuously update mainline. We will not promote stable until the next stable kernel is promoted by kernel.org, and even then, probably not until a bit later. This means there will be no change to the window in which linuxPackages (the default) is promoted, mostly. It only means we won't carry every stable version ever. Which really isn't that unusual, when you think about it.

We need to really have a bigger discussion about this, but maintaining like 20 kernel versions with a bunch of random patches (MIPS patches? We don't support MIPS, let's be real) is kind of ridiculous. I feel like rather than making Hydra do the work (it's already sick enough) we should punt that off to maintainers for things which aren't broadly useful. If you're a NixOS developer, it's reasonable to assume you can carry your own pet kernel patches out of tree. If you're a user, you probably just want the latest stable kernels and don't care much. If you _do_ want a random version - you're in luck! NixOS makes it easier to use a custom kernel than anyone else. This seems harsh but will seriously drop like a few hundred packages probably, and make it easier to keep track of the kernel packages we have. Also, the writing is on the wall with this one: once we move to the glorious kdbus based future, systemd isn't going to keep support for non-kdbus kernels on life support very long. That means at best, sometime in the next 6-8 months (by the 5.10 release, say, systemd 225), we're going to have to kill all old -stable packages for this anyway.

To be completely clear and fair, I'm even going to drop the grsecurity -stable branch as part of this (as I've removed all pre-3.18 kernels).

Alternatively, I propose we move all non-up-to-date stable and mainline builds to a staging subset of the kernel packages, and turn their Hydra builds off for now.

• Moves to AppArmor 2.9 by default and enables it by default (this might more work, but I would suggest we just say _it's always enabled_)
• Total overhaul of a new grsecurity interface (see below).
• Various other tweaks (updates, dependency fixes, etc)

Here's an example of an "Improved" grsecurity description using NixOps (NB: not representative of final overhaul):

with import <nixpkgs> {};
with pkgs.lib;

let
  grsec = (import <nixpkgs/pkgs/build-support/grsecurity> {}).customGrsecKernelPackages
    { kernel = rec
        { version       = "3.19.3";
          localver      = "-grsec";
          url           = "mirror://kernel/linux/kernel/v3.x/linux-${version}.tar.xz";
          sha256        = "0nis1r9fg562ysirzlyvfxvirpcfhxhhpfv3s13ccz20qiqiy46f";
        };

      patches =
        [ { url    = "https://grsecurity.net/test/grsecurity-3.1-3.19.3-201504021826.patch";
            sha256 = "0r3gsha4x9bkzg9n4rcwzi9f3hkbqrf8yga1dd83kyd10fns4lzm";
          }
        ];
    };

  node = { config, pkgs, ... }:
    { deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 1024;
      deployment.virtualbox.headless = true;

      security.grsecurity.enable = true;
      security.grsecurity.kernelPackages = grsec.kernelPackages;
    };
in
{
  network.description = "grsecurity tests";
  grsec-test = node;
}

Some points:

• This allows custom base kernels and extra patches. For example, @joachifm uses linux-libre with grsecurity, and I want to use grsecurity plus extra patches (to randomize my MAC addresses and TCP timestamps).
• The need for localversion is kind of weird right now, but occasionally you want to use custom patches with different localversion or EXTRAVERSIONS (like grsecurity itself, or the -ck patchset), so localversion and modDirVersion are exposed for full control to the end user with some sane defaults (I think).
• Multiple patches can be applied, for example, using BFQ I/O scheduler patches on top of grsecurity.
• This basically inverts the 'package build' support to instead take a record and instead produce a kernel build directly, rather than having all-packages.nix do it. This means all-packages.nix can easily create its own custom builds with a lot better abstraction, and then a user could say:

  node = { config, pkgs, ... }:
    { deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 1024;
      deployment.virtualbox.headless = true;

      security.grsecurity.enable = true;
      security.grsecurity.kernelPackages = pkgs.linuxPackagesGrsecDesktop;
    };

and still get proper service, udev rules, etc.

I still need to do some extra stuff. In particular this current interface doesn't offer any help in configuring kernel parameters or setting configurations, so it's definitely not merge ready. Also, I have only just begun adding AppArmor and systemd policies, so I'll probably bundle/test more of them, too.

Please let me know what you think. I'll open a PR later this week.

[thoughtpolice]

I should mention I would like to get all purely-NixOS level work out of the way first, because while it may imply some changes to a few things, it does _not_ imply an ungodly rebuild of all of stdenv. That can come after this has landed.

[copumpkin]

I was also thinking that we could probably also pull off W^X at the filesystem level.

[arno01]

Speaking on security "out-of-a-box", right now I am building a standard (non-hardened) LFS where I notice that systemd 219 by default enables all common protections like Canary (SSP with the buffer-size=4), PIE, Full RELRO and FORTIFY_SOURCE:

        CFLAGS:                   -pipe -Wall -Wextra -Wno-inline -Wundef -Wformat=2 -Wformat-security -Wformat-nonliteral -Wlogical-op -Wsign-compare -Wmissing-include-dirs -Wold-style-definition -Wpointer-arith -Winit-self -Wdeclaration-after-statement -Wfloat-equal -Wsuggest-attribute=noreturn -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-declarations -Wmissing-noreturn -Wshadow -Wendif-labels -Wstrict-aliasing=2 -Wwrite-strings -Wno-long-long -Wno-overlength-strings -Wno-unused-parameter -Wno-missing-field-initializers -Wno-unused-result -Werror=overflow -Wdate-time -Wnested-externs -ffast-math -fno-common -fdiagnostics-show-option -fno-strict-aliasing -fvisibility=hidden -ffunction-sections -fdata-sections -fstack-protector -fstack-protector-strong -fPIE --param=ssp-buffer-size=4 -ffat-lto-objects   -g -O2
        CPPFLAGS:                 -Wp,-D_FORTIFY_SOURCE=2
        LDFLAGS:                  -Wl,--as-needed -Wl,--no-undefined -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now -pie

[ http://www.linuxfromscratch.org/lfs/view/stable-systemd/chapter06/systemd.html ]

[thoughtpolice]

+1 to that! :)

Also, excellent news: GCC 5.1 (released today) actually included an optimization I missed that's kind of a big deal: reuse of hard PIC registers. This basically means that on 32bit systems, the PIC register will be locally reusable when profitable, rather than 'fixed' as stolen (meaning it could never be used by the register allocator), which should be a significant boost to 32bit shared libraries, and as a bonus, means it will not hurt as much to enable full PIE on 32bit binaries, compared to 64bit.

So basically we should also upgrade to GCC 5.1 when possible...

[thoughtpolice]

I have pushed a preliminary branch with some work I did a few weeks ago - nixos/hardened - but it still needs work, but it's a start, and mostly touches the NixOS-specific stuff before touching stdenv etc.

There are some obviously wrong commits in here, looking back on it, so I'll review them.

If anyone wants, they're free to make PRs to this branch - just tag the pr with security and mark me as the owner to merge it.

[thoughtpolice]

@joachifm The latest commit on that branch is probably of particular interest to you - in fact I designed it partially to subsume your usecase of a custom linux-libre build with an older grsecurity, so please let me know what you think.

[thoughtpolice]

Oh, and @wkennington, you should probably review that branch too - in particular 9bee9c3e0e964426271e99ec45059b38a0bf21e1 which contains a novel describing the changes. Let me know what you think.

[copumpkin]

@thoughtpolice might make sense to put the commits in a PR, since it's easier to accumulate comments in that UI than this one

[joachifm]

@thoughtpolice looks good to me. I hope to test this later and provide some actually useful feedback if I can.

[wkennington]

Lgtm

On Mon, May 25, 2015, 13:15 joachifm [email protected] wrote:

@thoughtpolice https://github.com/thoughtpolice looks good to me. I
hope to test this later and provide some actually useful feedback if I can.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/7220#issuecomment-105305360.

[copumpkin]

Has anyone considered lumping IMA support into this?

[fare]

@arno01 It doesn't make sense for _users_ to change security based on a flag. It should be the package maintainers who each set the security parameters for their package, with easily available defaults. And if some people disagree with some package's security settings... that's what forking / building a separate package is for.

Note that some programs just aren't amenable to the usual hardening techniques. For instance, some Lisp implementations expect a fixed layout without address randomization to dump and restore memory images, while some speed junkie applications include assembly code not compatible with hardened ABIs, etc. The maintainers of these packages, not the users, are the ones who know what to do, and thus the ones to be empowered with making the right decision.

[fare]

To raise the bar for security, see also what Qubes does: https://www.qubes-os.org/
Basically, running every application in its own isolated domain (they use Xen domains rather than Linux containers), with very restricted communication with other domains, in addition to usual capability restrictions.

[arno01]

@fare yes and no.

Yes - because I agree, it is up to the maintainers to provide this option.
No - because not all the maintainers would be eager to do this and here is exactly where Nix comes to the scene as it gives a really nice and simple way of enabling extra security for every user.

Of course this can break the build, slow the app. down, and bring other side effects. However, isn't it what UNIX-way philosophy is about? To bring people freedom of choice! ;-) (and such an option, even if it will bring to some problems/side effects => will help to identify these problems making at least one step closer to fix these problems.. otherwise how would it be possible to fix the problems without knowing what are the problems?!)

Why to wait until the maintainers will add an option to harden the packages? Why not to leverage Nix with its nice and easy options/flags for that? And then, I have no doubt the maintainers will be eager to see the results, coop to help fixing the side effects, etc...

Why to sit and wait? Security does not wait :-)

Just to bring up an example: Before all these "hardened-talks", NixOS had most of important Net-Faced services (postfix, nginx, ...) not hardened at all ! And me, as an "advanced-user" if I can say, noticed that and got scared. This is why it is not a really good idea to wait for the maintainers.

[fare]

@arno01 Changing the defaults so that packages have to opt out of security and nothing needs to be done to opt in is good. Making it generally easy for maintainers to pick the setting they need is good. But adding more knobs only decreases security and makes auditing harder. Also, it's all a one big git repo; if you're an advanced user, fork the repo, and edit the packages you think require a different setting.

[arno01]

@fare enabling well-known and widely-used security flags != adding more knobs. Not using these security features == losing the knobs.

Regards the fork, this is what I have done, shared with my modifications as they are working perfectly in my environment.

[8573]

I would be eager for this project to come to fruition; is there anything that I, as an inexperienced NixOS user, could do to help?

[copumpkin]

Has anyone experimented with passing in custom NIX_CFLAGS_COMPILE to the existing nixpkgs? I've been overriding the stdenv to pass in -pie -fPIC and am running into assorted build issues. Was wondering if there was a known robust way to do that.

[copumpkin]

How do we take this project forward? Does anyone want to or have the time to step up to lead it? Is the only thing missing an IRC channel? A mailing list? It seems like a lot of us want a hardened Nix(OS), and there are quite a few pieces of low-hanging fruit that we could start tackling if we broke this project out into smaller chunks.

I'm feeling inspired by the project/wiki page layout at https://wiki.gentoo.org/wiki/Project:Hardened :smile:

[domenkozar]

Fedora 20 builds all packages with -fstack-protector-strong

[copumpkin]

It seems like on the compiler side of things, we could break things into a set of "nearly no regrets" compiler flags, along with a set of "hardcore" ones that can impose a non-negligible performance cost. We'd default to the no regrets set, but also make the hardened set available. We could build a subset of the latter on Hydra so it isn't too much of a penalty to switch to.

[mbakke]

The performance overhead _is_ negligible, at least from an end user perspective, and pretty much every major distro uses it. I'd say enable by default and add a build.disableStackProtection passing -fno-stack-protector for HPC users and troublesome packages.

[copumpkin]

@mbakke yeah, I'm just wondering if there are other no-regrets hardening flags we can lump in with that one to minimize the number of mass rebuilds :smile:

[vcunat]

There are typically several mass rebuilds on staging every month, so feel free to file incremental PRs against it. It would also be good to test-build some significant/important portion of nixpkgs to verify the flags won't cause a mass breakage (perhaps we might have a separate jobset for that PR).

[copumpkin]

Okay, you've convinced me :+1:

Who has time to submit the PR? I can't right now, but if nobody does it today, I'll send one in this evening (US Eastern)

[globin]

You might want to look at https://github.com/mayflower/nixpkgs/commit/eb19810219ad4bc637b1d7325c173f7dcc0aa98f
/cc @fpletz

[copumpkin]

@globin @fpletz oh, the mayflower stuff looks great! Is there an upcoming PR for it?

[fpletz]

@copumpkin There will be a PR when I've sorted out all the build errors. :) Also, the naming of the hardening options needs to be improved. I'm planning to work on some more hardening stuff.

[copumpkin]

@fpletz very nice! it might be worth bugging @edolstra to get a branch on this repo and a hydra jobset (as @vcunat suggested) for it, so multiple people (even outside mayflower) can work on the hardening branch.

[globin]

We have a minimal jobset for it in our Hydra:
https://hydra.mayflower.de/jobset/nixos/hydra-jobs-hardened
We'll probably start a full run for it when more has been worked out, if someone is interested on working just give a shout and we can work something out there for now, but definitely +1 for a branch here, too.
Mind the branch currently also uses libressl 2.3 as default, this causes some things to break that we don't use

[mbakke]

@copumpkin Another low-hanging fruit is -D_FORTIFY_SOURCE=2. Both should be safe to enable globally with little to no breakage. -fPIC and -Wl,-z,relro are also default in Debian/Fedora AFAIK, but may break some packages (possibly silently/runtime), so introduce them separately to prevent hair loss.. :)

(not a C developer, but sysadmin and former Gentoo user..)

[thoughtpolice]

This looks great! Thanks for getting back up to speed on this. It's helpful to know other people are willing to put elbow grease to it. :)

Few points:

• The naming will need improvements, yes - I hate bikeshedding, but generally we go for camelCase I think. IMO, I say rather than noHardening_XXX, you have hardenWithXXX, hardenedXXX instead - so just invert the naming a bit, which I think flows a little better.
• I wonder about using NIX_CFLAGS still, because thinking about it more, it might introduce warnings. If you pass -z relro to GCC when it's not linking does it warn for example? Some of these are unavoidable, and this isn't a big deal, I was just wondering if you potentially ran into anything like this.
• I'd suggest that, regardless of what position on defaults we take, we continue to audit the package list and curate it a bit while we're at it if we have time. Certain packages can definitely be hardened even further beyond whatever defaults we choose, and it may be a good idea to do that. For example, I'd willingly trade some performance to know that OpenVPN (with its jillions of lines of code) has extra protections or something like that. Ultimately we'd have to talk to maintainers, but yes.

Probably more off the top of my head. I can also push any preliminary work to the nixos/hardened branch over time to test things as we merge it into staging.

[vcunat]

The naming will need improvements

Personally, I (also) prefer positively named variables with false as default in such cases.

I wonder about using NIX_CFLAGS still, because thinking about it more, it might introduce warnings.

We also have $NIX_CFLAGS_COMPILE.

[nkaretnikov]

Not directly related, but still: has anyone looked into building a NixOS template for Qubes? Not terribly familiar with the project, but it looks interesting. Are there any technical difficulties, or is it just the lack of interest?

[vcunat]

@nkaretnikov: there were some mentions of Qubes not long ago, e.g. from @mathnerd314 and @ts468.

[Mathnerd314]

Qubes has their own homegrown build system; I doubt that it's technically difficult to make a NixOS template, but it would be tedious. There's someone doing an Arch Linux port: https://www.qubes-os.org/doc/building-archlinux-template/.

[nkaretnikov]

@Mathnerd314 There's also https://www.qubes-os.org/doc/building-non-fedora-template/ I think I might try following this guide sometime soon (in a month or so). Again, let me know if someone is working on this, so we don't have to step on each other's toes.

[ts468]

@nkaretnikov

About Qubes, I think it would be interesting to bring the core functions of Qubes into NixOS, instead of just running NixOS in Qubes. NixOS already has a very nice way to define nested containers or virtual machines, all within one configuration file. And by sharing the nix-store, or at least part of it, one would get very light-weight, isolated VMs. There is basic support for Xen in NixOS already, all it would probably take is to port the message passing system that Qubes uses, plus some additional stuff for, e.g., the X system. The NixOS modules system would absorb all necessary changes quite nicely. The biggest problem I see is that Qubes seems to keep a lot of their patches locally within their build system. But maybe one could convince the people behind Qubes that it would be beneficial for them to switch to NixOS themselves ;). They would have my support!

[copumpkin]

Also related to the general NixOS security, I added a module to configure the kernel audit mechanism, if anyone wants to tinker: https://github.com/NixOS/nixpkgs/commit/63bfe20b7253fa579ca1c35d07d1d790475f74c5

[vcunat]

Notification: grsecurity people should know it's rather outdated, at least on master 4824f73cb3cd.

[copumpkin]

@vcunat yeah, I have a commit locally that updates it. I'll push it up at some point.

[copumpkin]

For anyone following this, see #12895

[domenkozar]

Bumping milestone, but really looking forward to this work being merged.

[vizanto]

Just noticed that in packages like nginx, hardening and compiler being GCC is checked.
However, on SmartOS this package now breaks because LD is assumed to be GNU LD when compiler is GCC...

Is there another flag to check for GNU LD?

[globin]

@vizanto you might want to look at and test #12895

[Nadrieril]

Not sure if this issue is still alive. Still, this is probably relevant: https://github.com/NixOS/nixpkgs/issues/14645 (The service interface should abide by the principle of least privilege).

[CMCDragonkai]

Just looking at this the first time. But if AppArmor is used by default, does this make it more difficult to use SELinux with NixOS in the future? Also has anybody considered Capsicum capabilities? See:

• https://www.youtube.com/watch?v=raNx9L4VH2k
• http://capsicum-linux.org/

[domenkozar]

I'm removing the milestone here as the description of the PR is fairly outdated. Is there anyone willing to take this meta-issue forward?

It's fairly done and spread out in issues like https://github.com/NixOS/nixpkgs/issues/14645 and https://github.com/NixOS/nixpkgs/pull/12895.

It's important to have an overview, but it doesn't help if it's outdated information since a year, that will mislead bycomers.

If there's noone willing to clean this up, I'll just close it in a few weeks.

[vcunat]

Let's close. Better re-formulate in a new ticket, if you (still) prefer to have this kind of general one.

[fare]

Please link the new issue(s) here...

[spacekitteh]

https://github.com/NixOS/nixpkgs/issues/18176

[aneeshusa]

See #14392 for another useful kernel hardening feature, the Yama LSM, which I'd like to get in before 16.09.

[sorokin]

Hello. I have updated to NixOS 16.09 and I have difficulty using GCC in it.

GCC inserts stack protector checks into my program. I can not disable it (there is no option -fno-stack-protector-strong provided by GCC).

I don't mind if packages will be built with hardening by default. But is it a bit inconvenient that the default gcc command enables hardening with no clear way to disable it.

[joachifm]

@sorokin a temporary workaround is to usehardeningDisable="stack-protector" cc foo.c, see https://github.com/NixOS/nixpkgs/issues/18995.

[sorokin]

@joachifm Thank you. I'll try to work around it this way.

[coverme76]

This should be re-opened!