Nixpkgs: security concerns of nix store paths in grub.cfg

[volth]

Issue description

Even if LUKS encrypted volumes is used, unencrypted /boot/grub/grub.cfg has nix store paths:

menuentry "NixOS - Default" {
search --set=drive1 --fs-uuid 11111111-2222-3333-4444-555555555555
  linux ($drive1)//kernels/11111111111111111111111111111111-linux-4.9.130-bzImage systemConfig=/nix/store/33333333333333333333333333333333-nixos-system-x220-19.03.1699 init=/nix/store/33333333333333333333333333333333-nixos-system-x220aj-19.03.1699 boot.shell_on_fail noibrs noibpb nopti noretpoline nospectre_v2 loglevel=4
  initrd ($drive1)//kernels/22222222222222222222222222222222-initrd-initrd
}

Knowing the hash of nix closure (33333333333333333333333333333333 in this example) allows the attacker to query nix caches (http-servers like cache.nixos.org or Cachix; private ones can be found using Shodan.io) and download the whole closure, which might contain sensitive information such as credentials to other resources (database servers, etc), SSL certificates, ...

This makes the hash of the system closure as well sensitive and it should not be left in the unencrypted area.

[ckauhaus]

Sounds like a "security by obscurity" problem

[volth]

I am only about using
"init=/nix/var/nix/profiles/system-12-link/init"
instead of
"init=/nix/store/33333333333333333333333333333333-nixos-system-x220aj-19.03.1699/init"

(and the same for systemClosure, kernel and initrd)

Anyway all the content of grub.cfg is formed from ls /nix/var/nix/profiles/system-*-link.
The only proposed change is not to resolve the symlinks from /nix/var/nix/profiles/ to their full /nix/store/... paths

[Ekleog]

The only sensitive information that should be in the system closure is the versions of the packages (well… once encryption will finally be merged). Sure that's enough to help mount attacks, but it's not really bad IMO, especially as most programs are fingerprint-able anyway.

Still, it does make sense to use the anonymized paths… if it's actually possible to rely on them never being reused, which I'm not sure of.

[Moredread]

Even then a user can put sensitive information into the nix store by accident, so it seems an easy change to prevent an attacker to exploit it.

[Ekleog]

That change won't prevent an attacker from exploiting anything. If your secrets are available on a cache server accessible from the internet (which is the assumption for an attack based on that), then you already have bigger problems. Like, most likely timing attacks would allow the attacker to list the contents of your cache anyway without having physical access to your computer (if there isn't an even easier way of accessing it, I haven't checked how binary caches were served).

That said, yes, why not anonymize the paths, it's still a tiny bit better for some marginal use cases, maybe :) (though I haven't seen proof of that yet)

But I think if it means there's a risk of a grub entry being re-used and pointing to the wrong entry, then this idea should be dropped to the floor, because the actual issue is that the secrets have been put in the store in the first place.

[volth]

because the actual issue is that the secrets have been put in the store in the first place.

Well, even it there is no explicit secrets on the cache server, there are a lot of information useful for an attacker: all system configuration (e.g. firewall rules, IP addresses of other machines, ...) and all the binaries (versions + possibility to check their exploitability offline)

[Ekleog]

Yes, that's what I was thinking of in my first message when mentioning “versions of packages.” But the thing is that an attacker can likely already get this information by attacking directly the cache without ever having access to the hard drive of the computer using it, by eg. timing attacks or other attacks on the cache.

Basically, I feel that this issue is indeed a security issue, but doesn't respect xkcd's law for knowing which vulnerabilities deserve being fixed quickly: it's likely easier to just hit on the cache than to access the shutdown hard drive of a computer using it.

[danbst]

Knowing the hash of nix closure (33333333333333333333333333333333 in this example) allows the attacker to query nix caches

but how? Are you assuming that system derivation is served from some public place?

[volth]

Knowing the hash of nix closure (33333333333333333333333333333333 in this example) allows the attacker to query nix caches

but how? Are you assuming that system derivation is served from some public place?

Yes.

If the deployment is not limited to one hosting provider, nix-serve should be served from a public place to allow nix-infecting machines on different hostings.

And there is almost nothing to do to make it less public: it should be available from all Amazon, DigitalOcean, Hetzner, Packet's ... IP addresses where a new server would pop up (if you own an IP space and everywhere mount it, you are happy; but I do not)

And then curl http://my-cache.org/33333333333333333333333333333333.narinfo is enough to start exploring the system closure

[volth]

And we could ask @domenkozar on how many system closures there are on Cachix.
I guess a lot

[domenkozar]

I don't want to give that information out (as just number can be sensitive, but I haven't checked). I do want to say that using cachix to store anything sensitive is a bad idea, nixos machines should be protected with security (cryptography) instead of obscurity (discoverability of nix hashes).

[volth]

nixos machines should be protected with security (cryptography) instead of obscurity (discoverability of nix hashes).

With the current architecture of Nix cache, they are protected only by discoverability of nix hashes.

So far I have seen no proposal for introducing cryptography to protect Nix caches which are exposed to public Internet.
With better Nix caches it won't be needed to protect the hashes, but it is about the far distant future

[Ekleog]

@volth Hmm… that is a good point. Maybe you could open a RFC for having an encrypted binary cache protocol? Now, the only use case I could see is your having to serve your configuration to the whole internet for nix-infect (not sure exactly which nix-infect you're talking about, would you have a link?), as otherwise you could just use a private cache (eg. in a VPN). Maybe it'd be possible to fix nix-infect instead?

[volth]

nix-infect (not sure exactly which nix-infect you're talking about, would you have a link?),

I refer here to the common method of turning

• Debian/CentOS machines
• machines booted from system-rescue-cd.iso
• machines booted from hoster's PXE rescue system

into NixOS by downloading .nars from a public Nix cache and then either kexec new kernel or installing grub and reboot. I have own code but there are plenty open source solutions on Github, even nixpkgs has one embedded (grep -i "lustrate") and nixops has another one (which I forked).

private cache (eg. in a VPN)

That means one has to set up the VPN on zoo of different systems, some of them (PXE booted) do not even have a package manager; and it also implies creation of a non-NixOS system for managing and distributing secrets (VPN keys/passwords).
Perhaps, one-time passwords would help here.

Now, the only use case I could see is your having to serve your configuration

Another is Cachix which presumable containing other's secrets protected only by discoverability of their hashes

[domenkozar]

I am in the process of implementing authenticated binary caches for cachix, as per https://github.com/cachix/cachix/issues/24 - using https and netrc (basic auth). It's not perfect, but it does the job.

[Ekleog]

@volth Oh, if you control your system then you could maybe download from https://user:[email protected]? This would avoid the overhead of setting a VPN and should be relatively secure (unless you assume the initial OS is compromised specially to attack you… in which case it could just edit your closure anyway)

@domenkozar Hmm… I guess this would be a paid feature? At least that sounds like a good way to make cachix self-sustainable by making private binary caches paid, as private binary caches don't profit the overall community :)

[Ekleog]

@volth Then again, if it's possible to hide the path leaked from grub without drawbacks like nixos-rebuild boot not working in certain circumstances (eg. multiple differently-named generation channels) or entries in grub being re-associated to wrong system generations after a garbage-collection (the two most obvious potential failure modes I just came up with), then why not? I just don't think it should be prioritized over… almost anything else. Now, I'll see myself out, having said all I wanted to say :)

[domenkozar]

@Ekleog yes, it will be a paid feature of cachix.

[volth]

and should be relatively secure (unless you assume the initial OS is compromised specially to attack you…

The attack vector I consider here is leaking out the backup images of virtual machines made by hostings (on some hostings it is not even possible to turn it off, they make backup images daily or weekly).

That's why I (and many others, as the feature is embedded in nixops) encrypt the disks of cloud instances with LUKS.
It does not help from advanced attacks, but it protects offline disk images to some extent.

Stealing laptops is another reason why people use LUKS. At least we have to be aware of the hashes left unencrypted in grub.cfg (and some less sensitive hashes in initrd) and what the attacker could recover knowing the hashes.

[andrew-d]

I think I'm missing something here, so if you don't mind clarifying: the concern is that an attacker would be able to obtain sensitive information from a public cache server, given physical access to an encrypted system (and thus the hash of the systemConfig, since presumably the kernel/initrd aren't particularly interesting)? In this case, we assume:

• The disk is encrypted, so an attacker can't simply read the Nix store
• The legitimate user of the system has pushed their entire Nix closure to a cache (since presumably if you don't push the root systemConfig, then you can't fetch that to traverse to anything sensitive).
• The cache is public, and an attacker can find it / fetch arbitrary things from it.

Is that an accurate summary?

And if so: given the lack of Secure Boot support in NixOS (#42127, #39286), my general view on this is "an attacker could also just backdoor your boot loader to capture your password and exfiltrate it", so I don't know that this is a huge risk? Again, possible that I'm missing something - clarification is appreciated!

[volth]

my general view on this is "an attacker could also just backdoor your boot loader to capture your password and exfiltrate it"

I'd say it is another treat: when your machine had been in the hands of the attacker and then returned to you.

I'm talking only about the attacker obtained your laptop (or cloud disk images) with the disk encrypted. The attacker cannot decrypt it (by installing a backdoor and prompting you to enter the password as though you are unaware of the laptop absence or by using the force or somehow else) and have to rely only on what can be found on the disk.
And there are hashes of systemConfig, which act like access tokens to public caches or private caches exposed to Internet.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[ckauhaus]

No one has come up with a PR, so I think it is reasonable to close this issue. Discussion should continue on Discourse if necessary.