Nixpkgs: security.lockKernelModules hides(?) /boot device

[NeQuissimus]

Issue description

So I enabled security.lockKernelModules...
The system boots fine but nixos-rebuild becomes entirely unusable because the device supposed to be mounted to /boot is somehow gone (I get No such device with an error code 19).

I assume that adding some module can fix this but I have two issues:

• What is that module?
• Why is it not added by default?

Here is my journal for when it fails:

Sep 05 07:50:07 nixus nix-daemon[2084]: accepted connection from pid 3064, user root (trusted)
Sep 05 07:50:07 nixus nix-daemon[2084]: accepted connection from pid 3079, user root (trusted)
Sep 05 07:50:10 nixus nix-daemon[2084]: accepted connection from pid 3085, user root (trusted)
Sep 05 07:50:12 nixus nix-daemon[2084]: accepted connection from pid 3965, user root (trusted)
Sep 05 07:50:12 nixus systemd[1]: boot.automount: Got automount request for /boot, triggered by 3972 (pbxy00fxvjniqw4)
Sep 05 07:50:12 nixus systemd[1]: Mounting EFI System Partition Automount...
Sep 05 07:50:12 nixus systemd[1]: boot.mount: Mount process exited, code=exited status=32
Sep 05 07:50:12 nixus systemd[1]: Failed to mount EFI System Partition Automount.
Sep 05 07:50:12 nixus systemd[1]: boot.mount: Unit entered failed state.
Sep 05 07:50:12 nixus systemd[1]: boot.automount: Got automount request for /boot, triggered by 3972 (pbxy00fxvjniqw4)
Sep 05 07:50:12 nixus systemd[1]: Mounting EFI System Partition Automount...
Sep 05 07:50:12 nixus systemd[1]: boot.mount: Mount process exited, code=exited status=32
Sep 05 07:50:12 nixus systemd[1]: Failed to mount EFI System Partition Automount.
Sep 05 07:50:12 nixus systemd[1]: boot.mount: Unit entered failed state.
Sep 05 07:50:28 nixus systemd[1]: boot.automount: Got automount request for /boot, triggered by 4029 (exa)
Sep 05 07:50:28 nixus systemd[1]: Mounting EFI System Partition Automount...
Sep 05 07:50:28 nixus systemd[1]: boot.mount: Mount process exited, code=exited status=32
Sep 05 07:50:28 nixus systemd[1]: Failed to mount EFI System Partition Automount.
Sep 05 07:50:28 nixus systemd[1]: boot.mount: Unit entered failed state.
Sep 05 07:51:04 nixus systemd[1]: boot.automount: Got automount request for /boot, triggered by 4236 (exa)
Sep 05 07:51:04 nixus systemd[1]: Mounting EFI System Partition Automount...
Sep 05 07:51:04 nixus systemd[1]: boot.mount: Mount process exited, code=exited status=32
Sep 05 07:51:04 nixus systemd[1]: Failed to mount EFI System Partition Automount.
Sep 05 07:51:04 nixus systemd[1]: boot.mount: Unit entered failed state.
Sep 05 07:51:08 nixus xsession[1601]: 1504612268700        [email protected]        WARN        Please specify whether you want browser_style or not in your browser_action options.
Sep 05 07:51:11 nixus xsession[1601]: Extension error: [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIPrefBranch.getCharPref]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: jar:file:///home/nequi/.mozilla/firefox/zmb0eif1.default/extensions/@testpilot-addon.xpi!/bootstrap.js :: sendInitialPrefs :: line 753"  data: no] undefined 753
Sep 05 07:51:11 nixus xsession[1601]: [[Exception stack
Sep 05 07:51:11 nixus xsession[1601]: sendInitialPrefs@jar:file:///home/nequi/.mozilla/firefox/zmb0eif1.default/extensions/@testpilot-addon.xpi!/bootstrap.js:753:23

Steps to reproduce

• Enable security.lockKernelModules in configuration.nix
• nixos-rebuild boot && boot
• Disable security.lockKernelModules in configuration.nix
• nixos-rebuild boot && boot <-- Won't work

Technical details

• System: 17.09pre114257.6b95cf646c (Hummingbird)
• Nix version:nix-env (Nix) 1.11.13
• Nixpkgs version: 17.09pre114257.6b95cf646c
• Sandboxing enabled: build-use-sandbox = true

Ping @joachifm

[edolstra]

Probably you need fat and vfat preloaded, and maybe nls_iso8859_1 and nls_cp437.

I guess we could try to add all modules required by known filesystems, similar to what we already do for filesystems required in the initrd (see e.g. nixos/modules/tasks/filesystems/vfat.nix).

[dezgeg]

Or the systemd unit that lockKernelModules runs could be ordered better.

[joachifm]

@dezgeg I'm interested in improving the ordering. Can you elaborate? I'm unsure how ordering relates to this problem, as the unit is activated after multi-user.target, so I'd expect it to run after all filesystems that are part of normal startup have been mounted. Is that not the case?

Not being able to mount other filesystems (whose modules are not already loaded) once the system has entered normal operation is expected. I'd like to add filesystem modules per @edolstra's suggestion (maybe there could be a system.fsModules to complement system.fsPackages, but then again this is a fairly niche problem so perhaps not worth it).

[dezgeg]

I have a recollection of that if a /boot mountpoint is specified incorrectly, the system seemingly boots to the terminal successfully, but once the mount unit times out, it throws you to emergency mode. I thought that would imply that there is a Requires dependency but no After dependency from boot.mount -> multi-user.target. But admittedly I am not too familiar on the finer points of unit ordering on systemd.

But now that I look at the error again:
Sep 05 07:50:12 nixus systemd[1]: boot.automount: Got automount request for /boot, triggered by 3972 (pbxy00fxvjniqw4) Sep 05 07:50:12 nixus systemd[1]: Mounting EFI System Partition Automount...
It looks like it's coming from an auto-generated automount unit, not from something in fileSystems."/boot" at all...

[joachifm]

@NeQuissimus can you elaborate a little bit on how your fileSystems is setup?

[NeQuissimus]

@joachifm Absolutely: https://github.com/NeQuissimus/DevSetup/blob/master/ux305c-hardware.nix :D Basically the simplest possible setup :)

https://github.com/NeQuissimus/DevSetup/blob/master/ux305c.nix has the rest of my config, including my LUKS setup

[joachifm]

@NeQuissimus thank you. I'm a little unsure why there would be an automount here, seems like /boot would be mounted during startup under this config. Does the module listing actually include vfat, nls-cp437, and nls-iso8859-1 before the rebuild? If so, that'd be quite surprising. Anyway, I'm working on deferred mounts/declared automounts, hopefully will manage to get it in before release.

[joachifm]

I've pushed some changes that at least fix the problem in a vm test.

[NeQuissimus]

@joachifm I will try it out and let you know how it went for me

[NeQuissimus]

Seems to work fine, I can ls /boot again and nixos-rebuild also works after enabling security.lockKernelModules

[joachifm]

@NeQuissimus that's great. Thank you for testing :)

[NeQuissimus]

@joachifm Bad news:

λ sudo ls /boot
ls: cannot open directory '/boot': No such device

Did you not cherry-pick to 17.09? I switched from master onto stable...

Edit: It is picked... Then something else is still wrong :D

[joachifm]

Are the modules loaded or not? What is the sequence of commands upto that error?

[NeQuissimus]

I did nothing but log in and open a terminal, let me pull my module list...

Edit: here we go

Looks like the above modules are not loaded

Click to expand

Module                  Size  Used by

ipt_MASQUERADE         16384  2

nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE

nf_conntrack_netlink    32768  0

xfrm_user              32768  1

xfrm_algo              16384  1 xfrm_user

xt_addrtype            16384  2

br_netfilter           24576  0

bridge                135168  1 br_netfilter

stp                    16384  1 bridge

llc                    16384  2 bridge,stp

ctr                    16384  4

ccm                    20480  6

af_packet              40960  16

nfnetlink_queue        24576  1

nfnetlink              16384  5 nfnetlink_queue,nf_conntrack_netlink

iptable_mangle         16384  1

xt_NFQUEUE             16384  2

xt_mark                16384  2

hid_generic            16384  0

usbhid                 45056  0

hid                   110592  2 hid_generic,usbhid

joydev                 20480  0

mousedev               20480  0

snd_hda_codec_hdmi     49152  1

snd_hda_codec_realtek    77824  1

snd_hda_codec_generic    69632  1 snd_hda_codec_realtek

snd_soc_skl            73728  0

snd_soc_skl_ipc        49152  1 snd_soc_skl

snd_soc_sst_ipc        16384  1 snd_soc_skl_ipc

snd_soc_sst_dsp        24576  1 snd_soc_skl_ipc

snd_hda_ext_core       20480  1 snd_soc_skl

snd_soc_sst_match      16384  1 snd_soc_skl

arc4                   16384  2

snd_soc_core          192512  1 snd_soc_skl

x86_pkg_temp_thermal    16384  0

intel_powerclamp       16384  0

snd_compress           20480  1 snd_soc_core

ac97_bus               16384  1 snd_soc_core

coretemp               16384  0

snd_pcm_dmaengine      16384  1 snd_soc_core

crct10dif_pclmul       16384  0

crc32_pclmul           16384  0

ghash_clmulni_intel    16384  0

intel_cstate           16384  0

rtsx_pci_ms            20480  0

rtsx_pci_sdmmc         24576  0

mmc_core              126976  1 rtsx_pci_sdmmc

iwlmvm                294912  0

memstick               16384  1 rtsx_pci_ms

iptable_nat            16384  1

mei_wdt                16384  0

nf_nat_ipv4            16384  1 iptable_nat

nf_nat                 24576  2 nf_nat_masquerade_ipv4,nf_nat_ipv4

i915                 1478656  6

mac80211              557056  1 iwlmvm

wmi_bmof               16384  0

nf_conntrack_ipv6      20480  1

nf_defrag_ipv6         36864  1 nf_conntrack_ipv6

nf_conntrack_ipv4      16384  6

nf_defrag_ipv4         16384  1 nf_conntrack_ipv4

xt_conntrack           16384  4

nf_conntrack          106496  8 nf_conntrack_ipv6,nf_conntrack_ipv4,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_nat

libcrc32c              16384  2 nf_conntrack,nf_nat

evdev                  20480  16

drm_kms_helper        135168  1 i915

iwlwifi               221184  1 iwlmvm

efi_pstore             16384  0

input_leds             16384  0

ip6t_rpfilter          16384  1

pstore                 20480  1 efi_pstore

ipt_rpfilter           16384  1

mac_hid                16384  0

snd_hda_intel          36864  3

ip6table_raw           16384  1

intel_uncore          106496  0

iptable_raw            16384  1

cfg80211              524288  3 iwlmvm,iwlwifi,mac80211

e1000e                225280  0

drm                   299008  5 i915,drm_kms_helper

snd_hda_codec         106496  4 snd_hda_intel,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek

xt_pkttype             16384  2

nf_log_ipv6            16384  1

serio_raw              16384  0

intel_rapl_perf        16384  0

nf_log_ipv4            16384  1

nf_log_common          16384  2 nf_log_ipv6,nf_log_ipv4

xt_LOG                 16384  2

psmouse               118784  0

xt_tcpudp              16384  4

efivars                20480  1 efi_pstore

snd_hda_core           65536  7 snd_hda_intel,snd_hda_codec,snd_hda_ext_core,snd_soc_skl,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek

snd_hwdep              16384  1 snd_hda_codec

snd_pcm                81920  8 snd_hda_intel,snd_hda_codec,snd_pcm_dmaengine,snd_hda_ext_core,snd_hda_core,snd_soc_skl,snd_hda_codec_hdmi,snd_soc_core

ptp                    20480  1 e1000e

pps_core               20480  1 ptp

snd_timer              28672  1 snd_pcm

rtsx_pci               49152  2 rtsx_pci_sdmmc,rtsx_pci_ms

thinkpad_acpi          77824  1

nvram                  16384  1 thinkpad_acpi

wmi                    20480  1 wmi_bmof

rfkill                 24576  4 thinkpad_acpi,cfg80211

intel_gtt              20480  1 i915

snd                    61440  17 snd_compress,snd_hda_intel,snd_hwdep,snd_hda_codec,snd_timer,thinkpad_acpi,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek,snd_soc_core,snd_pcm

soundcore              16384  1 snd

led_class              16384  3 iwlmvm,input_leds,thinkpad_acpi

agpgart                32768  2 intel_gtt,drm

i2c_algo_bit           16384  1 i915

ip6table_filter        16384  1

battery                20480  0

fb_sys_fops            16384  1 drm_kms_helper

ip6_tables             24576  2 ip6table_filter,ip6table_raw

ac                     16384  0

syscopyarea            16384  1 drm_kms_helper

iptable_filter         16384  1

sysfillrect            16384  1 drm_kms_helper

mei_me                 36864  1

sysimgblt              16384  1 drm_kms_helper

video                  36864  2 thinkpad_acpi,i915

intel_pch_thermal      16384  0

mei                    86016  3 mei_me,mei_wdt

button                 16384  1 i915

i2c_core               61440  5 i2c_algo_bit,psmouse,i915,drm_kms_helper,drm

thermal                20480  0

shpchp                 32768  0

msr                    16384  0

loop                   28672  0

kvm_intel             192512  0

kvm                   516096  1 kvm_intel

irqbypass              16384  1 kvm

efivarfs               16384  1

ip_tables              24576  4 iptable_mangle,iptable_filter,iptable_raw,iptable_nat

x_tables               24576  17 ipt_rpfilter,xt_LOG,xt_pkttype,iptable_mangle,ip_tables,ip6t_rpfilter,iptable_filter,xt_mark,xt_tcpudp,iptable_raw,ipt_MASQUERADE,xt_NFQUEUE,ip6table_filter,xt_addrtype,ip6table_raw,xt_conntrack,ip6_tables

ipv6                  413696  50 ip6t_rpfilter,nf_conntrack_ipv6,nf_defrag_ipv6,bridge

crc_ccitt              16384  1 ipv6

autofs4                36864  3

atkbd                  28672  0

libps2                 16384  2 atkbd,psmouse

xhci_pci               16384  0

xhci_hcd              188416  1 xhci_pci

usbcore               212992  3 usbhid,xhci_pci,xhci_hcd

sd_mod                 45056  3

usb_common             16384  1 usbcore

i8042                  24576  0

rtc_cmos               20480  0

serio                  20480  7 serio_raw,atkbd,psmouse,i8042

dm_mod                106496  0

dax                    20480  1 dm_mod

crc32c_generic         16384  0

crc32c_intel           24576  2

btrfs                1060864  1

xor                    24576  1 btrfs

raid6_pq              114688  1 btrfs

veth                   16384  0

aesni_intel           184320  5

aes_x86_64             20480  1 aesni_intel

crypto_simd            16384  1 aesni_intel

cryptd                 20480  3 crypto_simd,ghash_clmulni_intel,aesni_intel

glue_helper            16384  1 aesni_intel

ahci                   36864  2

libahci                28672  1 ahci

libata                208896  2 ahci,libahci

scsi_mod              196608  2 sd_mod,libata

[NeQuissimus]

Ah, you know... Friday is not my favourite day of the week...
Once I actually add vfat, nls-cp437 and nls-iso8859-1 to my kernelModules, it is all good.
I must have removed them at some point...

[joachifm]

Did you switch into the configuration from a non-locked state or boot into a new configuration with that option enabled? If the latter it should work because you should have all the required modules loaded but in the former case it might not work, depending on whether you had the requisite modules already loaded.

Switching into a configuration with this option enabled without also having already loaded all requisite kernel modules will break anything that depends on those kernel modules until reboot. Worst case scenario is you need to boot into an older generation.

I recommend testing in a vm before booting into the configuration and certainly never switch into it without being certain that you have all kernel modules you need already loaded (and that they won't be unloaded for some reason during switching).

[NeQuissimus]

I had it unlock, enabled it, did nixos-rebuild boot --upgrade && reboot and then ran the sudo ls /boot.
Should the module auto-load the modules maybe?

[joachifm]

Ah, comment race :) Well, I'm a little confused why the module does not add the requisite kernel modules itself, it should do so if any fileSystems has type vfat. That was the "fix" for the deferred mounts issue.

[NeQuissimus]

I don't have a fileSystem that is vfat, that is why!
boot.loader.systemd-boot.enable = true; does the /boot stuff for me.

[NeQuissimus]

λ mount | grep '/boot'                                                                                                                                         
systemd-1 on /boot type autofs (rw,relatime,fd=28,pgrp=1,timeout=120,minproto=5,maxproto=5,direct)

[joachifm]

Aha, that explains the reference to automount in the original error message

[joachifm]

I hope the fix will be to also consider systemd.automounts when selecting fs modules to include.

[NeQuissimus]

I have to run now, otherwise I would take a look. If you have anything, let me know and I'll try it out. Or maybe on the weekend I may take a jab at it.

[NeQuissimus]

In case anybody runs into issues with Docker (17.09-ce right now) with this:

boot.initrd.kernelModules = [
  "bridge"
  "br_netfilter"
  "nf_nat"
  "veth"
  "xt_conntrack"
  "xt_nat"
];