Nixpkgs: nixos-rebuild wrongly(?) thinks systemd-boot is "not installed in ESP"

[8573]

Describe the bug
For years, I think as long as I've been using NixOS, I've been using systemd-boot, but, recently, seeming starting circa 2020-06-20 according to the nixos-upgrade.service logs, nixos-rebuild has begun complaining, seemingly on each rebuild, that "systemd-boot [is] not installed in ESP" and it "could not find any previously installed systemd-boot". For example, here's the output of nixos-rebuild switch when there's nothing to rebuild:

$ nixos-rebuild --keep-going switch --upgrade
unpacking channels...
building Nix...
building the system configuration...
systemd-boot not installed in ESP.
could not find any previously installed systemd-boot
activating the configuration...
setting up /etc...
[...]

Here are the files in my EFI System Partition, mounted at /boot:

$ find /boot -type f
/boot/EFI/gummiboot/gummibootx64.efi
/boot/EFI/Boot/BOOTX64.EFI
/boot/EFI/nixos/h3nmpipd7rsip3ni1xk1fpxcizb2nrrq-linux-5.4.46-bzImage.efi
/boot/EFI/nixos/4izqilbfch08d9i4sxh33av39chngg1p-initrd-linux-5.4.46-initrd.efi
/boot/EFI/nixos/ka3ddjw9w5hj0d5qdxwx1lnvv1fhx24h-linux-5.4.46-bzImage.efi
/boot/EFI/nixos/xxz0bb66r4gyjv8nl1zcpibbips1dxjx-initrd-linux-5.4.46-initrd.efi
/boot/EFI/nixos/kyn2y7h8001r8a96w3v0ygp28d4d4hf8-initrd-linux-5.4.46-initrd.efi
/boot/EFI/nixos/m88lhs7igr1qyy1v2pzz0zrcs2ps0j2x-linux-5.4.47-bzImage.efi
/boot/EFI/nixos/bai3l9b0kxdpmmkrhnagpzvqzrnmvm98-initrd-linux-5.4.47-initrd.efi
/boot/EFI/nixos/7ysn9377x9ac4s9605gh47lwv3cjwhl0-linux-5.4.49-bzImage.efi
/boot/EFI/nixos/2fz4c3vr9is9pnrnzx7rf1c97q6zhgr7-initrd-linux-5.4.49-initrd.efi
/boot/EFI/nixos/660pcsfpa3jwwhcc92jf46sbdrva8css-linux-5.4.50-bzImage.efi
/boot/EFI/nixos/5apilnxgvbq07zcwah174b99m0xzs7al-initrd-linux-5.4.50-initrd.efi
/boot/EFI/nixos/9b6mx7cykl27yfilxgl32dn26m67jarc-linux-5.4.51-bzImage.efi
/boot/EFI/nixos/57x08bza5vq3s68w5ydrr9bh10idqhgd-initrd-linux-5.4.51-initrd.efi
/boot/EFI/nixos/iw1cvzqf2pmxhz6kqssx9j1q33fazibf-linux-5.4.51-bzImage.efi
/boot/EFI/nixos/0z0w74n2157nfp6fchbs82425xz783lq-initrd-linux-5.4.51-initrd.efi
/boot/loader/entries/nixos-generation-860.conf
/boot/loader/entries/nixos-generation-861.conf
/boot/loader/entries/nixos-generation-862.conf
/boot/loader/entries/nixos-generation-863.conf
/boot/loader/entries/nixos-generation-864.conf
/boot/loader/entries/nixos-generation-865.conf
/boot/loader/entries/nixos-generation-866.conf
/boot/loader/entries/nixos-generation-867.conf
/boot/loader/entries/nixos-generation-868.conf
/boot/loader/entries/nixos-generation-869.conf
/boot/loader/entries/nixos-generation-870.conf
/boot/loader/entries/nixos-generation-871.conf
/boot/loader/entries/nixos-generation-872.conf
/boot/loader/entries/nixos-generation-873.conf
/boot/loader/entries/nixos-generation-874.conf
/boot/loader/entries/nixos-generation-875.conf
/boot/loader/loader.conf

My uptime is less than two weeks, so this system has booted successfully since nixos-rebuild started thinking systemd-boot is not installed, so I would guess that systemd-boot is installed despite nixos-rebuild's complaints.

Expected behavior
nixos-rebuild, if it wants to find an installation of systemd-boot, finds the one that I thought I've been using for years.

Notify maintainers
@danielfullmer I guess (from nixos/tests/systemd-boot.nix, as nixos/modules/system/boot/loader/systemd-boot/systemd-boot.nix has no listed maintainers)

Metadata

• system: "x86_64-linux"
• host os: Linux 5.4.49-hardened, NixOS, 20.09pre234242.c71518e75bf (Nightingale)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.7
• channels(root): "nixos-20.09pre234242.c71518e75bf"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: [] # I guess?
# a list of nixos modules affected by the problem
module: [boot.loader.systemd-boot] # I guess?

[danielfullmer]

So the change that happened in June was https://github.com/NixOS/nixpkgs/pull/90386 which is intended to address this issue https://github.com/NixOS/nixpkgs/issues/86422

NixOS will now automatically update systemd-boot, but only does so if it detects that systemd-boot has previously been installed, and is currently an older version. It looks like you have gummiboot in your /boot, which is a quite old version (before it was renamed to systemd-boot). Unfortunately the name change means that bootctl status doesn't detect the fact that you have gummiboot/systemd-boot installed. In it's current state, you just aren't getting the systemd-boot updates properly, but it shouldn't cause any other issues.

You should be able to just run bootctl install once to force reinstallation of systemd-boot.

[8573]

You should be able to just run bootctl install once to force reinstallation of systemd-boot.

Thanks, I did that and now nixos-rebuild no longer complains that systemd-boot isn't installed. Would you recommend additionally removing the old gummiboot?

[danielfullmer]

Yeah, after verifying that it boots correctly with the newly installed /boot/EFI/systemd/systemd-bootx64.efi, it should be fine to remove the old gummiboot.

[flokli]

Let's close this, we probably shouldn't automatically try to remove gummiboot, and apart from that, there's not much else doable here.

[Mathnerd314]

It might be worth putting a note in the manual about it. But it has a distinct error message and this issue shows up in GitHub search just fine.