Nixpkgs: nixos/systemd-boot: Cannot run nixos-rebuild switch

Created on 8 Sep 2020 · 30Comments · Source: NixOS/nixpkgs

Describe the bug

After updating to the latest nixos-unstable (a31736120c5de6e632f5a0ba1ed34e53fc1c1b00), I can no longer run nixos-rebuild switch. Here is the traceback:

Traceback (most recent call last):
  File "/nix/store/1l11d4mqg6kgw90cv8lav8qiijasmsdk-systemd-boot-builder.py", line 256, in <module>
    main()
  File "/nix/store/1l11d4mqg6kgw90cv8lav8qiijasmsdk-systemd-boot-builder.py", line 203, in main
    sdboot_status = subprocess.check_output(["/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/bin/bootctl", "--path=/boot", "status"], universal_newlines=True)
  File "/nix/store/bs03sg8b0gq2zr4v252hh9psp780qj5q-python3-3.8.5/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/nix/store/bs03sg8b0gq2zr4v252hh9psp780qj5q-python3-3.8.5/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/bin/bootctl', '--path=/boot', 'status']' returned non-zero exit status 1.
warning: error(s) occurred while switching to the new configuration

The reason why bootctl status returns 1 is in src/boot/bootctl.c line 1256, which happens because my laptop is too old to have the efi variable OsIndicationsSupported.

I don't know if it is the correct behavior for bootctl to return an error in this case or not, but it certainly should not stop me from being able to run nixos-rebuild switch.

To Reproduce
Steps to reproduce the behavior:

Setup an older laptop (e.g. an x220) to use NixOS with UEFI
Run nixos-rebuild switch with the newest nixos-unstable channel.

Expected behavior
nixos-rebuild switch should work even on older UEFI systems.

Additional context
This code was introduced in June in #90386.

I managed to switch to the new configuration by running nixos-rebuild build && NIXOS_INSTALL_BOOTLOADER=1 ./result/bin/switch-to-configuration switch.

Notify maintainers

@flokli @danielfullmer

Metadata

system: "x86_64-linux"
host os: Linux 5.4.62, NixOS, 20.09.20200907.a317361 (Nightingale)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 3.0pre20200829_f156513
nixpkgs: /nix/store/8jdcp4ilj7h4swa89w9c9hxyl78qa5ia-source

bug nixos

Source

TethysSvensson

All 30 comments

duplicate https://github.com/NixOS/nixpkgs/issues/97426

turboMaCk on 8 Sep 2020

@turboMaCk I think we should validate if this is really the same bug or just a related one.

Could you provide the stack trace from systemd-boot-builder.py along with the output of the following command?

bootctl --path /boot status; echo $?

TethysSvensson on 8 Sep 2020

I can no longer reproduce this by running regular nixos-rebuild switch --upgrade.

I have these versions in store:

878yngamvpk4hkshl59c7vydapkc8p41-systemd-boot-builder.py.drv
c98r56rf8iijhybmjwx8bl5gr95g27by-systemd-boot-builder.py
m89ljdrs6icsx416i1b24z3g3ynlms34-systemd-boot-builder.py
qp0cchnv8b21a87v2py02cpq5y5rkw1d-systemd-boot-builder.py.drv
yzw594jcz8gzfb2r02qfng7lczjz98ak-systemd-boot-builder.py

I'm not sure how I can reproduce it.

this bootctl status:

System:
     Firmware: UEFI 2.70 (American Megatrends 5.14)
  Secure Boot: disabled
   Setup Mode: user
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 245
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
          ESP: /dev/disk/by-partuuid/5093df98-2dfa-43ff-9453-172ba9c30519
         File: └─/EFI/BOOT/BOOTX64.EFI

Random Seed:
 Passed to OS: no
 System Token: not set
       Exists: no

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/5093df98-2dfa-43ff-9453-172ba9c30519)
         File: └─/EFI/systemd/systemd-bootx64.efi (systemd-boot 246)
         File: └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 246)

Boot Loaders Listed in EFI Variables:
        Title: UEFI OS
           ID: 0x0001
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/5093df98-2dfa-43ff-9453-172ba9c30519
         File: └─/EFI/BOOT/BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0002
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/6cae72a9-fc4d-4774-b023-a0b834a3c6fa
         File: └─/EFI/BOOT/BOOTX64.EFI

Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/5093df98-2dfa-43ff-9453-172ba9c30519)

Default Boot Loader Entry:
        title: NixOS (Generation 561 NixOS 20.09pre242714.a31736120c5, Linux Kernel 5.7.19, Built on 2020-09-08)
           id: nixos-generation-561.conf
       source: /boot/loader/entries/nixos-generation-561.conf
      version: Generation 561 NixOS 20.09pre242714.a31736120c5, Linux Kernel 5.7.19, Built on 2020-09-08
   machine-id: 60251ade5f1044d19a3ede448d2c9360
        linux: /efi/nixos/vp67gicihrwz5j86ygswag5sm5dsh5dr-linux-5.7.19-bzImage.efi
       initrd: /efi/nixos/ivfkbkriggp6z0589ldzmsjvq9yqy79f-initrd-linux-5.7.19-initrd.efi
      options: systemConfig=/nix/store/dxfbbixyxz14rclnc9zh8isf2vg99v65-nixos-system-nixos-mainframe-20.09pre242714.a31736120c5 init=/nix/store/dxfbbixyxz14rclnc9zh8isf2vg9

turboMaCk on 8 Sep 2020

@TethysSvensson does bootctl status --no-variables also exit with nonzero exit code on your system, and what is its output?

xaverdh on 8 Sep 2020

Yes, it also exits with exit code 1. This is the output:

System:
     Firmware: UEFI 2.00 (Lenovo 0.5216)
  Secure Boot: disabled
   Setup Mode: user
 Boot into FW: not supported

Current Boot Loader:
      Product: systemd-boot 246
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
          ESP: /dev/disk/by-partuuid/631e1c5d-b0a0-467c-837d-6b499c9c1123
         File: └─/EFI/BOOT/BOOTX64.EFI

Random Seed:
 Passed to OS: no
 System Token: not set
       Exists: yes

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/631e1c5d-b0a0-467c-837d-6b499c9c1123)
         File: └─/EFI/systemd/systemd-bootx64.efi (systemd-boot 246)
         File: └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 246)

Boot Loaders Listed in EFI Variables:
Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/631e1c5d-b0a0-467c-837d-6b499c9c1123)

Default Boot Loader Entry:
        title: NixOS (Generation 353 NixOS 20.09.20200907.a317361, Linux Kernel 5.4.62, Built on 2020-09-08)
           id: nixos-generation-353.conf
       source: /boot/loader/entries/nixos-generation-353.conf
      version: Generation 353 NixOS 20.09.20200907.a317361, Linux Kernel 5.4.62, Built on 2020-09-08
   machine-id: 8c49c84720df4763a5113c06974e7d50
        linux: /efi/nixos/mvx9hym5nw3wprz2ng3h2k1ympx32w5a-linux-5.4.62-bzImage.efi
       initrd: /efi/nixos/lrjk3dirl8id56cgnj9k4cg908prp2b2-initrd-linux-5.4.62-initrd.efi
      options: systemConfig=/nix/store/8m6m6qj88fxwfyhlsrmqj6hz1i3lvrca-nixos-system-nixtop-20.09.20200907.a317361 init=/nix/store/8m6m6qj88fxwfyhlsrmqj6hz1i3lvrca-nixos-system-nixtop-20.09.20200907.a317361/init loglevel=4

As far as I can tell, the exit code is still because of src/boot/bootctl.c line 1256

TethysSvensson on 8 Sep 2020

hmm perhaps the builder should just ignore the exit status and sanity check if /EFI/systemd/ is nonempty instead (thats what bootctl is-installed does)..

xaverdh on 8 Sep 2020

Did someone poke inside bootctl source/docs on when it does exit 1? I'd prefer understanding the exit code instead of just ignoring it…

flokli on 8 Sep 2020

Did someone poke inside bootctl source/docs on when it does exit 1? I'd prefer understanding the exit code instead of just ignoring it…

AFAICT the following two changes in combination changed bootctls behavior to fail when it thinks booting into the "firmware" (BIOS/UEFI setup?) is not possible:

I'm not sure this is intentional.

sareyko on 8 Sep 2020

Did someone poke inside bootctl source/docs on when it does exit 1? I'd prefer understanding the exit code instead of just ignoring it…

AFAICT the following two changes in combination changed bootctls behavior to fail when it thinks booting into the "firmware" (BIOS/UEFI setup?) is not possible:
* [systemd/systemd@8a96369](https://github.com/systemd/systemd/commit/8a96369e0b6e97729e784b04c2ddc176354759ab)

* [systemd/systemd@002914e](https://github.com/systemd/systemd/commit/002914e6887571388e84a7753bd4deca925ab064)
I'm not sure this is intentional.

I don't think this is responsible, verb_status from bootctlcalls into efi_get_reboot_to_firmware, not verb_reboot_to_firmware. Also the output posted above still shows all the stuff after that point in the code.

xaverdh on 8 Sep 2020

Ah ok the second patch might still be responsible, the return value from efi_get_reboot_to_firmware probably survives until the function exits, as TethysSvensson suggested above..

xaverdh on 8 Sep 2020

Can you open a bug report upstream to check, if it's not documented somewhere this is intended behavior?

flokli on 8 Sep 2020

I just tried updating my system (running unstable).
and got this error:

building '/nix/store/8frj3fznlrh70k8gs6bgmrk2y23illbs-nixos-system-hogwarts-20.09pre242714.a31736120c5.drv'...
updating systemd-boot from 245 to 246
Skipping "/boot/EFI/systemd/systemd-bootx64.efi", since it's owned by another boot loader.
Copied "/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
Traceback (most recent call last):
  File "/nix/store/c98r56rf8iijhybmjwx8bl5gr95g27by-systemd-boot-builder.py", line 256, in <module>
    main()
  File "/nix/store/c98r56rf8iijhybmjwx8bl5gr95g27by-systemd-boot-builder.py", line 214, in main
    subprocess.check_call(["/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/bin/bootctl", "--path=/boot", "update"])
  File "/nix/store/bs03sg8b0gq2zr4v252hh9psp780qj5q-python3-3.8.5/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/bin/bootctl', '--path=/boot', 'update']' returned non-zero exit status 1.
warning: error(s) occurred while switching to the new configuration

Not had this error before, seems related to this.
Also I see no reason why "/boot/EFI/systemd/systemd-bootx64.efi" should be owned by another boot loader, I don't have any other OS'es on thes nixos disk.

Edit: After updating to 242769.61525137fd1 the switch works fine.

afreakk on 8 Sep 2020

Can you open a bug report upstream to check, if it's not documented somewhere this is intended behavior?

Here you go: https://github.com/systemd/systemd/issues/16989

sareyko on 9 Sep 2020

While we wait for systemd, could we implement a work-around in NixOS? Maybe just patch bootctl.c to set r=0 in the "not implemented case"?

TethysSvensson on 9 Sep 2020

Patching systemd is a pretty hefty world rebuild.

I propose updating the python code to ignore the error code, as an intermediate workaround, with a comment linking to this (and the upstream issue).

flokli on 9 Sep 2020

👍1

Rebooting after this errors occurs results in an unbootable system in my case. For some reason none of the previous generations are present in the boot menu anymore.

alexbakker on 9 Sep 2020

I am not sure if there are two separate (but perhaps related issues) at play here.

My error is specific to the call to bootctl status, which as far as I can tell means that the switch will not be completed.

As far as my surface-level understand of nixos-rebuild, this error means that nothing will be updated and the old config will continue to be used.

The error @afreakk gets is in bootctl update seems a bit more serious. This might also be related to @alexbakker's problem with not being able to boot.

TethysSvensson on 9 Sep 2020

For anybody interested, here is my current work-around for being able to run nixos-rebuild switch:

nixos-rebuild build && sudo nix-env -p /nix/var/nix/profiles/system --set $(readlink -f ./result) && NIXOS_INSTALL_BOOTLOADER=1 sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER ./result/bin/switch-to-configuration switch

TethysSvensson on 9 Sep 2020

❤1

nixos-rebuild build && sudo nix-env -p /nix/var/nix/profiles/system --set $(readlink -f ./result) && NIXOS_INSTALL_BOOTLOADER=1 sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER ./result/bin/switch-to-configuration switch

I’ve not tried that exact set of commands. But doing just the following seems to work fine for me:

NIXOS_INSTALL_BOOTLOADER=1 sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER nixos-rebuild switch

sareyko on 9 Sep 2020

nixos-rebuild build && sudo nix-env -p /nix/var/nix/profiles/system --set $(readlink -f ./result) && NIXOS_INSTALL_BOOTLOADER=1 sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER ./result/bin/switch-to-configuration switch

I’ve not tried that exact set of commands. But doing just the following seems to work fine for me:
NIXOS_INSTALL_BOOTLOADER=1 sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER nixos-rebuild switch

That does not work for me, possibly because I am using flakes?

TethysSvensson on 9 Sep 2020

I can confirm that I had to use @TethysSvensson's command, and @sareyko's did not work for me. Not using flakes here

NickHu on 10 Sep 2020

🚀1

It looks like systemd has addressed the issue: https://github.com/systemd/systemd/issues/16989. Will test the patch later.

alexbakker on 10 Sep 2020

I can confirm that I had to use @TethysSvensson's command, and @sareyko's did not work for me. Not using flakes here

I just tried again and can confirm that it doesn't work for me either. I guess I misremembered. Logging in as root (using su) and using either of the following commands works for me, however:

NIXOS_INSTALL_BOOTLOADER=1 nixos-rebuild switch

nixos-rebuild switch --install-bootloader

I guess sudo is doing something to the environment which prevents nixos-rebuild from passing NIXOS_INSTALL_BOOTLOADER to systemd-boot-builder.py. Using sudo -i and using the above commands doesn't work either.

sareyko on 10 Sep 2020

👍1

sorry for late reply but I finally got the stack trace from my case:

Copied "/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/s
Copied "/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOT
Random seed file /boot/loader/random-seed successfully written (512 bytes).
Failed to test system token validity: No data available
Traceback (most recent call last):
  File "/nix/store/c98r56rf8iijhybmjwx8bl5gr95g27by-systemd-boot-builder.py", line 256, in <module>
    main()
  File "/nix/store/c98r56rf8iijhybmjwx8bl5gr95g27by-systemd-boot-builder.py", line 197, in main
    subprocess.check_call(["/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/bin/bootctl", "--path=/boot", "install"])
  File "/nix/store/bs03sg8b0gq2zr4v252hh9psp780qj5q-python3-3.8.5/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/kmlzasz1lbkigpirk4vwyg14g206wq18-systemd-246/bin/bootctl', '--path=/boot', 'on-zero exit status 1.

turboMaCk on 10 Sep 2020

https://github.com/systemd/systemd/pull/17001 was merged, and I asked if it can be backported to 246-stable as well.

Is someone up to propose a PR to our python code to (silently) ignore the error code (which we can revert once nixpkgs is on a systemd version containing that fix)

flokli on 10 Sep 2020

🎉1

# nixos-rebuild boot --install-bootloader

worked for me on a 2008 MacBook.

alexeymuranov on 10 Sep 2020

I'm getting the same error with nixos-install on a new system. I've tried this with the 21.03 (unstable) and 20.03 (stable) installation media.

matthuszagh on 15 Sep 2020

@matthuszagh be aware that there is also this bug affecting nixos-install on some systems https://github.com/NixOS/nixpkgs/issues/97426 I worked around this but switching to grub (https://github.com/turboMaCk/Dotfiles/commit/e81a9503c3c9bf7728dac850df8ee44c58456e05)

turboMaCk on 15 Sep 2020

This should be fixed in master and 20.09 since https://github.com/NixOS/nixpkgs/pull/97689.

flokli on 15 Sep 2020

I got the same error when upgrading to 20.09 now (from an unstable commit in april (b61999e4ad6)) (using nixos-rebuild boot)

updating systemd-boot from 243 to 246
Copied "/nix/store/3ri1jzc32mdlywwwxpj4lzzc0b1w9awf-systemd-246.4/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Skipping "/boot/EFI/BOOT/BOOTX64.EFI", since it's owned by another boot loader.
Traceback (most recent call last):
  File "/nix/store/cmr3s11j6mk84alf5zvx5fh6d0wp7w35-systemd-boot-builder.py", line 258, in <module>
    main()
  File "/nix/store/cmr3s11j6mk84alf5zvx5fh6d0wp7w35-systemd-boot-builder.py", line 216, in main
    subprocess.check_call(["/nix/store/3ri1jzc32mdlywwwxpj4lzzc0b1w9awf-systemd-246.4/bin/bootctl", "--path=/boot", "update"])
  File "/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/3ri1jzc32mdlywwwxpj4lzzc0b1w9awf-systemd-246.4/bin/bootctl', '--path=/boot', 'update']' returned non-zero exit status 1.
warning: error(s) occurred while switching to the new configuration

Running nixos-rebuild boot again gave no error and the system booted fine afterwards.