Nixpkgs: fwupd doesn't work for EFI updates

[superm1]

Issue description

This is coming from an upstream perspective in fwupd. I don't have any experience directly with NixOS. I also haven't reproduced this on my own.
I've however seen reports from our reporting system come in that fwupd fails to install updates on the XPS 9360.

Example:

UpdateError=UEFI firmware update failed: {error #0} libfwup.c:1479 get_fd_and_media_path(): open of /nix/store/v2rcywfc2gvxqrbcx24afgmidxyfv9yd-fwupdate-10/boot/EFI/debian/fw/fwupdate-MLI3SW.cap failed: No such file or directory {error #1} libfwup.c:1513 get_fd_and_media_path(): mkostemps(/nix/store/v2rcywfc2gvxqrbcx24afgmidxyfv9yd-fwupdate-10/boot/EFI/nixos/fw/fwupdate-qR3wd9.cap) failed: Read-only file system, UpdateState=failed, VersionNew=0.2.6.2, VersionOld=0.2.4.2

This is with fwupd 1.0.5 and fwupdate 10.

There's two different errors, so let me unpack them.

1. The first error is because the person had Debian installed and fwupdate/libfwup tried to re-use the location of the capsule on the ESP last used for Debian (boot/EFI/debian/fw/fwupdate-MLI3SW.cap)
   this didn't exist so it tried to make a new one.
2. The second error is the attempt to create a new capsule but unable to create one as that directory is read only. The capsule needs to live in the EFI system partition for this to succeed.

I can't emphasize enough the importance of relying on the real path to the ESP. As you can read in the documentation for fwupdate the capsule must be staged on the ESP to be installed on the next boot. The EFI application to install it must be on the ESP as well.
If the application can't find the real location to the ESP it will fail.

So your package will need to either:

1. Make a directory symlink to the real ESP location.
2. Customize the ESP location in the conf file for the UEFI plugin.
   https://github.com/hughsie/fwupd/blob/master/plugins/uefi/uefi.conf

Steps to reproduce

1. Run a UEFI update on fwupd
2. Observe failure.

[superm1]

CC @jtojnar

[jtojnar]

cc @dtzWill, @moredtea since they investigated the UEFI support in https://github.com/NixOS/nixpkgs/pull/35065

By the way, fwupd is not really my area of interest, I am mostly touching it because GNOME Software depends on it. Would someone be willing to become a dedicated maintainer for it?

[jtojnar]

We also have an issue open https://github.com/NixOS/nixpkgs/issues/38935, though this one is more informative.

[superm1]

CC @Moredread I think was who you were wanting from that thread.

I do think that is on the right path with those fixes linked in https://github.com/Moredread/nixpkgs/commit/67250f386d73e4a71b30271a6e1b09f7566442d1

Does NixOS mount the ESP read-only by default? If so, a quick experiment to mount -o remount,rw should be able to fix it locally with those other fixes @Moredread linked. If that does help I think a patch is needed for fwupdate to detect it's mounted readonly and remount it read/write to help with this NixOS issue.

[Anton-Latukha]

:+1:

It is amazing that guys from the Dell interested in fwupd on NixOS.

---

P.S.

• [x] _(now in upstream)_ package of fwupd needs fwupd.service.
• [x] _(additional)_ on-demand/timer/socket-type service. It does not need to be constantly running.

I thought about solving fwupd issues, but I have no hardware that is in firmware repo, to be able to work and test fwupd.

[jtojnar]

@Anton-Latukha do you have services.fwupd.enable = true; in your configuration.nix?

[jtojnar]

@Anton-Latukha fwupd is already socket activated so it does not run until needed. Regarding auto-stopping, see the following upstream issue: https://github.com/hughsie/fwupd/issues/417

[Anton-Latukha]

@jtojnar Ok. I rely on my experience with stable. I just had seen and am seeing that daemon is still running. In this case I should read the full source code on the topic. Thank you. My messages was in vain. And can be removed from conversation.

[jtojnar]

@Anton-Latukha yes, once it is started it will continue running, see the upstream issue. It does not run before it is needed, though.

[Moredread]

@superm1 I can write to the path manually and it is mounted rw, but I have to check whether there is a general permission problem for the fwupd process.

[superm1]

@Moredread the ESP is FAT32 which doesn't support ACLs. Does the fwupd daemon not run as root on nixos?

[jtojnar]

@superm1 It runs as root.

[superm1]

@jtojnar @moredread my guess then would be this:
https://github.com/hughsie/fwupd/blob/master/data/fwupd.service.in#L19
If bootdir isn't also set to /boot that systemd rule will probably fail.

[jtojnar]

@superm1 I think that would cause the service not to start with a cryptic error. But since the bootdir is prefixed with -, it should not matter according to systemd.exex(5).

[superm1]

It doesn't matter if the path exists. But if the path is configured to the wrong path in bootdir then fwupd shouldn't be allowed to write to it. Systemd prevents all paths to be written to except those in that whitelist.

[jtojnar]

Oh, that’s neat. Either way, bootdir seems to be set the same way as in https://github.com/Moredread/nixpkgs/commit/67250f386d73e4a71b30271a6e1b09f7566442d1.

[superm1]

@Moredread i'm really not sure why it's read only then. Is there something else influencing the process that wraps it in Nix or something?

[Moredread]

@superm1 It seems to be just a "normal" permission issue. When I mount the partition with fmask=0000,dmask=0000 fwupd works fine now (on a XPS 9370). :) chmod wasn't enough though, so it didn't work when I tried it last.

What is the user the fwupd process is running as? I couldn't find the service file where that is defined and I'm not that familiar with systemd. Or should it run as root? In that case it is strange that it can't write as root has rwx permissions.

[Moredread]

Another issue I just noticed:

1. The EFI files need manual installation, i.e. copying from the fwupdate package. Would it be a good idea to handle those by services.fwupd as well, maybe as a separate option?
2. After installing those files nixos-rebuild switch fails with

Traceback (most recent call last):
  File "/nix/store/ss74k40cj1ydgn6pnfyzh0wms9czr4zc-systemd-boot-builder.py", line 210, in <module>
    main()
  File "/nix/store/ss74k40cj1ydgn6pnfyzh0wms9czr4zc-systemd-boot-builder.py", line 195, in main
    remove_old_entries(gens)
  File "/nix/store/ss74k40cj1ydgn6pnfyzh0wms9czr4zc-systemd-boot-builder.py", line 148, in remove_old_entries
    os.unlink(path)
IsADirectoryError: [Errno 21] Is a directory: '/boot/efi/nixos/fw'
warning: error(s) occurred while switching to the new configuration

Any idea how to 'ignore' those paths? Or do we need to adjust the systemd-boot-builder.py script?

[superm1]

@Moredread:

What is the user the fwupd process is running as? I couldn't find the service file where that is defined and I'm not that familiar with systemd. Or should it run as root? In that case it is strange that it can't write as root has rwx permissions.

fwupd daemon should be running as root. Can you please check your /etc/fstab for odd mount options that would prevent the ESP from being mounted normally? Are you setting umask/dmask/fmask in an odd way by default?

The EFI files need manual installation, i.e. copying from the fwupdate package. Would it be a good idea to handle those by services.fwupd as well, maybe as a separate option?

the EFI files are supposed to be installed by the "fwupdate" package today at package install time. Not sure if NixOS is separating those from fwupd or not. Keep in mind there is an issue filed upstream about having automatic management of those assets but it's waiting for someone to help with (Feel free if you would like to volunteer) https://github.com/rhboot/fwupdate/issues/97 . It only affects distros that choose not to install to the ESP at package install time.

After installing those files nixos-rebuild switch fails with < snip >
Any idea how to 'ignore' those paths? Or do we need to adjust the systemd-boot-builder.py script?

Not sure on that. Maybe @jtojnar can comment.

[jtojnar]

@Moredread

What is the user the fwupd process is running as? I couldn't find the service file where that is defined and I'm not that familiar with systemd. Or should it run as root? In that case it is strange that it can't write as root has rwx permissions.

It is running as root, which is the default value, see User= under systemd.exec(5).

1. The EFI files need manual installation, i.e. copying from the fwupdate package. Would it be a good idea to handle those by services.fwupd as well, maybe as a separate option?

We should probaly add boot.loader.efi.applications option, that would be used by the systemd-boot-builder.py script to copy EFI applications from packages. (cc @shlevy) fwupd module would then contain boot.loader.efi.applicationPackages = [ pkgs.fwupdate ].

1. After installing those files nixos-rebuild switch fails with

Traceback (most recent call last):
  File "/nix/store/ss74k40cj1ydgn6pnfyzh0wms9czr4zc-systemd-boot-builder.py", line 210, in <module>
    main()
  File "/nix/store/ss74k40cj1ydgn6pnfyzh0wms9czr4zc-systemd-boot-builder.py", line 195, in main
    remove_old_entries(gens)
  File "/nix/store/ss74k40cj1ydgn6pnfyzh0wms9czr4zc-systemd-boot-builder.py", line 148, in remove_old_entries
    os.unlink(path)
IsADirectoryError: [Errno 21] Is a directory: '/boot/efi/nixos/fw'
warning: error(s) occurred while switching to the new configuration

Any idea how to 'ignore' those paths? Or do we need to adjust the systemd-boot-builder.py script?

The script does not really expect any foreign files. All files under /boot/efi/nixos in each of packages listed in the applications option would need to be added to the known_paths whitelist; the clean-up, then, would need to call shutil.rmtree for directories. File conflicts between different application packages should be also handled.

@superm1

the EFI files are supposed to be installed by the "fwupdate" package today at package install time. Not sure if NixOS is separating those from fwupd or not.

We do not really do anything during install time, other that adding files to the Nix store. If we somehow want to modify the system, it needs to be done during system generation building.

[superm1]

The script does not really expect any foreign files. All files under /boot/efi/nixos in each of packages listed in the applications option would need to be added to the known_paths whitelist; the clean-up, then, would need to call shutil.rmtree for directories. File conflicts between different application packages should be also handled.

In that case can you set /boot/efi/nixos/fw (and all contents) to be created and owned specifically by these packages?

We do not really do anything during install time, other that adding files to the Nix store. If we somehow want to modify the system, it needs to be done during system generation building.

So both creating /boot/efi/nixos/fw and copying the EFI application will have to happen at that time (or someone will need to work out a patch for the upstream issue to have fwupdate manage creation of that directory and bootloader)

[Moredread]

@superm1 I mount /boot with default options.

Effectively those are (when running mount ) rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro

Edit: Making everything group, but not user or world, writable with fmask=0202,dmask=0202 also works, so I guess the group permission is important there, not the user permission. Any idea why that is? ls -lah /boot shows that all files and directories are owned by root:root, so I have no idea why it doesn't work...

Edit 2: I thought it works with 1.0.7, but it doesn't.

[superm1]

@Moredread A potential thought is that ProtectSystem=full is causing problems because your boot directory is /boot. See https://www.freedesktop.org/software/systemd/man/systemd.exec.html under ProtectSystem. It's not clear to me what happens if you put the same path under ReadWritePaths and enable ProtectSystem. So you might need to just patch the systemd unit in your packaging.

[terlar]

I got the following on my Dell XPS 15:

No upgrades for XPS 15 9560 TPM 2.0, current is 1.3.2.8: 1.3.1.0=older
Downloading 0.1.12.1 for XPS 15 9560 System Firmware...
Decompressing…         [***************************************]
Authenticating…        [***************************************]
Updating XPS 15 9560 System Firmware from 0.1.5.0 to 0.1.12.1… ]
Scheduling…            [***************************************]
{error #0} linux.c:389 device_get(): readlink of /sys/block/nvme0n1/device/driver failed: No such file or directory     
{error #1} util.h:323 get_file(): could not open file "/sys/class/block/nvme0n1/eui" for reading: No such file or directory   
{error #2} util.h:323 get_file(): could not open file "/sys/class/block/nvme0n1/device/eui" for reading: No such file or directory      
{error #3} linux.c:436 device_get(): parsing nvme failed: No such file or directory   
{error #4} creator.c:194 efi_va_generate_file_device_path_from_esp(): could not get ESP disk info: No such file or directory 
{error #5} creator.c:383 efi_generate_file_device_path(): could not generate File DP from ESP: No such file or directory: failed to efi_generate_file_device_path(/boot/EFI/nixos/fw/fwupd-34578c72-11dc-4378-bc7f-b643866f598c.cap)

I will look into the comments to see if anything works, could this be the same issue?

[superm1]

That looks different to me. What version of efivar? Please try with 37 if you can.

[terlar]

Didn't know I was using efivar, is it a dependency of fwupd?

The only thing mentioning efi in my config is this place, I think this is my relevant config:

  boot = {
    # Use the systemd-boot EFI boot loader.
    loader.systemd-boot.enable = true;
    loader.efi.canTouchEfiVariables = true;
    # Use the newer but stable kernel packages.
    kernelPackages = pkgs.linuxPackages_latest;

    # Kernel modules:
    kernelModules = [
      "fuse"
    ];
    blacklistedKernelModules = [
      "fbcon"
      "noveau"
      "bbswitch"
    ];
  };

  # Update support for firmware.
  services.fwupd.enable = true;

Does the loader.efi.canTouchEfiVariables enable efivar? Thanks for the feedback.

[superm1]

Yes it's a dependency of fwupd. Specifically I'm asking because there have been reports of "similar" failures on efivar 36 that were fixed in efivar 37.

[terlar]

Thank you, I am running on master updated today for my nixpkgs, so it is 36:
https://github.com/NixOS/nixpkgs/blob/a5a239ca2d2229e10d35ce2c2c9c31aba7510c8f/pkgs/tools/system/efivar/default.nix#L5

I will see of I can add an overlay to use 37 and I will report back here.

[terlar]

It worked! Thanks a lot, I finally have my firmware updated now.

For reference I added this overlay:

self: super:
{
  efivar = super.efivar.overrideDerivation (finalAttrs: rec {
    version = "37";
    src = super.fetchFromGitHub {
      owner = "rhinstaller";
      repo = "efivar";
      rev = version;
      sha256 = "1z2dw5x74wgvqgd8jvibfff0qhwkc53kxg54v12pzymyibagwf09";
      # date = 2018-12-05T11:17:42-05:00;
    };
  });
}

[andersk]

Seems like this can be closed now? All the linked PRs are merged, efivar was updated to 37 in #52322, and I’ve successfully installed multiple EFI updates on a ThinkPad Yoga X1 Gen 4 from NixOS 20.03 with no special configuration beyond services.fwupd.enable = true.