Nixpkgs: firmwareLinuxNonfree 2019-07-17 breaks iwlwifi

[peti]

05570075d7a3f9b6ccb9d0358a646477781045bb broke my systems iwlwifi driver; my laptop can no longer use the wifi interface. Reverting the firmware to the previous version, 2019-04-16, fixes the issue. I see this on my Dell Precision 5530 laptop immediately after startup:

~
iwlwifi 0000:3b:00.0: Microcode SW error detected. Restarting 0x0.
iwlwifi 0000:3b:00.0: Start IWL Error Log Dump:
iwlwifi 0000:3b:00.0: Status: 0x00000100, count: 6
iwlwifi 0000:3b:00.0: Loaded firmware version: 46.93e59cf4.0
iwlwifi 0000:3b:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
iwlwifi 0000:3b:00.0: 0x00A022F0 | trm_hw_status0
iwlwifi 0000:3b:00.0: 0x00000000 | trm_hw_status1
iwlwifi 0000:3b:00.0: 0x00488472 | branchlink2
iwlwifi 0000:3b:00.0: 0x00479392 | interruptlink1
iwlwifi 0000:3b:00.0: 0x00030926 | interruptlink2
iwlwifi 0000:3b:00.0: 0x0001A836 | data1
iwlwifi 0000:3b:00.0: 0x00001000 | data2
iwlwifi 0000:3b:00.0: 0xE0000008 | data3
iwlwifi 0000:3b:00.0: 0x00000000 | beacon time
iwlwifi 0000:3b:00.0: 0x0003E720 | tsf low
iwlwifi 0000:3b:00.0: 0x00000000 | tsf hi
iwlwifi 0000:3b:00.0: 0x00000000 | time gp1
iwlwifi 0000:3b:00.0: 0x0003E721 | time gp2
iwlwifi 0000:3b:00.0: 0x00000001 | uCode revision type
iwlwifi 0000:3b:00.0: 0x0000002E | uCode version major
iwlwifi 0000:3b:00.0: 0x93E59CF4 | uCode version minor
iwlwifi 0000:3b:00.0: 0x00000321 | hw version
iwlwifi 0000:3b:00.0: 0x18489004 | board version
iwlwifi 0000:3b:00.0: 0x00240405 | hcmd
iwlwifi 0000:3b:00.0: 0x00022000 | isr0
iwlwifi 0000:3b:00.0: 0x00000000 | isr1
iwlwifi 0000:3b:00.0: 0x20001802 | isr2
iwlwifi 0000:3b:00.0: 0x404000C0 | isr3
iwlwifi 0000:3b:00.0: 0x00000000 | isr4
iwlwifi 0000:3b:00.0: 0x00240405 | last cmd Id
iwlwifi 0000:3b:00.0: 0x0001A836 | wait_event
iwlwifi 0000:3b:00.0: 0x00000000 | l2p_control
iwlwifi 0000:3b:00.0: 0x00000020 | l2p_duration
iwlwifi 0000:3b:00.0: 0x00000000 | l2p_mhvalid
iwlwifi 0000:3b:00.0: 0x00000000 | l2p_addr_match
iwlwifi 0000:3b:00.0: 0x0000000D | lmpm_pmg_sel
iwlwifi 0000:3b:00.0: 0x19071250 | timestamp
iwlwifi 0000:3b:00.0: 0x00001840 | flow_handler
iwlwifi 0000:3b:00.0: Start IWL Error Log Dump:
iwlwifi 0000:3b:00.0: Status: 0x00000100, count: 7
iwlwifi 0000:3b:00.0: 0x20000034 | NMI_INTERRUPT_WDG
iwlwifi 0000:3b:00.0: 0x00000000 | umac branchlink1
iwlwifi 0000:3b:00.0: 0xC008888E | umac branchlink2
iwlwifi 0000:3b:00.0: 0xC0084C80 | umac interruptlink1
iwlwifi 0000:3b:00.0: 0xC0080708 | umac interruptlink2
iwlwifi 0000:3b:00.0: 0x00000400 | umac data1
iwlwifi 0000:3b:00.0: 0xC0080708 | umac data2
iwlwifi 0000:3b:00.0: 0xDEADBEEF | umac data3
iwlwifi 0000:3b:00.0: 0x0000002E | umac major
iwlwifi 0000:3b:00.0: 0x93E59CF4 | umac minor
iwlwifi 0000:3b:00.0: 0x0003E707 | frame pointer
iwlwifi 0000:3b:00.0: 0xC0886C44 | stack pointer
iwlwifi 0000:3b:00.0: 0x00240405 | last host cmd
iwlwifi 0000:3b:00.0: 0x00004008 | isr status reg
ieee80211 phy0: Hardware restart was requested
iwlwifi 0000:3b:00.0: iwlwifi transaction failed, dumping registers
iwlwifi 0000:3b:00.0: iwlwifi device config registers:
iwlwifi 0000:3b:00.0: 00000000: 25268086 00100406 02800029 00000010 ed400004 00000000 00000000 00000000
iwlwifi 0000:3b:00.0: 00000020: 00000000 00000000 00000000 40108086 00000000 000000c8 00000000 0000010b
iwlwifi 0000:3b:00.0: iwlwifi device memory mapped registers:
iwlwifi 0000:3b:00.0: 00000000: 18489004 00000040 00000000 00000000 00000000 00000000 00027e1f 00000000
iwlwifi 0000:3b:00.0: 00000020: 00000000 0c040005 00000321 d55555d5 d55555d5 d55555d5 80008040 041f0042
iwlwifi 0000:3b:00.0: iwlwifi device AER capability structure:
iwlwifi 0000:3b:00.0: 00000000: 14c10001 00000000 00000000 00462031 00000000 00002000 00000000 00000000
iwlwifi 0000:3b:00.0: 00000020: 00000000 00000000 00000000
iwlwifi 0000:3b:00.0: iwlwifi parent port (0000:00:1c.0) config registers:
iwlwifi 0000:00:1c.0: 00000000: a3388086 00100407 060400f0 00810010 00000000 00000000 003b3b00 200000f0
iwlwifi 0000:00:1c.0: 00000020: ed40ed40 0001fff1 00000000 00000000 00000000 00000040 00000000 0012010b
iwlwifi 0000:3b:00.0: iwlwifi root port (0000:00:1c.0) AER cap structure:
iwlwifi 0000:00:1c.0: 00000000: 14010001 00000000 00010000 00060011 00000000 00002000 00000000 00000000
iwlwifi 0000:00:1c.0: 00000020: 00000000 00000000 00000000 00000000 00000000 00000000
iwlwifi 0000:3b:00.0: FW error in SYNC CMD GEO_TX_POWER_LIMIT
CPU: 11 PID: 1338 Comm: dhcpcd Tainted: G O 5.1.14 #1-NixOS
Hardware name: Dell Inc. Precision 5530/0NFGCT, BIOS 1.8.1 02/01/2019
Call Trace:
dump_stack+0x5c/0x7b
iwl_trans_pcie_send_hcmd+0x4c1/0x500 [iwlwifi]
? wait_woken+0x80/0x80
iwl_trans_send_cmd+0x5e/0xc0 [iwlwifi]
iwl_mvm_send_cmd+0x23/0x80 [iwlmvm]
iwl_mvm_send_cmd_pdu+0x58/0x80 [iwlmvm]
? iwl_mvm_up+0x95b/0x9c0 [iwlmvm]
iwl_mvm_up+0x95b/0x9c0 [iwlmvm]
__iwl_mvm_mac_start+0x298/0x330 [iwlmvm]
iwl_mvm_mac_start+0x46/0x100 [iwlmvm]
drv_start+0x43/0x100 [mac80211]
ieee80211_do_open+0x149/0x850 [mac80211]
? ieee80211_check_concurrent_iface+0x166/0x1d0 [mac80211]
__dev_open+0xd8/0x170
__dev_change_flags+0x17d/0x1d0
dev_change_flags+0x23/0x60
devinet_ioctl+0x5b8/0x700
inet_ioctl+0x16e/0x250
? dev_get_by_name_rcu+0x69/0x80
? sock_do_ioctl+0x4d/0x150
? inet_stream_connect+0x50/0x50
sock_do_ioctl+0x4d/0x150
? sock_ioctl+0x1e2/0x330
sock_ioctl+0x1e2/0x330
? do_vfs_ioctl+0xa4/0x630
? dlci_ioctl_set+0x30/0x30
do_vfs_ioctl+0xa4/0x630
ksys_ioctl+0x70/0x80
__x64_sys_ioctl+0x16/0x20
do_syscall_64+0x4e/0x110
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7fb573206b57
Code: 00 00 00 48 8b 05 29 53 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 52 >
RSP: 002b:00007ffd291dfc58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000d67280 RCX: 00007fb573206b57
RDX: 00007ffd291dfc60 RSI: 0000000000008914 RDI: 0000000000000009
RBP: 0000000000001043 R08: 00007ffd291dfc6f R09: 00007ffd291f00b0
R10: 00007ffd291dfcb0 R11: 0000000000000246 R12: 0000000000001002
R13: 0000000000000000 R14: 00007ffd291dfc60 R15: 0000000000d693c0
iwlwifi 0000:3b:00.0: Collecting data: trigger 2 fired.
iwlwifi 0000:3b:00.0: Firmware not running - cannot dump error
~

I have also observed similar crashes on an older Dell Latitude laptop, but they occur rarely on that machine whereas the Precision laptop crashes reliably every time I boot with the newer firmware.

I am using nixos-unstable and Linux kernel version 5.2.8 (the latest one).

[peti]

ping @eadwu

[eadwu]

Haven't had any errors on my XPS (if it even uses any of the firmware) though there is a commit [1] upstream that updates iwlwifi binaries tagged after the release though I don't know whether it'll help or not.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=f9b0071de99f29159718d3c67030256799901a2b

[peti]

I tried that newer version, too, and it made no difference, i.e. it still crashes my kernel driver.

[eadwu]

Looks like there are (or were) problems with iwlwifi beforehand [1]. Judging from that post I'm guessing 20190618 should be latest revision where everything should be working fine.

[1] https://bugs.archlinux.org/task/63117

[contrun]

A new version of firmware has been released. This issue will be hopefully solved I didn't try it yet. https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tag/?h=20190815

[peti]

According to https://bugs.launchpad.net/hwe-next/+bug/1834415, the fix for this issues needs kernel patches:

• https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/commit/?id=f86c47462c24135dc2cce72559629ebca8be2c7a
• https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/commit/?id=33693c2f3ae9eafcfd69d2c2977dcc8cbaf199fc

[eadwu]

The first should've already been in 5.2.8 already. Can't seem to find anything about the second but has it been tested?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=437fde6cda74bb31705a9f37f14d481cdd431ad8

[peti]

The issue is gone on my laptop after updating to Linux 5.2.13.

[fpletz]

Since even all release branches are now on 5.2.15 I think we can close this. Thanks!

[madjam002]

I've just upgraded my 19.09 system which uses linuxPackages_latest and switched onto Kernel 5.4.6 and I'm getting the exact same error above. I've imported linuxPackages_5_3 from unstable whilst leaving the rest of my system on 19.09 and it's working fine now.

[frio]

I'm getting the same issue. So far I've tried combinations of:

• hardware.enableRedistributableFirmware (doesn't seem to make a difference)
• kernel 4.19 (doesn't work), 5.4 (doesn't work), 5.5 (doesn't work), 5.6-rc2 (doesn't compile)
• trying to backport iwlwifi patches to kernel's 5.4 and 5.5 (doesn't apply, the changes listed in the backported patch at https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/commit/?id=895fe853f8ad30d8ac8460c02cf2d2943360fda4 seem to have already been applied)

I'm at a bit of a loose end now, as 5.3 has been removed from nixpkgs-unstable, and doesn't appear to be available in 19.09. I guess I'll have to dig some commits up and define a custom kernel to give it a go. If I get that working I'll leave a comment for future googlers.

[frio]

Right, it seems like one of (several, but the others are applied) kernel patches is currently missing in 5.4.22 upstream.

This can be observed by checking the revision of the card with lspci and observing that it's the 201NGW. Per the various patches flying around (eg. https://www.mail-archive.com/[email protected]/msg59299.html), this card should receive the c0 firmware (iwlwifi-Qu-c0-hr-b0-50.ucode), not the b0 firmware (iwlwifi-Qu-b0-hr-b0-50.ucode) -- but if you check dmesg, it's receiving the b0 firmware on kernels 5.4 and 5.5 under NixOS. So, we need the applicable patch; if you naively rename the c0 firmware to b0, there's a test somewhere that checks the firmware size and fails.

I've picked 5.4 as the target kernel rather than 5.5, as it's the stable kernel for 19.09. I grabbed the patch from Arch, but it's from somewhere upstream; there was just a useful thread on their forums. The other two patches seem to already be in 5.4.22, but the first patch is (seemingly) missing:

  linux_lts_iwlwifiPatched = let old = pkgs.linux_5_4; in old.override {
    # passthru = old.passthru;
    modDirVersionArg = "5.4.22";
    kernelPatches = old.kernelPatches ++ [
        {
          name = "wifi-patch-1";
          patch = ./patches/fd1d146698d09aaac6709ae4fde90d80dbd153cd.patch;
        }
      ];
    };

    linuxPackages_lts_iwlwifiPatched = pkgs.recurseIntoAttrs (pkgs.linuxPackagesFor linux_lts_iwlwifiPatched);
  };

The modDirVersionArg might not be necessary, but I had to build this on an old machine and after the two hour build, I really didn't want to fuck with it.

Once that's in, there's also maybe a need to grab newer firmware; there's lots of other threads floating around about firmware versions. For the record, I seem to have everything working with the -48 that's in firmware-linux-nonfree, but from all the other error reports out there, some people need the newer version. The -50 firmware seems to be missing (even upstream in https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/). You've got to grab it from from https://chromium.googlesource.com/chromiumos/third_party/linux-firmware/+/1202eeada8fa518772191bab3e643d13566402a5 (note the -50 on iwlwifi-Qu-c0-hr-b0-50.ucode), courtesy of the notes at https://gitlab.com/emrose/xps13-7390_debian/issues/5#note_240886447.

So, for now, based on someone adding firmware to linux-firmware-nonfree at https://discourse.nixos.org/t/trying-to-include-custom-firmware-but-it-doesnt-appear-under-run-current-system/3044 and some other similar code at https://github.com/NixOS/nixos-hardware/blob/master/dell/xps/13-9360/default.nix, I've got:

  hardware.firmware = [
    (pkgs.runCommandNoCC "add-new-Qu-firmware" {} ''
      mkdir -p $out/lib/firmware
      cp ${./iwlwifi-Qu-c0-hr-b0-50.ucode} $out/lib/firmware/iwlwifi-Qu-c0-hr-b0-50.ucode
    '')
  ];

... and my wifi is working. Hope this helps whoever stumbles on this issue next. I'll raise some other tickets/PRs/etc. to try and get this into the nixos-hardware repo, and I'll check on what's happening in the kernel mailing lists as to why this patch is missing from the later 5.4.x kernels. I'm also unsure why the requisite firmware isn't in the mainline nonfree repo, but I'll leave that to greater minds.