Nixpkgs: Kernel oops with 4.19.36 at boot

[ambrop72]

Issue description

After updating from NixOS 18.09 / kernel 4.19.34 to NixOS 19.03 / kernel 4.19.36, the kernel oops'es at boot resulting in udevd crash and boot hanging.

Technical details

Hardware: Lenovo Legion Y720
Googling turned up this: https://marc.info/?l=linux-input&m=154081045228829&w=2
Error message:

apr 23 23:28:02 nixos kernel: i2c_hid i2c-ITE33D1:00: i2c-ITE33D1:00 supply vdd not found, using dummy regulator
apr 23 23:28:02 nixos kernel: i2c_hid i2c-ITE33D1:00: Linked as a consumer to regulator.0
apr 23 23:28:02 nixos kernel: i2c_hid i2c-ITE33D1:00: i2c-ITE33D1:00 supply vddl not found, using dummy regulator
apr 23 23:28:02 nixos kernel: BUG: unable to handle kernel paging request at 00000d6100000000
apr 23 23:28:02 nixos kernel: PGD 0 P4D 0 
apr 23 23:28:02 nixos kernel: Oops: 0000 [#1] SMP NOPTI
apr 23 23:28:02 nixos kernel: CPU: 7 PID: 704 Comm: systemd-udevd Tainted: P           O      4.19.36 #1-NixOS
apr 23 23:28:02 nixos kernel: Hardware name: LENOVO 80VR/LNVNB161216, BIOS 4GCN31WW 09/27/2017
apr 23 23:28:02 nixos kernel: RIP: 0010:strcmp+0x4/0x20
apr 23 23:28:02 nixos kernel: Code: 09 48 83 c2 01 80 3a 00 75 f7 48 83 c6 01 0f b6 4e ff 48 83 c2 01 84 c9 88 4a ff 75 ed f3 c3 0f 1f 80 00 00 00 00 48 83 c7 01 <0f> b6 47 ff 48 83 c6 01 3a 46 ff 75 07 84 c0 75 eb 31 c0 c3 19 c0
apr 23 23:28:02 nixos kernel: RSP: 0018:ffffb66d0288f748 EFLAGS: 00010202
apr 23 23:28:02 nixos kernel: RAX: 0000000000000000 RBX: ffffffffc046fbd0 RCX: 0000000000000000
apr 23 23:28:02 nixos kernel: RDX: ffffb66d0288f76e RSI: ffffffffc046fbe1 RDI: 00000d6100000001
apr 23 23:28:02 nixos kernel: RBP: 0000000000000000 R08: ffff8ce17ebdaca8 R09: ffffb66d0288f76e
apr 23 23:28:02 nixos kernel: R10: ffffb66d0288f6d0 R11: 000000000000038f R12: 00000000fffffffe
apr 23 23:28:02 nixos kernel: R13: ffffffffc046fbe1 R14: ffff8ce1701d2000 R15: ffff8ce1701d2000
apr 23 23:28:02 nixos kernel: FS:  00007f85d6537cc0(0000) GS:ffff8ce17ebc0000(0000) knlGS:0000000000000000
apr 23 23:28:02 nixos kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
apr 23 23:28:02 nixos kernel: CR2: 00000d6100000000 CR3: 00000004b269a006 CR4: 00000000003606e0
apr 23 23:28:02 nixos kernel: Call Trace:
apr 23 23:28:02 nixos kernel:  dmi_matches+0x8c/0xc0
apr 23 23:28:02 nixos kernel:  dmi_first_match+0x26/0x40
apr 23 23:28:02 nixos kernel:  i2c_hid_get_dmi_i2c_hid_desc_override+0x16/0x40 [i2c_hid]
apr 23 23:28:02 nixos kernel:  i2c_hid_probe+0x391/0x7c0 [i2c_hid]
apr 23 23:28:02 nixos kernel:  ? acpi_device_wakeup_disable+0x15/0x50
apr 23 23:28:02 nixos kernel:  ? i2c_hid_start+0x130/0x130 [i2c_hid]
apr 23 23:28:02 nixos kernel:  i2c_device_probe+0x1e7/0x260 [i2c_core]
apr 23 23:28:02 nixos kernel:  really_probe+0x262/0x3d0
apr 23 23:28:02 nixos kernel:  ? __driver_attach+0x100/0x100
apr 23 23:28:02 nixos kernel:  driver_probe_device+0x10f/0x120
apr 23 23:28:02 nixos kernel:  ? __driver_attach+0x100/0x100
apr 23 23:28:02 nixos kernel:  bus_for_each_drv+0x66/0xb0
apr 23 23:28:02 nixos kernel:  __device_attach+0xd9/0x150
apr 23 23:28:02 nixos kernel:  bus_probe_device+0x8a/0xa0
apr 23 23:28:02 nixos kernel:  device_add+0x48f/0x660
apr 23 23:28:02 nixos kernel:  i2c_new_device+0x15f/0x350 [i2c_core]
apr 23 23:28:02 nixos kernel:  i2c_acpi_register_device+0x20/0x60 [i2c_core]
apr 23 23:28:02 nixos kernel:  i2c_acpi_add_device+0x69/0x70 [i2c_core]
apr 23 23:28:02 nixos kernel:  acpi_ns_walk_namespace+0xda/0x1aa
apr 23 23:28:02 nixos kernel:  ? i2c_acpi_register_device+0x60/0x60 [i2c_core]
apr 23 23:28:02 nixos kernel:  ? i2c_acpi_register_device+0x60/0x60 [i2c_core]
apr 23 23:28:02 nixos kernel:  acpi_walk_namespace+0x9c/0xcb
apr 23 23:28:02 nixos kernel:  i2c_acpi_register_devices+0x40/0x60 [i2c_core]
apr 23 23:28:02 nixos kernel:  i2c_register_adapter+0x168/0x400 [i2c_core]
apr 23 23:28:02 nixos kernel:  i2c_dw_probe+0x3e5/0xe30 [i2c_designware_core]
apr 23 23:28:02 nixos kernel:  dw_i2c_plat_probe+0x30d/0x6c0 [i2c_designware_platform]
apr 23 23:28:02 nixos kernel:  platform_drv_probe+0x3a/0x90
apr 23 23:28:02 nixos kernel:  ? driver_sysfs_add+0x75/0xd0
apr 23 23:28:02 nixos kernel:  really_probe+0x262/0x3d0
apr 23 23:28:02 nixos kernel:  driver_probe_device+0x10f/0x120
apr 23 23:28:02 nixos kernel:  __driver_attach+0xdc/0x100
apr 23 23:28:02 nixos kernel:  ? driver_probe_device+0x120/0x120
apr 23 23:28:02 nixos kernel:  bus_for_each_dev+0x76/0xc0
apr 23 23:28:02 nixos kernel:  bus_add_driver+0x161/0x260
apr 23 23:28:02 nixos kernel:  ? 0xffffffffc0474000
apr 23 23:28:02 nixos kernel:  driver_register+0x5b/0xe0
apr 23 23:28:02 nixos kernel:  ? 0xffffffffc0474000
apr 23 23:28:02 nixos kernel:  do_one_initcall+0x46/0x1c8
apr 23 23:28:02 nixos kernel:  ? _cond_resched+0x15/0x40
apr 23 23:28:02 nixos kernel:  ? kmem_cache_alloc_trace+0x153/0x1c0
apr 23 23:28:02 nixos kernel:  do_init_module+0x5b/0x202
apr 23 23:28:02 nixos kernel:  load_module+0x20c9/0x27a0
apr 23 23:28:02 nixos kernel:  ? __do_sys_init_module+0x159/0x190
apr 23 23:28:02 nixos kernel:  ? _cond_resched+0x15/0x40
apr 23 23:28:02 nixos kernel:  __do_sys_init_module+0x159/0x190
apr 23 23:28:02 nixos kernel:  do_syscall_64+0x4e/0x100
apr 23 23:28:02 nixos kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
apr 23 23:28:02 nixos kernel: RIP: 0033:0x7f85d6acbc2a
apr 23 23:28:02 nixos kernel: Code: 48 8b 0d 59 c2 0b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 26 c2 0b 00 f7 d8 64 89 01 48
apr 23 23:28:02 nixos kernel: RSP: 002b:00007ffdcfb5cde8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af
apr 23 23:28:02 nixos kernel: RAX: ffffffffffffffda RBX: 00005588f1002d70 RCX: 00007f85d6acbc2a
apr 23 23:28:02 nixos kernel: RDX: 00007f85d6c238ad RSI: 00000000000041e8 RDI: 00005588f1039220
apr 23 23:28:02 nixos kernel: RBP: 00007f85d6c238ad R08: ffffffffffffffe0 R09: 0000000000004c20
apr 23 23:28:02 nixos kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f1039220
apr 23 23:28:02 nixos kernel: R13: 00005588f0ff7bb0 R14: 0000000000020000 R15: 0000000003938700
apr 23 23:28:02 nixos kernel: Modules linked in: arc4 nvidia_drm(PO+) nvidia_modeset(PO) nvidia_uvm(O) snd_hda_codec_hdmi iwlmvm i915 mac80211 i2c_designware_platform(+) i2c_designware_core intel_wmi_thunderbolt iTCO_wdt wmi_bmof intel_pmc_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvmgt vfio_mdev mdev vfio_iommu_type1 iwlwifi vfio uvcvideo kvm videobuf2_vmalloc videobuf2_memops btusb videobuf2_v4l2 btrtl btbcm btintel cec bluetooth irqbypass deflate nls_iso8859_1 videobuf2_common cfg80211 crct10dif_pclmul drm_kms_helper efi_pstore pstore crc32_pclmul snd_hda_intel videodev nls_cp437 ghash_clmulni_intel r8169 vfat intel_cstate snd_hda_codec media realtek ecdh_generic intel_uncore joydev evdev mousedev nvidia(PO) fat mac_hid intel_rapl_perf psmouse drm serio_raw efivars snd_hda_core rfkill mei_me
apr 23 23:28:02 nixos kernel:  libphy intel_gtt idma64 virt_dma i2c_i801 snd_hwdep mei i2c_algo_bit agpgart intel_lpss_pci intel_pch_thermal intel_lpss fb_sys_fops syscopyarea sysfillrect sysimgblt i2c_hid tpm_crb battery tpm_tis pinctrl_sunrisepoint tpm_tis_core tpm pinctrl_intel acpi_pad rng_core button video wmi ac pcc_cpufreq ipmi_devintf ipmi_msghandler i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore loop cpufreq_powersave vboxnetflt(O) vboxnetadp(O) vboxdrv(O) efivarfs ip_tables x_tables ipv6 crc_ccitt autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto dm_crypt hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sd_mod ahci xhci_pci libahci input_leds led_class xhci_hcd libata atkbd libps2 nvme usbcore aesni_intel scsi_mod aes_x86_64 crypto_simd cryptd glue_helper nvme_core crc32c_intel
apr 23 23:28:02 nixos kernel:  usb_common i8042 rtc_cmos serio dm_mod
apr 23 23:28:02 nixos kernel: CR2: 00000d6100000000
apr 23 23:28:02 nixos kernel: ---[ end trace 805fdf4212e17958 ]---
apr 23 23:28:02 nixos kernel: psmouse serio1: synaptics: queried max coordinates: x [..5660], y [..4570]
apr 23 23:28:02 nixos kernel: RIP: 0010:strcmp+0x4/0x20
apr 23 23:28:02 nixos kernel: Code: 09 48 83 c2 01 80 3a 00 75 f7 48 83 c6 01 0f b6 4e ff 48 83 c2 01 84 c9 88 4a ff 75 ed f3 c3 0f 1f 80 00 00 00 00 48 83 c7 01 <0f> b6 47 ff 48 83 c6 01 3a 46 ff 75 07 84 c0 75 eb 31 c0 c3 19 c0
apr 23 23:28:02 nixos kernel: RSP: 0018:ffffb66d0288f748 EFLAGS: 00010202
apr 23 23:28:02 nixos kernel: RAX: 0000000000000000 RBX: ffffffffc046fbd0 RCX: 0000000000000000
apr 23 23:28:02 nixos kernel: RDX: ffffb66d0288f76e RSI: ffffffffc046fbe1 RDI: 00000d6100000001
apr 23 23:28:02 nixos kernel: RBP: 0000000000000000 R08: ffff8ce17ebdaca8 R09: ffffb66d0288f76e
apr 23 23:28:02 nixos kernel: R10: ffffb66d0288f6d0 R11: 000000000000038f R12: 00000000fffffffe
apr 23 23:28:02 nixos kernel: R13: ffffffffc046fbe1 R14: ffff8ce1701d2000 R15: ffff8ce1701d2000
apr 23 23:28:02 nixos kernel: FS:  00007f85d6537cc0(0000) GS:ffff8ce17ea00000(0000) knlGS:0000000000000000
apr 23 23:28:02 nixos kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
apr 23 23:28:02 nixos kernel: CR2: 00007f90c1f00130 CR3: 00000004b269a005 CR4: 00000000003606f0
apr 23 23:28:02 nixos systemd-udevd[670]: worker [704] terminated by signal 9 (KILL)
apr 23 23:28:02 nixos systemd-udevd[670]: worker [704] failed while handling '/devices/pci0000:00/0000:00:15.2/i2c_designware.1'

[aethelz]

Same problem after upgrade on Acer Swift 3 SF314-56-5403

[kuznero]

Same or similar problem on Dell XPS :(

[markuskowa]

Maybe related: https://github.com/NixOS/nixpkgs/issues/60158

Might be related to this commit in linux 4.19.36:
https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19.36

commit 8e271100efa63ef6dc13c22e68c91260d4c83614
Author: Julian Sax <[email protected]>
Date:   Wed Sep 19 11:46:23 2018 +0200

    HID: i2c-hid: override HID descriptors for certain devices

    [ Upstream commit 9ee3e06610fdb8a601cde59c92089fb6c1deb4aa ]

    A particular touchpad (SIPODEV SP1064) refuses to supply the HID
    descriptors. This patch provides the framework for overriding these
    descriptors based on DMI data. It also includes the descriptors for
    said touchpad, which were extracted by listening to the traffic of the
    windows filter driver, as well as the DMI data for the laptops known
    to use this device.

    Relevant Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1526312

CC @NeQuissimus

[ambrop72]

Indeed it is causes by that commit. I have applied the patch and it solves the problem for me. I already emailed the LKML and Greg KH, who has already queued up the fix for the stable trees.

[kuznero]

@ambrop72, is there a way to work it around in my configuration.nix somehow?

[ambrop72]

@ambrop72, is there a way to work it around in my configuration.nix somehow?

Add the patch from my merge request to boot.kernelPatches (and put the patch itself in the same directory).

[kuznero]

Trying it out now. Will post an update when done.

[kuznero]

Worked out great! Thanks, @ambrop72!

[juaningan]

Solved on Dell XPS 13 (9343). Thanks!!

[NeQuissimus]

The patch is in master, 18.09 and 19.03 now. Please close this ticket if things are working now.
I will remove the patch with the round of kernel updates that bring this patch natively.

[kionka]

I also hit this on my Dell XPS. I'm surprised more people haven't experienced this. This was a nightmare to fix since when I tried to use an earlier build, the wifi refused to connect. I ended up looking through the issues. It's surprisingly hard to get this working without a mouse, so I just want to document more explicitly how to fix this.

I tried using the nixos-unstable and the 19.03 channel, but it must not have made it in yet.

Add this to your /etc/nixos/configuration.nix:

  boot.kernelPatches = [
    {
      name = "i2c-oops";
      patch = ./i2c-oops.patch;
    }
  ];

And put this file in /etc/nixos/i2c-oops.patch:

diff --git a/drivers/hid/i2c-hid/i2c-hid-dmi-quirks.c b/drivers/hid/i2c-hid/i2c-hid-dmi-quirks.c
index 1d645c9ab417bf..cac262a912c124 100644
--- a/drivers/hid/i2c-hid/i2c-hid-dmi-quirks.c
+++ b/drivers/hid/i2c-hid/i2c-hid-dmi-quirks.c
@@ -337,7 +337,8 @@ static const struct dmi_system_id i2c_hid_dmi_desc_override_table[] = {
            DMI_EXACT_MATCH(DMI_PRODUCT_NAME, "FlexBook edge11 - M-FBE11"),
        },
        .driver_data = (void *)&sipodev_desc
-   }
+   },
+   { } /* Terminate list */
 };

Or you can download it from here: https://raw.githubusercontent.com/ambrop72/nixpkgs/4474b90989643d84be2812da5e308c2d97bf74e6/pkgs/os-specific/linux/kernel/i2c-oops.patch

Which comes from this PR as mentioned above: https://github.com/NixOS/nixpkgs/pull/60172

[snaar]

As an alternative workaround with less typing, try 5.0 kernel by adding "boot.kernelPackages = pkgs.linuxPackages_5_0;" to your configuration.nix - worked for me on XPS.

[domenkozar]

4.19.37 is out

[markuskowa]

The patch is nicely commented by Linus himself :)

From the 4.19.37 changelog:

commit 9c1862566176250bae343a5cee617a5ce41efa54
Author: Linus Torvalds <[email protected]>
Date:   Sat Oct 27 09:10:48 2018 -0700

    i2c-hid: properly terminate i2c_hid_dmi_desc_override_table[] array

    commit b59dfdaef173677b0b7e10f375226c0a1114fd20 upstream.

    Commit 9ee3e06610fd ("HID: i2c-hid: override HID descriptors for certain
    devices") added a new dmi_system_id quirk table to override certain HID
    report descriptors for some systems that lack them.

    But the table wasn't properly terminated, causing the dmi matching to
    walk off into la-la-land, and starting to treat random data as dmi
    descriptor pointers, causing boot-time oopses if you were at all
    unlucky.

    Terminate the array.

    We really should have some way to just statically check that arrays that
    should be terminated by an empty entry actually are so.  But the HID
    people really should have caught this themselves, rather than have me
    deal with an oops during the merge window.  Tssk, tssk.

[ambrop72]

Indeed, and it was caught before the original patch was merged to mainline, but somehow the patch got into the stable releases without the fix.

[ambrop72]

This is solved, kernel updates (e.g. 4.19.37) got deployed to 19.03 channel at least.

[domenkozar]

Fun fact: I actually bought a mouse due to this bug. There's a known issue in Dell XPS battery growing in size damaging trackpad, so I thought that's what happened.