Nixpkgs: Raspberry Pi 3 B+ can't boot with latest kernel 5.7.19

[tobiasBora]

Describe the bug
Since my last kernel upgrade on the Raspberry Pi 3B+, I can't boot anymore (I lost ssh access). I tried to burn a new image from scrach, update the channel, and create the most basic configuration, and the problem persists.

To Reproduce
Steps to reproduce the behavior:

1. Download the Aarch64 sdcard image here
2. Unpack, and burn it with dd on a sdcard
3. Make sure that you have the following config.txt (enable_uart is useful to boot with uart, and force_turbo was required to have less noise in the characters read by uart):

kernel=u-boot-rpi3.bin

# Boot in 64-bit mode.
arm_control=0x200

# U-Boot used to need this to work, regardless of whether UART is actually used or not.
# TODO: check when/if this can be removed.
enable_uart=1

# Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
# when attempting to show low-voltage or overtemperature warnings.
avoid_warnings=1
force_turbo=1

1. Boot on this sdcard, and I recommend to use UART + screen to boot with a serial console
2. Configure internet, put the following file in /etc/nixos/configuration.nix:

{ config, pkgs, lib, ... }:
{
  # NixOS wants to enable GRUB by default
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;

  # !!! Otherwise (even if you have a Raspberry Pi 2 or 3), pick this:
  boot.kernelPackages = pkgs.linuxPackages_latest;

  # !!! Needed for the virtual console to work on the RPi 3, as the default of 16M doesn't seem to be enough.
  # If X.org behaves weirdly (I only saw the cursor) then try increasing this to 256M.
  # On a Raspberry Pi 4 with 4 GB, you should either disable this parameter or increase to at least 64M if you want the USB ports to work.
  boot.kernelParams = ["cma=32M"];

  # File systems configuration for using the installer's partition layout
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };

  # !!! Adding a swap file is optional, but strongly recommended!
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];
}

and then run:

sudo nix-channel --update
sudo nixos-rebuild switch

Reboot: the OS stops booting at Starting kernel ..., even if you add in the kernel parameters console=ttyS1,115200n8 or console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7 (via /boot/extlinux/extlinux.conf):

------------------------------------------------------------
1:      NixOS - Default
2:      NixOS - Configuration 2 (2020-09-04 00:54 - 20.03.2873.10aede9bcfe)
3:      NixOS - Configuration 1 (1970-01-01 00:00 - 20.03.2869.0d60b0b10ea)
Enter choice: 1:        NixOS - Default
Retrieving file: /boot/extlinux/../nixos/jkjjy892iirdhns5mxqx063xn374ph80-initrd-linux-5.7.19-initrd
7726240 bytes read in 322 ms (22.9 MiB/s)
Retrieving file: /boot/extlinux/../nixos/hmwv1a4bnlshi1zlnzsk83sg41ybaf73-linux-5.7.19-Image
39021056 bytes read in 1614 ms (23.1 MiB/s)
append: systemConfig=/nix/store/jrmziij585fgr7n82dg3ff435s13gnhb-nixos-system-nixos-20.03.2873.10aede9bcfe init=/nix/store/jrmziij585fgr7n82dg3ff435s13gnhb-nixos-system-nixos-20.03.2873.10aede9bcfe/init cma=32M console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7
Retrieving file: /boot/extlinux/../nixos/hmwv1a4bnlshi1zlnzsk83sg41ybaf73-linux-5.7.19-dtbs/broadcom/bcm2837-rpi-3-b-plus.dtb
14622 bytes read in 6 ms (2.3 MiB/s)
## Flattened Device Tree blob at 02600000
   Booting using the fdt blob at 0x2600000
   Using Device Tree in place at 0000000002600000, end 000000000260691d

Starting kernel ...

Moreover, I think it is not a problem with UART, because on my first system I couldn't ssh my system that was working fine during 2 years, and the green LED does not blink during the expected boot period.

Expected behavior
I would expect it to boot properly.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I tried with the kernel 5.6 and it boots fine. I experienced issues with the kernel 5.7.18 and 5.7.19

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[Mic92]

I can confirm this issue. My current workaround is to use:

boot.kernelPackages = pkgs.linuxPackages_rpi3; # v4.19.118 on my system

or:

boot.kernelPackages = pkgs.linuxPackages; # at the time of writing v5.4.66

instead of:

boot.kernelPackages = pkgs.linuxPackages_latest; # at the time of writing v5.8.11

Note that hdmi does not display the boot for pkgs.linuxPackages but with pkgs.linuxPackages_rpi3, it works after a while however.

[lucernae]

Can also confirm this issue. My current workaround is to use:

boot.kernelPackages = pkgs.linuxPackages_5_4

then nixos-rebuild switch.

For anyone noobs like me, in the original issue after doing upgrade you lose access of the pi after upgrading to boot.kernelPackages = pkgs.linuxPackages_latest. You can't use keyboard and the boot are stuck to the default boot with Starting kernel... messages hanging via HDMI. SSH doesn't seems to work (maybe the pi hasn't boot at all). To recover access, I did the following:

1. My recovery pi setup consists of USB keyboard + HDMI + Ethernet. But sometimes your USB keyboard is undetected. Try to unplug and put it on other slot. Sometimes it works. When it works, immediately press any button to get into uboot. Proceed with the boot by typing boot, but then you get to choose which generation of Nixos you want to boot. Choose your last known configuration then, work from there to recover.

2. If above method fails, extract your SD card and mount it to any linux OS. However, I only have SD card reader over MacOS, but mac can't mount ext4 device. So I use live usb in my macos just to get into a linux OS. From there mount the sdcard manually:

mount -t ext4 /dev/disk/by-label/NIXOS_SD /mnt/pi

Where /mnt/pi is your mount point. From there, edit /mnt/pi/boot/extlinux/extlinux.conf and change the default boot to your last know configuration to recover.

I think for useful reason this notes can be included in the NixOS ARM wiki. It could be helpful for noobs like me out there. A better way to test new image would be to just use nixos-rebuild boot then manually choose the new generation at boot, so if it doesn't work, the boot will choose your working default setup.

[Mic92]

I fixed the wiki for the time being: https://nixos.wiki/wiki/NixOS_on_ARM#NixOS_installation_.26_configuration Maybe we should mark 5.7+ as broken on raspberry pi 4?

[rjpcasalino]

confirming that boot.kernelPackages = pkgs.linuxPackages_5_4 helped me work around this issue whereas the wiki still mentions: boot.kernelPackages = pkgs.linuxPackages;. I'm on a Pi 3. My machine would hang on
Starting kernel ...