Nixpkgs: Raspberry Pi 3 B+ doesn't boot on kernel 5.4.23

[ertw]

Describe the bug
Following the ARM guide in the wiki, I attempted to boot using boot.kernelPackages = pkgs.linuxPackages_latest; After fiddling around with my config for a while, I eventually discovered that I could get it to boot with boot.kernelPackages = pkgs.linuxPackages_4_19;

Using 5.4.23, I'm able to select which generation to boot into, but selecting one with 5.4.23 never gets past Starting kernel ...

I'm able to press ctrl-alt-delete at that stage to reboot and successfully boot into a generation with 4.19.107.

To Reproduce
Steps to reproduce the behavior:

1. Follow the wiki guide, and copy the recommended default config for Raspberry Pi 3
2. Run nixos-rebuild switch and reboot

Expected behavior
Boot into latest kernel

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"aarch64-linux"`
 - host os: `Linux 4.19.107, NixOS, 19.09.2201.7d31bbceaa1 (Loris)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.3`
 - channels(root): `"nixos-unstable-20.09pre215947.82b54d49066, nixos-19.09.2201.7d31bbceaa1"`
 - channels(ertw): `"nixpkgs-unstable-20.09pre215991.93ba4ecd586"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[ertw]

I've been continuing to try to get this working, and am still unable to do so. One thing I noticed was that I was on unstable, so I switched back to stable, and then tried boot.kernelPackages = pkgs.linuxPackages_latest; again. Still no luck though, and boot never gets past Starting kernel.... I've attached a photo of the display output in case that is helpful.

Interestingly it isn't totally hung here- I can press ctrl-alt-delete to reboot, and then select a previous generation and boot into it fine. Is there any way for me to get additional debug output at this point to better understand where it is getting stuck?

What I've tried so far:

• set boot.consoleLogLevel = 7;, and this gives the output seen above.
• run journalctl -b, but I only see messages from the current, successfully booted generation, not the previous one that did not boot.

[1000101]

Same problem here. Doesn't boot on 5.4.28 either, only 4.19.x works.

[mebubo]

5.5.13 from the today's nixos-unstable (ae6bdcc5358) doesn't work for me either, hangs at "Starting kernel ..." on my Raspberry 3 B+.

The only usable kernel at the moment is 4.19.113. I have to use it with the nixos-19.09 (6d445f8398d), as it is the only channel where 4.19 is pre-built.

[mebubo]

It turns out that for me the 5.5.13 _does_ boot, it just doesn't initialize hdmi (kudos to Valodim from #nixos-aarch64 for the tip).

Don't know how to fix the hdmi problem yet.

[matthewbauer]

I think https://github.com/NixOS/nixpkgs/issues/66960 might be related. /cc @samueldr

[samueldr]

Only if it is from an update on a system using an older u-boot release. #66960 is only when using a u-boot release that doesn't know about the Raspberry Pi 3B+, which would use the Raspberry Pi 3B(not plus) device tree.

Briefly said, if upgrading from a 19.03 system, #66960 is relevant, otherwise it probably isn't.

[samueldr]

Using the following unmodified image:

• https://hydra.nixos.org/build/115752495

I was able to successfully boot to the VT prompt, on a Raspberry Pi 3B (non-plus)

Fellow users, can you report the status with this image, by stating conclusively which Raspberry Pi board you are using?

I wonder if the display being used is also an issue. Some marginal behaviour in the kernel related to the use of your particular display with the vc4 modesetting driver.

I have tested using a Nexdock as the display.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/go-no-go-meeting-nixos-20-03-markhor/6495/16

[samueldr]

For now this issue has been tentatively documented here:

• https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi#HDMI_output_issue_with_kernel_5.4_.28NixOS_20.03_or_NixOS_unstable.29

[leonardp]

Using the following unmodified image:

* https://hydra.nixos.org/build/115752495

I was able to boot this image with a Pi 3B+ using hdmi output with a desktop monitor.
Bluetooth was complaining a lot after booting and i couldn't get a serial terminal.

[evils]

i can confirm that my Raspberry Pi 3B+ with build 115752495 boots

there's some odd stuff going on with video output though

• if connected to an active hdmi to vga adapter and then to power, it shows the entire boot process
• if connected to a monitor via a passive hdmi to dvi adapter and then to power, the display gets stuck at displaying starting kernel..., but it does boot
• if power is applied (for >1 minute for in this case), and then hdmi directly to an hdmi tv, it starts showing its boot from the start, i can't confirm if it was booted already
• if power is applied, and nothing else, then a keyboard, sshd enabled and a password set (blindly), and then ethernet, i can log in via ssh
  
  
  
  • no video output is available on either hdmi to dvi or hdmi to vga, i did not try it on the hdmi tv
  
  

[leonardp]

Using the following unmodified image:

* https://hydra.nixos.org/build/115752495

to get a proper serial output from this image using a pi 3b+ i needed to:
edit the "APPEND" parameter in NIXOS_SD/boot/extlinux/extlinux.conf so that the
only console argument is

console=ttyS1,115200n8

without doing that, the serial console stops it's output after "Starting Kernel".
i also disabled the bluetooth overlay by adding dtoverlay=disable-bt to FIRMARE/config.txt
however i don't think that is strictly necessary

plugging in HDMI before booting works but plugging it in after booting gave "Input not Supported".

[samueldr]

Plugging-in HDMI after booting I would assume was never supported by mainline, for the VT at least. Though that's a baseless assumption.

---

It does look like it is setup-dependent, which makes this hard to test.

[Valodim]

I downloaded the image from hydra mentioned above and tried it in two Rpi3B (non-plus) several times, with oddly different results:

• I tried it with HDMI on an LG projector first, no adapter. The first pi got stuck at "Starting kernel...", the second cut off output right after the same message.
• I flashed the sd card again, and tried the same two Pis first on a Samsung monitor, and then an LG one. All those configurations worked as expected.
• I went back to the LG projector, but used the HDMI cable from the monitors before to exclude that source of errors. Everything worked as expected.
• I went back to the LG projector with the original HDMI cable - and everything still worked as expected.

It's possible the errors were caused flashing the sd card for the first time - I have no idea. I now can't reproduce the error state anymore, it just seems to work on the Rpi3B.

[leonardp]

and if i understand correctly the firmware package supplied with nixpkgs is 1.20190925 while 1.20200212 is available.
i think this could be a potential candidate for the observed issues as a mode change occurs after the kernel is initialized (don't quote me on that).
maybe upgrading the raspberrypi firmware could resolve this?

[mebubo]

Observation:

• if I reboot into a 5.4 from a powered-up state (either by unplugging the power for a second, or by issuing a reboot command), the display always gets stuck on 'Starting kernel...' (but it actually boots fine)
• if I unplug the pi, let it and the power supply cool down, and then boot directly into 5.4, the HDMI is initialized correctly

The above is also true for 5.5 and 5.6 from nixos-unstable.

Perhaps this issue may have something to do with power, temperature, dynamic frequency scaling etc.

[Valodim]

In my weird case (see my last comment), the status went from not working "cold" to working after a while. The previous comment led me to test the setup again. Everything else being exactly the same, booting and hdmi still all works as expected. Leads me to believe I did something wrong on the first test and things are generally working again on the Rpi 3B non-plus

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/go-no-go-meeting-nixos-20-03-markhor/6495/19

[lloeki]

Raspberry Pi B failing to display anything past Starting Kernel here on a LG TV. The above suggested image did not work. Tried multiple HDMI cables, to no avail.

Steps:

• download latest 20.03 img as of today from Hydra
• diskutil list, diskutil unmountdisk disk5
• sudo dd if=nixos-sd-image-20.03.1879.91cdcf31357-aarch64-linux.img of=/dev/rdisk5 bs=1m
• diskutil eject disk5
• mount root fs on another machine able to mount ext4 RW (I'm on macOS, so I used a linux VM and passed the USB card reader to VMware Fusion)
• mkdir /mnt/path/home/nixos/.ssh && chmod 700 /mnt/path/home/nixos/.ssh
• added my pubkeys to /mnt/path/home/nixos/.ssh/authorized_keys and chmod 600 /mnt/path/home/nixos/.ssh/authorized_keys
• boot sd card on Pi while plugged via ethernet, wait ~1min
• find IP via router's dhcp client table (alternatively, scan with nmap for closed but unfiltered port 22)
• blindly press enter a couple of times, then sudo systemctl start sshd
• wait a bit, then attempt ssh nixos@<ip>

It's supposed to be a headless system ultimately, so not too bothersome, but typing thinks blindly when you're unfamiliar with nix can leave one dumbfounded (I ventured the default auto-login user was root so missed sudo)

[lloeki]

Has anyone tried kernel flags such as nomodeset to rule out any kms framebuffer issue? (still getting the hang of NixOS so can't try this just yet).

[lloeki]

Tried nomodeset, unsuccessfully.

[matthewbauer]

@lloeki If Linux 5.x isn't important, you might want to try Nixiosk:

https://github.com/matthewbauer/nixiosk/

It should at least get you ssh + mdns so you can ssh via root@<hostname>.

[lloeki]

Thanks, but I did manage to get in and setup what I needed, as I can live with headless and 5.x.

I did go down to 4.19 ultimately though, but that was because wlan0 wasn't showing up at all as an interface and wanted to check if it did on 4.19 (and it does show up there, but wifi performance is bad with dropped packets and wildly varying latencies, which is something I had on ArchLinux a long time ago but not on RetroPie 4.5.1 where it was perfect).

[lafka]

Can confirm no HDMI display RPI 3B+ with build 122365321.

Booting up with serial console (first setting console=ttyS1,115200n8 in /boot/extlinux/extlinux.conf)
Setting up minimal configuration and reverting to 4.19 kernel works:

{ config, pkgs, lib, ... }:
{
  boot.loader.grub.enable = false;
  boot.loader.generic-extlinux-compatible.enable = true;

  boot.kernelPackages = pkgs.linuxPackages_4_19;

  boot.kernelParams = ["cma=32M"];

  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
}

Going back to latest kernel does not have any effect. Connected to VGA on monitor, HDMI adapter used on rpi.

One question, what power supply do you use? I'm using a 2.4A supply but still getting a few "undervoltage" messages on console, maybe related?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cant-boot-into-new-configuration-after-nix-rebuild-raspberry-pi3-b/7812/1

[mebubo]

I still reliably get a working display output after boot if I cool the board down to below ~18°C prior to boot, and no display output past "Starting kernel..." if the board is warmer than that at boot time %)

I compared dmesg in the 2 cases and there are multiple entries like this in the failure case but not in the success case:

bcm2835-power bcm2835-power: Timeout waiting for grafx power OK

It seems like it has to do with the upstream switching from a firmware-based driver to bcm2835-power in 5.1: https://github.com/raspberrypi/linux/issues/3046

The revert https://github.com/raspberrypi/linux/issues/3046#issuecomment-607982630 seems to be applied upstream in 5.8-rc1 and 5.7.6.

Unfortunately, at the moment 5.7 completely hangs at boot on my 3B+ (green LED stops flashing) (but for now I only tried 5.7._4_, the latest in nixos-unstable).

[lloeki]

Looks like there could be a workaround

arm_freq=1200, no cpufreq => no timeout

https://github.com/raspberrypi/linux/issues/3046#issuecomment-526678860

EDIT: tested by adding arm_freq=1200 and blacklisting raspberrypi_cpufreq module, still no display.

[lloeki]

I tried a shot at this one and HDMI spawned to life.

For an immediate fix, here's what I did. Use at your own risk.

curl -O https://github.com/raspberrypi/linux/files/3801365/bcm2837-rpi-3-b.dtb.zip
echo '660e6bc6bb652257f969db60d048670a34456ebf78ba4d76ed6c6daeacea8614  bcm2837-rpi-3-b.dtb.zip' | sha256sum --check
unzip bcm2837-rpi-3-b.dtb.zip
cat /boot/extlinux/extlinux.conf | grep -A5 'LABEL nixos-default' | grep FDTDIR | awk '{ print $2 }'
ls /boot/extlinux/$(cat /boot/extlinux/extlinux.conf | grep -A5 'LABEL nixos-default' | grep FDTDIR | awk '{ print $2 }')/broadcom/bcm2837-rpi-3-b.dtb
cat bcm2837-rpi-3-b.dtb > /boot/extlinux/$(cat /boot/extlinux/extlinux.conf | grep -A5 'LABEL nixos-default' | grep FDTDIR | awk '{ print $2 }')/broadcom/bcm2837-rpi-3-b.dtb 

The upstream fix was to revert a change that uses in-tree power management instead of the firmware one:

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20200402&id=e7b7daeb48e0bf5d8412d77f11069750ee7032bb

There is another (unrelated) issue with X hanging (I don't use X):

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20200504&id=b1e7396a1d0e6af6806337fdaaa44098d6b3343c

[lloeki]

Unfortunately, at the moment 5.7 completely hangs at boot on my 3B+ (green LED stops flashing) (but for now I only tried 5.7.4, the latest in nixos-unstable).

I have the same problem. Rolled back to 5.6.13.

[deifactor]

Another datapoint: I was able to get my 3B+ to boot fine using 5.4.50 (by setting boot.kernelPackages = pkgs.linuxPackages_5_4). Using 5.7 didn't work, though I'm not sure if it was a complete hang or just the 'boots but HDMI breaks' issue.

[renanqts]

Same issue here.

[benley]

Here as well: 5.7.x wouldn't boot, but 5.4.x and 5.6.x are both working. In my case it was _not_ a "boots but breaks HDMI" issue as the system in question is headless.

[ajsalminen]

My Model 3B also seemed to have this issue with 5.7 but not with 5.6.

[tobiasBora]

Hum, I can confirm like benley that the issue is not just a "boots but breaks HDMI" in my case: after some upgrade, it appears that I also can't boot anymore (since I don't reboot often, it quite surprising to lose complete ssh access). It hangs at "Starting kernel" while it starts the kernel 5.7.18, even if I configure in /config.txt:

enable_uart=1
force_turbo=1

and if I add in the console parameters console=ttyS1,115200n8. I installed this system quite a while ago, in 2018, and I was not having any issue so far. However, the kernel 5.6 is still booting fine. Will try to reinstall the new image to see if it helps.

[tobiasBora]

The new image still have an issue with kernel 5.7, so maybe it is a different bug. I filled a new bug https://github.com/NixOS/nixpkgs/issues/97064.