Fenix: LineageOS netlink kernel panic on versions less than 16

[kolAflash]

Tested on three devices.
On two devices I can repeatedly reproduce the bug.
On the third the bug doesn't appear.
(see below for details)

 

Steps to reproduce

Install 2019-09-28 nightly from PlayStore by joining the beta. (tested fresh install and upgrade)
Then start the app.

 

Expected behavior

Everything is fine.

 

Actual behavior

Apps hangs for a moment (<10 seconds), then the phone reboots.
After reboot completes, the phone always reboots again after ~1 minute. (loop!)

 

Device information

• Android device: FairPhone 2
• OS: LineageOS-15.1 (2019-03-27)
• OS modifications: Google-Apps, Rooted, F-Droid Helper, XPosed
• first known broken Fenix version: nightly 2019-09-28
• last known working Fenix version: 2.0.0 (2019-09-17)
   
• Android device: Galaxy S5 LTE (G900F)
• OS: LineageOS-16.0 (lineage.microg.org, 2019-03-11)
• OS modifications: Rooted, F-Droid Helper (NO XPosed)
• Fenix version: nightly 2019-09-28
• bug info: bug doesn't appear automatically after reboot, but only when manually opening the app
   
• Android device: Nexus 5X (bullhead)
• OS: LineageOS-15.1 (lineage.microg.org, 2019-02-04)
• OS modifications: Rooted, F-Droid Helper, XPosed
• Fenix version: nightly 2019-09-28
• bug info: NO bug (works fine)

 

Workaround

Try to uninstall the app before the phone reboots again.

 

Logcat & Dmesg from the FairPhone 2

Received via ADB (USB). Probably something is missing at the end, because the phone reboots.
dmesg.txt
logcat.txt

[kolAflash]

I really hope this bug never gets into a productive version!
It's a worst case for many people!

It's still an issue in the PlayStore beta 2019-10-07.

But version 2.1.0 is fine.
https://github.com/mozilla-mobile/fenix/releases/tag/v2.1.0

[aNNiMON]

Sony Xperia Z1, Android 9 AICP, rooted, no XPosed: reboot after launching an app
Firefox Preview 2.2.0 works fine

[kolAflash]

Firefox-Preview-3.0.0 from PlayStore has the same problem!

Interesting / Workaround:
The bug doesn't appear if the phone is in airplane mode. (no mobile and no wifi connection)
And the bootloop is only being triggered after the SIM pin has been entered or if wifi is connected.

Version 2.3.0 was fine.
 

Question: Are there archived nightlies available?
I tested some nightlies from apkmirror.com. But they don't have all nightlies available.
Results:

• no crash: 190701 12:46
• no nightlies in between on apkmirror.org :-(
• crash: 190923 18:00
• crash: 191213 06:01
   

Same problem for "Mozilla Reference Browser".
Reported the bug upstream: https://github.com/mozilla-mobile/reference-browser/issues/969
https://play.google.com/store/apps/details?id=org.mozilla.reference.browser

[aNNiMON]

Confirm. 3.0.0 causes a reboot on Sony Xperia Z1, but works fine in Airplane mode without Wifi connection.

[loveless90]

The same on an Galaxy S4 mini (I9195/serranoltexx) with LineageOS 16.0 and without root (removed with "adddonsu-remove").

System crashes when background-process or fenix itself get started by the system (after entering pin and password) or the user . Haven't tested Airplane-mode because i removed fenix after i noticed the bug (system got unusable).

[snorp]

For those of you that can reproduce this crash, can you please look in the following location for logs, and send them to us if you're able?

/data/tombstones/
/data/dontpanic/
/data/system/dropbox/
/proc/last_kmsg
/sys/fs/pstore/console-ramoops

You'll need root access for most of these, but many of the crashes seem to be using a custom ROM, so hopefully that won't be a problem.

[snorp]

Also, if anyone with an affected device would like to try using mozregression for this, that would be very helpful.

https://mozilla.github.io/mozregression/quickstart.html

[csadilek]

@aNNiMON Can you share with us what OS and version you're using?

[aNNiMON]

@csadilek it was mentioned before:
Sony Xperia Z1, Android 9 AICP, rooted, no XPosed Framework installed
Kernel version 3.4.113

@snorp
/data/tombstones/ — empty
/data/dontpanic/ — not exists
/data/system/dropbox/ — [email protected]
/proc/last_kmsg — last_kmsg
/sys/fs/pstore/console-ramoops — not exists
Tested on 3.0.0 (Build #13431955) from Play Store

[kolAflash]

I got the bug also verified on a S4-Mini i9195, running LineageOS-15.1 (without root).
 

Looks like I can't test Fenix/Firefox-Preview directly via mozregression. But I was able to reproduce the crash using org.mozilla.geckoview_example (aka. gve) via mozregression (tested on my Fairphone 2). Actually the testing with gve was easier, because gve seems not to autorun on boot. So there's no auto-crash after boot (no bootloop) with gve.

$ mozregression --app gve --good 2019-03-11 --bad 2019-09-23 --mode classic
[...]
Last good revision: 461a6e9e829bcbb4ef83cc660f0fdfa98e7c6a75
First bad revision: 56e11fddf939f61bcfac9e179e13ff62a8ad5d82
************* Switching to mozilla-central
Unable to exploit the merge commit. Origin branch is mozilla-inbound, and the commit message for 56e11fdd was:
Merge mozilla-central to mozilla-inbound

 

I was also able to manually bisect on project.mobile.fenix.v2.nightly builds from taskcluster.
Last good revision: 2019-09-22 c866ae1a1 (armeabi-v7a apk)
First bad revision: 2019-09-23 cb7701584 (armeabi-v7a apk)
 

Find the requested files attached here. (tombstones, dropbox and last_kmsg - no other files existing)
kolaflash_fenix_issue-5663_logs.tar.xz.zip
 
 

I just wanted to add, that I'm very unhappy with the way this went!

The bug is known since nightly builds in September, I just had to tell my girlfriend via a landline phonecall how to fastly uninstall Firefox-Preview before her phone reboots for the 20th time.

And there are probably people out there resetting their phones right now, because they don't have a clue why their phones started bootlooping. Maybe people even throw their phones in the trash, if they reinstall Firefox-Preview after resetting, the phone starts bootlooping again and they don't get the connection.

Such a serious bug really shouldn't have made it into a stable (non-nightly) release!
Acutally #7128 has already been marked as a Release Blocker. So I think honestly Mozilla should withdraw the Firefox-Preview-3.0.0 update from PlayStore!
(return PlayStore to Firefox-Preview-2.3.0 until this is resolved)

[snorp]

Thank you for the logs and the bisecting, that was very helpful!

We can see now that the phone is crashing due to a kernel panic.

Process Netlink Monitor (pid: 23606, stack limit = 0xe543e2f0) <snip some stack stuff> [<c0aeb7a8>] (rt_fill_info.isra.29.constprop.33+0x22c/0x39c) from [<c0aed968>] (inet_rtm_getroute+0x214/0x250) [<c0aed968>] (inet_rtm_getroute+0x214/0x250) from [<c0aa0cc4>] (rtnetlink_rcv_msg+0x2b0/0x2ec) [<c0aa0cc4>] (rtnetlink_rcv_msg+0x2b0/0x2ec) from [<c0aba3bc>] (netlink_rcv_skb+0x58/0xb4) [<c0aba3bc>] (netlink_rcv_skb+0x58/0xb4) from [<c0a9ff20>] (rtnetlink_rcv+0x1c/0x28) [<c0a9ff20>] (rtnetlink_rcv+0x1c/0x28) from [<c0ab9df8>] (netlink_unicast+0x154/0x1f0) [<c0ab9df8>] (netlink_unicast+0x154/0x1f0) from [<c0aba1e8>] (netlink_sendmsg+0x2a0/0x320) [<c0aba1e8>] (netlink_sendmsg+0x2a0/0x320) from [<c0a7e714>] (sock_sendmsg+0xb0/0xd8) [<c0a7e714>] (sock_sendmsg+0xb0/0xd8) from [<c0a7f6f4>] (__sys_sendmsg.part.6+0x1b4/0x260) [<c0a7f6f4>] (__sys_sendmsg.part.6+0x1b4/0x260) from [<c0a80b04>] (sys_sendmsg+0x5c/0x94) [<c0a80b04>] (sys_sendmsg+0x5c/0x94) from [<c0108924>] (__sys_trace_return+0x0/0x1c)

This appears to be a bug in the LineageOS kernel. AFAIK, the only users experiencing this issue are running LineageOS, so this makes sense. I found a post from a user experiencing the same panic on a HTC M8 with a different product here[1], and there is apparently a patch in LineageOS which may fix it[2].

Now as for why Fenix only recently started crashing, the change found during bisect includes Bug 1561944[3], which enables the netlink Gecko service for Android. I would encourage LineageOS users to pursue a fix for their individual ROMs, but we may also be able to have a workaround in Gecko. I'll file a followup for that shortly.

[1] https://forum.syncthing.net/t/phones-reboot-with-syncthing-1-30-under-lineageos-16-on-htc-m8/13849/26
[2] https://github.com/LineageOS/android_kernel_cyanogen_msm8974/commit/f44b4b19da1541fce8440ffa1e59cdb722cf10b5
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1561944

[snorp]

I filed a bug for the Gecko workaround here: https://bugzilla.mozilla.org/show_bug.cgi?id=1604056

[loveless90]

If the bug realy would be solved by the commit you've linked, than i won't be affected. The developer of the build, that i'm using, wrote at xda, that the patch is already included.

https://forum.xda-developers.com/showpost.php?p=81211447&postcount=1231

I'll try to post the files that i have on my phone later.

[kolAflash]

I took the time to finally update my "Fairphone 2" (FP2) to LineageOS-16.0-20191212-nightly. And indeed, the bug is gone and Firefox-Preview-3.0.0 runs fine! So that 2019-03-24 LineageOS patch seems fine.
https://github.com/LineageOS/android_kernel_cyanogen_msm8974/commit/f44b4b19da1541fce8440ffa1e59cdb722cf10b5

Nevertheless, there are probably hundreds of phones with LineageOS <= 2019-03-24 out there.
If there's a fix for Firefox-Preview, I'll happily test it on one of my phones running older LineageOS versions.
(Nexus 5X / bullhead with LineageOS-15.1-2019-02-04)
(Galaxy S5 LTE / G900F LineageOS-16.0-2019-03-11)

[loveless90]

Now Firefox-Preview-Nightly and Firefox-Preview works fine for me, too. And like kolAflash i updated my phone to the newest build (20191213). I don't think that's the bug you mentioned, because the patch is included for months and the last build i had installed was of 2019-11-13.

I would say it's a combination of both, and maybe i should downgrade again, and post the documents i wanted to post today.

A link to the builds:
https://www.mediafire.com/folder/o4cxf3pl2ibax/lineage-16.0

[elsandosgrande]

The same on an Galaxy S4 mini (I9195/serranoltexx) with LineageOS 16.0 and without root (removed with "adddonsu-remove").

System crashes when background-process or fenix itself get started by the system (after entering pin and password) or the user . Haven't tested Airplane-mode because i removed fenix after i noticed the bug (system got unusable).

Same issue here, Samsung Galaxy S4 GT-I9505 (jfltexx) running rooted Resurrection Remix OS. I have not tried airplane mode either.

I took the time to finally update my "Fairphone 2" (FP2) to LineageOS-16.0-20191212-nightly. And indeed, the bug is gone and Firefox-Preview-3.0.0 runs fine! So that 2019-03-24 LineageOS patch seems fine.
LineageOS/android_kernel_cyanogen_msm8974@f44b4b1

Nevertheless, there are probably hundreds of phones with LineageOS <= 2019-03-24 out there.
If there's a fix for Firefox-Preview, I'll happily test it on one of my phones running older LineageOS versions.
(Nexus 5X / bullhead with LineageOS-15.1-2019-02-04)
(Galaxy S5 LTE / G900F LineageOS-16.0-2019-03-11)

I used to run LineageOS, but between the fact that one update corrupted the kernel image and that it was, at least at the time last year, stuck on Android 7, and the fact that Resurrection Remix OS has fancy user interface customization options, I eventually switched over to RR.

I just noticed that LineageOS now has 16.0 support my phone. If anybody needs any testing to be done on an unpatched kernel, do mention it before the end of the year.

[cagapisto]

Confirm this bug also for Android 10 (Q) based LineageOS ROM

Phone: Xiaomi Mi4 (cancro)
ROM: Mokee (MK100.0-cancro-202002100449-NIGHTLY).
Kernel: 3.4.113-perf-gadc4d9a3160
Root: Yes. Magisk 20.3.06
Xposed: No

[elsandosgrande]

Has a patch for this been developed yet? It seems that even newer ROMs on some devices are experiencing the issue. What's funny is that my mom's phone, which is a low-end Prestigio on Android 7, does not have this issue and my mom can use Firefox Preview freely, other than the slowness of the phone itself that is.

[MightyPork]

I'm running LineageOS 15.1 on OnePlus One (Bacon). The latest update of the Playstore version has this bug - the app unceremoniously reboots the phone immediately after start, the whole UI freezes (back/home buttons) first, then it goes to the boot logo. I tried logcat and also dmesg under root, but there was nothing, it just stopped and rebooted.

What makes this worse: If firefox is the default browser, then GMail crashes after a few seconds, presumably using FF for some kind of previews.

I solved it by replacing FF with Fennec from F-Droid, that seems to not have this bug (yet).

[kbrosnan]

The solution is to upgrade your OS. Linage has had a fix in the tree since 2019-03-24. https://github.com/LineageOS/android_kernel_cyanogen_msm8974/commit/f44b4b19da1541fce8440ffa1e59cdb722cf10b5

Closing this due to it being an OS bug that has been resolved by the distributor.

[Naklarsuper]

Same issue here. There are devices out there with Lineage OS 15.1 being last updated last year. And devices / OS are not that old.
I don't quite buy that the "problem and solution lies within the OS" - all other browsers are working fine, and indeed I feel forced to use another browser until this issue is resolved. After all, Firefox was working just fine yesterday, before updating to 80.