Fenix: FNX2-16462 ⁃ [Bug] Firefox Nightly crashes in background every 15 minutes

[equeim]

• [x] Glean Bugzilla bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1644290

Steps to reproduce

???
Randomly started a couple of days ago.
Logcat output:
loglog.log
I would guess that it has something to do with telemetry.

Expected behavior

It doesn't crash.

Actual behavior

Every 15m I get notifications about Firefox crashes.

Device information

• Android device: OnePlus 5T, Android 10
• Fenix version: Nightly 200608

[Augures01]

Same
https://crash-stats.mozilla.org/report/index/5c5c669a-99e2-46de-aa33-3e9c50200608

[vishalnandagopal]

Same. I had to force stop it to prevent sending crash reports or notifications every fifteen minutes.

Android 10 Build 21580628

[ekager]

06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: IOException while uploading ping
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: java.io.IOException: org.mozilla.geckoview.WebRequestError: Request failed, error=0x43, category=0x3
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at mozilla.components.browser.engine.gecko.fetch.GeckoViewFetchClient.fetch(GeckoViewFetchClient.kt:49)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at mozilla.components.service.glean.net.ConceptFetchHttpUploader.performUpload$service_glean_release(ConceptFetchHttpUploader.kt:2)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at mozilla.components.service.glean.net.ConceptFetchHttpUploader.upload(ConceptFetchHttpUploader.kt:12)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at mozilla.telemetry.glean.net.BaseUploader.upload(Unknown Source:17)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at mozilla.telemetry.glean.net.BaseUploader.doUpload$glean_release(BaseUploader.kt:2)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at mozilla.telemetry.glean.scheduler.PingUploadWorker.doWork(PingUploadWorker.kt:11)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at androidx.work.Worker$1.run(Worker.java:1)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at java.lang.Thread.run(Thread.java:919)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: Caused by: org.mozilla.geckoview.WebRequestError: Request failed, error=0x43, category=0x3
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
06-08 19:56:16.132 27292 31072 W glean/ConceptFetchHttpUploader: at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:88)
06-08 19:56:16.132 27292 31072 E libglean_ffi: glean_core::upload: Recoverable upload failure while attempting to send ping c80cb5df-e268-4d68-a6ac-803a7af62919, will retry. Error was RecoverableFailure
06-08 19:56:16.132 27292 31072 I libglean_ffi: glean_core::upload::directory: Processing ping at: /data/user/0/org.mozilla.fenix.nightly/glean_data/deletion_request/c80cb5df-e268-4d68-a6ac-803a7af62919
06-08 19:56:16.133 27292 27343 W glean/ConceptFetchHttpUploader: IOException while uploading ping
06-08 19:56:16.133 27292 27343 W glean/ConceptFetchHttpUploader: java.io.IOException: org.mozilla.geckoview.WebRequestError: Request failed, error=0x43, category=0x3
06-08 19:56:16.133 27292 27343 W glean/ConceptFetchHttpUploader: at mozilla.components.browser.engine.gecko.fetch.GeckoViewFetchClient.fetch(GeckoViewFetchClient.kt:49)
06-08 19:56:16.133 27292 27343 W glean/ConceptFetchHttpUploader:

[liuche]

Filed a Glean bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1644290

[liuche]

This could also be a GV bug, based on the webrequest errors - should the retry request retry so many times that it overflows and crashes?

cc @boek for GV triage - lmk if putting it in the triage column is correct, or if it should be somewhere else.

Caused by: org.mozilla.geckoview.WebRequestError: Request failed, error=0x43, category=0x3
06-08 19:56:16.140 27292 31072 W glean/ConceptFetchHttpUploader:    at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
06-08 19:56:16.140 27292 31072 W glean/ConceptFetchHttpUploader:    at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:88)

[fluffyemily]

In progress with Glean team.

[pwd-github]

Still seeing this

[ekager]

A fix landed in Glean, but it will take some time for Fenix to get the update.

[liuche]

@mdboom how do we get this into the next AC version? Does Glean need to make a release and open a PR against AC to bump the Glean version?

[badboy]

Release is out and about to land in AC: https://github.com/mozilla-mobile/android-components/pull/7336

I will have that reviewed in no time, so it can land in the next AC nightly.

[badboy]

The Glean update finally landed, so I expect this to be in the next nightly

[sv-ohorvath]

We didn't get the background crash notifications anymore on Fenix 5.2-beta2 or latest Nightly while testing, so we can consider this fixed.