Fenix: FNX2-18885 ⁃ [Bug] Missing Socorro links for native crashes

[hwinnemoe]

Steps to reproduce

1. Have a native crash
2. Open _Settings_, _About Firefox Nightly_, _Crashes_

Expected behavior

According to Logging-Crash-Information, the native crashes should have Socorro links associated with them.

Actual behavior

I do not see any Socorro links for my native crashes.

Device information

• Android device: Sony Xperia Z2, Android 6.0.1
• Fenix version: Nightly 200804 09:33 (Build #22170942)
  AC: 53.0.20200803131508, 54c08c6a4
  GV: 81.0a1-20200802093514
  AS: 61.0.10

[ekager]

So AFAIK the Socorro link only shows up if you press the crash notification and then "Send to Mozilla". Were you doing that as well?

[hwinnemoe]

Yes, every time.

[kbrosnan]

@rocketsroger or @pocmo Any thoughts here? Might be something for a-c/lib/crash?

[hwinnemoe]

Here is a short video using about:crashparent

[rocketsroger]

Here is a short video using about:crashparent

May I please have some of your device info? Which device is this and what Android OS version is it on? Thanks,

[hwinnemoe]

Sony Xperia Z2, Android 6.0.1

[rocketsroger]

Sony Xperia Z2, Android 6.0.1

Thanks, I'll look into this. I suspect its the crash dialog is not working as expected. On Android 10 I could not reproduce the same issue.

[rocketsroger]

Couldn't reproduce with Android 7.0 emulator. The crash dialog is working as expected. I'll look into if this is Android 6.0.1 related.

[pocmo]

A missing Socorro link means that we do not get a crash ID from Socorro when trying to upload the crash (We do not retry currently, https://github.com/mozilla-mobile/android-components/issues/7006).

@hwinnemoe Can you provide a logcat dump? I wonder if it mentions that it couldn't upload the crash.
@rocketsroger Can you check what info we log in the case of an upload error/problem and whether we may need to log more (e.g. http status code) to understand problems?

[rocketsroger]

@rocketsroger Can you check what info we log in the case of an upload error/problem and whether we may need to log more (e.g. http status code) to understand problems?

Currently we log success with the line "Crash reported to Socorro: $id". Failures are logged with two outcomes:

1. "Server rejected crash report"
2. "failed to send report to Socorro" followed by the exception.

[hwinnemoe]

@hwinnemoe Can you provide a logcat dump? I wonder if it mentions that it couldn't upload the crash.

logcat_GH-13278_2020-08-05.zip

Steps performed while logging

1. Open about:crashparent
2. Make sure to check the mark for _Send crash report to Mozilla_
3. Tap _Restart Firefox Nightly_
4. Open _Settings_, _About Firefox Nightly_, _Crashes_
5. Wait for 20 seconds for any timeouts to be logged

Note: Logging-Crash-Information should also include a step to first clear the logcat via adb logcat -c to make the output minimal.

[pocmo]

08-05 17:59:43.410 I/mozac/CrashReporter(16105): Submitted crash to Sentry (id=null, url=null)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105): failed to send report to Socorro
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105): java.io.FileNotFoundException: https://crash-reports.mozilla.com/submit?id={eeb82917-e434-4870-8148-5c03d4caa81b}&version=Nightly 200805 06:08&android_component_version=53.0.20200804130112
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:238)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at mozilla.components.lib.crash.service.MozillaSocorroService.sendReport$lib_crash_release(MozillaSocorroService.kt:16)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at mozilla.components.lib.crash.service.MozillaSocorroService.report(MozillaSocorroService.kt:8)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at mozilla.components.lib.crash.CrashReporter$submitReport$2.invokeSuspend(CrashReporter.kt:8)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:3)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:21)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:1)
08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105):    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:11)
08-05 17:59:44.390 I/mozac/CrashReporter(16105): Submitted crash to Socorro (id=null, url=null)

[pocmo]

08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105): java.io.FileNotFoundException: https://crash-reports.mozilla.com/submit?id={eeb82917-e434-4870-8148-5c03d4caa81b}&version=Nightly 200805 06:08&android_component_version=53.0.20200804130112

Looking at the version it seems like we do not urlencode at least the version parameter. Not sure if that matters only on that version of Android?

Anyhow, looking at Android's code it seems like a FileNotFoundException is thrown for any response code >= HTTP_BAD_REQUEST ... so it would definitely be interesting to log the response code too. :)

[hwinnemoe]

08-05 17:59:41.911 I/ActivityManager( 1269): Process org.mozilla.fenix:mozilla.components.lib.crash.CrashHandler (pid 16824) has died

Is this supposed to happen?

[rocketsroger]

08-05 17:59:44.390 E/mozac/MozillaSocorroCra(16105): java.io.FileNotFoundException: https://crash-reports.mozilla.com/submit?id={eeb82917-e434-4870-8148-5c03d4caa81b}&version=Nightly 200805 06:08&android_component_version=53.0.20200804130112

Looking at the version it seems like we do not urlencode that parameter. Not sure if that matters only on that version of Android?
Anyhow, looking at Android's code it seems like aFileNotFoundExceptionis thrown for any response code>= HTTP_BAD_REQUEST` ... so it would definitely be interesting to log the response code too. :)

Sure, I'll add logging for the response code.

[hwinnemoe]

Should I get another logcat dump once https://github.com/mozilla-mobile/android-components/pull/8021 lands here? I think it closely missed the last AC update to this repository here https://github.com/mozilla-mobile/fenix/commit/dbc1001d0346b3fb5515d0b2ca6cecfa367b6114?

[pocmo]

Yeah, a new logcat dump would be perfect. Hopefully the response code gives us some hints.

[hwinnemoe]

logcat_GH-13278_2020-08-08_error-code.zip

Steps performed while logging

1. Open about:crashparent
2. Make sure to check the mark for _Send crash report to Mozilla_
3. Tap _Restart Firefox Nightly_
4. Wait for 20 seconds for any timeouts to be logged

Highlights

08-08 11:19:37.420 I/ActivityManager( 1180): Process org.mozilla.fenix:mozilla.components.lib.crash.CrashHandler (pid 3091) has died
08-08 11:19:37.420 W/ActivityManager( 1180): Scheduling restart of crashed service org.mozilla.fenix/mozilla.components.lib.crash.handler.CrashHandlerService in 1000ms
[...]
08-08 11:19:41.012 E/mozac/MozillaSocorroCra( 3170): failed to send report to Socorro with 400
08-08 11:19:41.012 E/mozac/MozillaSocorroCra( 3170): java.io.FileNotFoundException: https://crash-reports.mozilla.com/submit?id={eeb82917-e434-4870-8148-5c03d4caa81b}&version=Nightly 200808 06:09&android_component_version=54.0.20200807130552

Nightly version
Nightly 200808 06:09 (Build #2015756721)
AC: 54.0.20200807130552, 1d5e94f0d
GV: 81.0a1-20200806033456
AS: 61.0.10

[rocketsroger]

08-08 11:19:41.012 E/mozac/MozillaSocorroCra( 3170): failed to send report to Socorro with 400
08-08 11:19:41.012 E/mozac/MozillaSocorroCra( 3170): java.io.FileNotFoundException: https://crash-reports.mozilla.com/submit?id={eeb82917-e434-4870-8148-5c03d4caa81b}&version=Nightly 200808 06:09&android_component_version=54.0.20200807130552

I'll investigate. Thanks for the update.

[rocketsroger]

@hwinnemoe one question, is this only reproducible on this device with Nightly builds? Can you please help try the release build? Thanks,

[rocketsroger]

Looking at the version it seems like we do not urlencode at least the version parameter. Not sure if that matters only on that version of Android?

I suspect this is the case. I can't reproduce it on my emulator. But it probably make sense to URL encode the parameters.

[hwinnemoe]

@hwinnemoe one question, is this only reproducible on this device with Nightly builds? Can you please help try the release build? Thanks,

I did try it on Beta 80.0.0-beta.6 which does not reproduce the issue, see https://crash-stats.mozilla.org/report/index/89161726-49fa-4259-a2c6-cdd1a0200818. I can try it on Release but I expect it to work as well since there are no spaces in the version.

Is there any way to determine which devices/android version fail to send to Socorro? E.g. using https://crash-stats.mozilla.org/signature/?product=Fenix&signature=java.lang.IllegalStateException%3A%20at%20androidx.fragment.app.FragmentManager.ensureExecReady%28FragmentManager.java%29&date=%3E%3D2020-08-11T13%3A06%3A00.000Z&date=%3C2020-08-18T13%3A06%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_columns=android_version&_sort=version&_sort=-android_version&page=2 and sorting by Android version and Fenix version reveals that crash reports for a common crash were never send by devices lower than API level 24 for the Nightly version.

[rocketsroger]

I did try it on Beta 80.0.0-beta.6 which does not reproduce the issue, see https://crash-stats.mozilla.org/report/index/89161726-49fa-4259-a2c6-cdd1a0200818. I can try it on Release but I expect it to work as well since there are no spaces in the version.

Great, then my recent fix might work. I'll update once it is in Nightly.

sorting by Android version and Fenix version reveals that crash reports for a common crash were never send by devices lower than API level 24 for the Nightly version.

That's very interesting. I could reproduce in emulator with API 23 and Nightly. However, I couldn't reproduce with Nightly debug builds. It's good now that we know this issue only occurs on Nightly and probably only on devices below API 24. This narrows down the possibility of what might have caused it.

[rocketsroger]

Found a way to reproduce properly with debug builds. Confirmed fix is working correctly. Once the fix is in Nightly, I'll update. @hwinnemoe thanks for your help narrowing down the issue.

[hwinnemoe]

And it works: https://crash-stats.mozilla.org/report/index/b9cf30a0-34d5-4b5a-a698-0bb100200820 🎉

[rocketsroger]

And it works: https://crash-stats.mozilla.org/report/index/b9cf30a0-34d5-4b5a-a698-0bb100200820 🎉

Thanks for confirming. Closing as fixed.