Sentry-dart: [Crash] tgkill crash reported on Play Console

[ayush221b]

_Platform:_

• [ ] Dart
• [ x] Flutter Android or iOS
• [ ] Flutter Web

_split-debug-info and obfuscate (Flutter Android or iOS) or CanvasKit (Flutter Web):_

• [ x] Enabled
• [ ] Disabled

_Platform installed with:_

• [x ] pub.dev
• [ ] GitHub

Output of the command flutter doctor -v below:

❯ flutter doctor -v
[✓] Flutter (Channel stable, 1.22.0, on Mac OS X 10.15.7 19H114, locale en-IN)
    • Flutter version 1.22.0 at /Users/ayushshekhar/development/flutter
    • Framework revision d408d302e2 (3 months ago), 2020-09-29 11:49:17 -0700
    • Engine revision 5babba6c4d
    • Dart version 2.10.0


[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.1)
    • Android SDK at /Users/ayushshekhar/Library/Android/sdk
    • Platform android-30, build-tools 30.0.1
    • ANDROID_HOME = /Users/ayushshekhar/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.0

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] IntelliJ IDEA Community Edition (version 2020.2.3)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin installed
    • Dart plugin version 202.7319.5

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (2 available)
    • HD1901 (mobile)                    • 8141b52c      • android-arm64 • Android 10 (API 29)
    • Android SDK built for x86 (mobile) • emulator-5554 • android-x86   • Android 7.0 (API 24) (emulator)

! Doctor found issues in 1 category.

The version of the SDK (See pubspec.lock):
4.0.0

---

I have the following issue:
We have started to see these tgkill crash reports in Play Console, since integrating sentry_flutter.

Details:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> io.connectcourses.app <<<

backtrace:
  #00  pc 000000000006b518  /system/lib64/libc.so (tgkill+8)
  #00  pc 000000000006899c  /system/lib64/libc.so (pthread_kill+64)
  #00  pc 0000000000023ee8  /system/lib64/libc.so (raise+24)
  #00  pc 000000000001c96c  /system/lib64/libc.so (abort+52)
  #00  pc 0000000000430040  /system/lib64/libart.so (art::Runtime::Abort()+352)
  #00  pc 00000000000e5e98  /system/lib64/libart.so (art::LogMessage::~LogMessage()+1204)
  #00  pc 000000000024bf38  /system/lib64/libart.so (art::IndirectReferenceTable::Add(unsigned int, art::mirror::Object*)+308)
  #00  pc 000000000034a2d0  /system/lib64/libart.so (art::JNI::NewStringUTF(_JNIEnv*, char const*)+620)
  #00  pc 000000000000281c  /data/app/io.connectcourses.app-1/split_config.arm64_v8a.apk!lib/arm64-v8a/libsentry-android.so (offset 0x2775000)
  #00  pc 0000000000cf768c  /data/app/io.connectcourses.app-1/oat/arm64/base.odex (offset 0xcce000)

Note: This is primarily happening on Android 6 and Android 7.

[marandaneto]

hey @ayush221b thanks for reporting this.

I can't really reproduce the issue on Android 6/7 emulators, do you have any more info? could you reproduce yourself?

libc tgkill is a pretty common Android error (basically the shutdown stack trace) for Android 6/7, See https://stackoverflow.com/questions/44080809/android-7-native-crash-libc-so-tgkill

Not really sure that this is caused by Sentry, will investigate.

@Swatinem ideas?

[Swatinem]

Hm, the issue you linked mentions some assertions being hit.
We do have art::LogMessage::~LogMessage() in the stack as well, coming from art::JNI::NewStringUTF originally.

Would be nice to get that log message, without it its just speculation. But I would speculate that this originates from our JNI bridge (libsentry-android.so is the bridge code, not sentry-native itself), maybe we are missing a null check when constructing a jni string? Or we have some outright corrupted data?

[marandaneto]

@Swatinem that's what I thought as well, but every NewStringUTF is guarded by !sentry_value_is_null(var) and only then we call sentry_value_as_string(var) and consequently NewStringUTF(var).

about outright corrupted data, it could be, but this data come from sentry-native using sentry_value_get_by_key and not the JNI layer, is there a way to detect corrupted data at runtime?

[Swatinem]

every NewStringUTF is guarded by !sentry_value_is_null(var) and only then we call sentry_value_as_string(var) and consequently NewStringUTF(var)

I think you can unconditionally call sentry_value_as_string. That will give you NULL whenever its not a string, which can happen just as well if its not a null value.

[marandaneto]

@Swatinem not really, I remember getting segfault if calling sentry_value_as_string directly, then you've pointed out the sentry_value_is_null function.

the thing is, I'm assuming that NewStringUTF is called by us when loading the module list, so we can symbolicate Flutter stack traces, but the stack trace shared above does not show the source of calling such method, I can't even tell for sure it's Sentry to blame.

[ayush221b]

hey @ayush221b thanks for reporting this.

I can't really reproduce the issue on Android 6/7 emulators, do you have any more info? could you reproduce yourself?

libc tgkill is a pretty common Android error (basically the shutdown stack trace) for Android 6/7, See https://stackoverflow.com/questions/44080809/android-7-native-crash-libc-so-tgkill

Not really sure that this is caused by Sentry, will investigate.

@Swatinem ideas?

@marandaneto Thank you for your reply. This might be a common error in Android 6/7 but we started seeing this on our play console only after we went live with our Sentry integration.

Here's another one, it occured a few hours ago:

Stacktrace:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> io.connectcourses.app <<<

backtrace:
  #00  pc 000000000004ab94  /system/lib/libc.so (tgkill+12)
  #00  pc 0000000000048333  /system/lib/libc.so (pthread_kill+34)
  #00  pc 000000000001de0d  /system/lib/libc.so (raise+10)
  #00  pc 0000000000019511  /system/lib/libc.so (__libc_android_abort+34)
  #00  pc 000000000001718c  /system/lib/libc.so (abort+4)
  #00  pc 000000000031c1dd  /system/lib/libart.so (art::Runtime::Abort(char const*)+328)
  #00  pc 00000000000b5569  /system/lib/libart.so (art::LogMessage::~LogMessage()+1132)
  #00  pc 00000000001bd4e5  /system/lib/libart.so (art::IndirectReferenceTable::Add(unsigned int, art::mirror::Object*)+308)
  #00  pc 000000000027d2c1  /system/lib/libart.so (art::JNI::NewStringUTF(_JNIEnv*, char const*)+424)
  #00  pc 0000000000002019  /data/app/io.connectcourses.app-2/split_config.armeabi_v7a.apk!lib/armeabi-v7a/libsentry-android.so (offset 0x3286000)
  #00  pc 0000000000ced0dd  /data/app/io.connectcourses.app-2/oat/arm/base.odex (offset 0xcce000)

Device: Samsung Galaxy On5 Pro (Android 7.1 SDK 25)

[marandaneto]

@ayush221b the stack trace seems legitimate, but I can't really reproduce the issue, do you have any more info? or only this peace of stack trace.

what the user was doing? was during SDK init? was while trying to capture an event? etc... thanks

[ayush221b]

Hey @marandaneto I only have these stacktraces.

Here's the latest one and it is a bit different from the ones that I have already shared.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> io.connectcourses.app <<<

backtrace:
  #00  pc 000000000006b558  /system/lib64/libc.so (tgkill+8)
  #00  pc 00000000000689dc  /system/lib64/libc.so (pthread_kill+64)
  #00  pc 0000000000023f28  /system/lib64/libc.so (raise+24)
  #00  pc 000000000001c9ac  /system/lib64/libc.so (abort+52)
  #00  pc 0000000000431de4  /system/lib64/libart.so (art::Runtime::Abort(char const*)+456)
  #00  pc 00000000000e5e8c  /system/lib64/libart.so (art::LogMessage::~LogMessage()+1576)
  #00  pc 000000000024cfbc  /system/lib64/libart.so (art::IndirectReferenceTable::Add(unsigned int, art::mirror::Object*)+308)
  #00  pc 000000000034b548  /system/lib64/libart.so (art::JNI::NewStringUTF(_JNIEnv*, char const*)+620)
  #00  pc 0000000000002788  /data/app/io.connectcourses.app-2/lib/arm64/libsentry-android.so (Java_io_sentry_android_ndk_NativeModuleListLoader_nativeLoadModuleList+588)
  #00  pc 0000000000cf768c  /data/app/io.connectcourses.app-2/oat/arm64/base.odex (offset 0xcce000)

[marandaneto]

@ayush221b that indeed helps, thanks.

@Swatinem it leads us to https://github.com/getsentry/sentry-java/blob/3.2.0/sentry-android-ndk/src/main/jni/sentry.c#L307-L362 as suspected.
but I can't really see how it could cause a segfault, I ran on emulators and could not find anything.

[bruno-garcia]

@marandaneto do we send debug_info in all events or only when we detect split-debug-info was used and it's needed by the backend?

[marandaneto]

@marandaneto only when it's needed, either if split-debug-info is enabled and/or added the Sentry Gradle Plugin + Proguard/R8

[ayush221b]

Hi team,

We have now started seeing this issue surface on our Sentry dashboard as well.
There have been 900+ events for 120+ users as of now.

OS Version: Android 7.0 (NRD90M release-keys)
Report Version: 104

Exception Type: Unknown (SIGABRT)

Application Specific Information:
Abort

Thread 0 Crashed:
0   libc.so                         0x72148c9958        tgkill
1   libc.so                         0x72148c6d7c        pthread_kill
2   libc.so                         0x721488009c        raise
3   libc.so                         0x7214878940        abort
4   libart.so                       0x721200b860        art::Runtime::PreZygoteFork
5   libart.so                       0x7211cc02fc        art::LogMessage::LogLine
6   libart.so                       0x7211e2703c        art::IndirectReferenceTable::Add
7   libart.so                       0x7211f25050        <unknown> + 489927364688
8   libsentry-android.so            0x71ed8cd920        Java_io_sentry_android_ndk_NativeModuleListLoader_nativeLoadModuleList
9   base.odex                       0x71f5e0d690        oatexec



EOF

[marandaneto]

@ayush221b could you link us to a Sentry event? feel free to send it over to [email protected] thanks :)

[marandaneto]

@ayush221b thanks, apparently the issue happens only on Android 7.0 or 7.1.2, always arm64-v8a, let's see if I can reproduce when focusing on this arch.

[marandaneto]

ok I could reproduce it and confirm the bug: local reference table overflow (max=512)

apparently, we're not releasing the allocated jstring on the JNI bridge, which we thought was done automatically by the JVM.
https://stackoverflow.com/questions/6238785/newstringutf-and-freeing-memory this thread resumes my findings.

and why only on Android < 8?

In Android versions prior to Android 8.0, the number of local references is capped at a version-specific limit. Beginning in Android 8.0, Android supports unlimited local references.

https://developer.android.com/training/articles/perf-jni.html#local_and_global_references

[marandaneto]

We'll do a patch on the Android SDK and Bump the Flutter SDK

[ayush221b]

Thank you so much, looking forward to it.

[marandaneto]

https://github.com/getsentry/sentry-dart/releases/tag/4.0.2

thank you all for helping us out debugging this issue.