Realm-java: Native abort in nativeGetSchemaVersion

[DanylNysom]

Goal

A stable app

Actual Results

We're seeing loads of native crashes (Signal 11's, etc.) on the Play Store. A lot of them involve a call to OsResults.nativeSize, and in particular I've found a few that start with nativeGetSchemaVersion calling nativeSize. They seem like potentially the most useful crash logs, but I can provide many more.

Here's the most complete trace:

********** Crash dump: **********
pid: 0, tid: 0 >>> [our-app] <<<
Stack frame #00  pc 0000000000022170  /system/lib64/libc.so (abort+116)
Stack frame #01  pc 00000000003496e0  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so (__gnu_cxx::__verbose_terminate_handler()+236): Routine __gnu_cxx::__verbose_terminate_handler() at /s/ndk-toolchain/src/gcc/gcc-4.9/libstdc++-v3/libsupc++/vterminate.cc:95
Stack frame #02  pc 000000000031916c  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so (__cxxabiv1::__terminate(void (*)())+8): Routine __cxxabiv1::__terminate(void (*)()) at /s/ndk-toolchain/src/gcc/gcc-4.9/libstdc++-v3/libsupc++/eh_terminate.cc:47
Stack frame #03  pc 00000000003191d8  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so (std::terminate()+12): Routine std::terminate() at /s/ndk-toolchain/src/gcc/gcc-4.9/libstdc++-v3/libsupc++/eh_terminate.cc:57 (discriminator 1)
Stack frame #04  pc 00000000003198b0  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so (__cxa_pure_virtual+28): Routine __cxa_pure_virtual at /s/ndk-toolchain/src/gcc/gcc-4.9/libstdc++-v3/libsupc++/pure.cc:50
Stack frame #05  pc 000000000024abbc  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so: Routine realm::ParentNode::aggregate_local(realm::QueryStateBase*, unsigned long, unsigned long, unsigned long, realm::SequentialGetterBase*) at :?
Stack frame #06  pc 000000000021a404  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so: Routine realm::Query::aggregate_internal(realm::Action, realm::DataType, bool, realm::ParentNode*, realm::QueryStateBase*, unsigned long, unsigned long, realm::SequentialGetterBase*) const at :?
Stack frame #07  pc 0000000000224970  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so: Routine realm::Query::do_count(unsigned long, unsigned long, unsigned long) const at :?
Stack frame #08  pc 0000000000225ed4  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so: Routine realm::Query::count(realm::DescriptorOrdering const&) at :?
Stack frame #09  pc 00000000000bc264  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/lib/arm64/librealm-jni.so (Java_io_realm_internal_OsResults_nativeSize+48): Routine Java_io_realm_internal_OsResults_nativeSize at ??:?
Stack frame #10  pc 000000000015ddf8  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/oat/arm64/base.odex (io.realm.internal.OsObjectStore.nativeGetSchemaVersion [DEDUPED]+152)
Stack frame #11  pc 00000000001c017c  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/oat/arm64/base.odex (com.myapp.util.DatabaseUtil.getNonDuplicateCertWithStartTime+3052)
Stack frame #12  pc 00000000001c1ea4  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/oat/arm64/base.odex (com.myapp.util.DatabaseUtil.getNonDuplicateCerts+3508)
Stack frame #13  pc 00000000001b3720  /data/app/com.myapp-jGwFWZtKMRs0OowF2z0v4g==/oat/arm64/base.odex (com.myapp.rtl.RTLRuleService$RTLRunnable.run+976)
Stack frame #14  pc 00000000002cb5e4  /system/framework/arm64/boot.oat (java.util.concurrent.Executors$RunnableAdapter.call+68)
Stack frame #15  pc 0000000000371f40  /system/framework/arm64/boot.oat (java.util.concurrent.FutureTask.runAndReset+208)
Stack frame #16  pc 00000000004267b4  /system/framework/arm64/boot.oat (java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run+164)
Stack frame #17  pc 00000000003daa14  /system/framework/arm64/boot.oat (java.util.concurrent.ThreadPoolExecutor.runWorker+996)
Stack frame #18  pc 00000000003d7790  /system/framework/arm64/boot.oat (java.util.concurrent.ThreadPoolExecutor$Worker.run+64)
Stack frame #19  pc 000000000025e1c8  /system/framework/arm64/boot.oat (java.lang.Thread.run+72)
Stack frame #20  pc 0000000000557188  /system/lib64/libart.so (art_quick_invoke_stub+584)
Stack frame #21  pc 00000000000cfac8  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
Stack frame #22  pc 000000000045def4  /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
Stack frame #23  pc 000000000045efb0  /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
Stack frame #24  pc 000000000048a360  /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
Stack frame #25  pc 0000000000083840  /system/lib64/libc.so (__pthread_start(void*)+36)
Stack frame #26  pc 0000000000023d80  /system/lib64/libc.so (__start_thread+68)

Steps & Code to Reproduce

I'm not sure how to use the trace to find the line number in our java code (Proguard or whatever means it doesn't line up), but there are only two Realm usages in our getNonDuplicateCertWithStartTime function:

• Realm.where()...findAll()
• Realm.where()...count()

If requested I should be able to email some more of the code for context. I'll need to get approval from a higher authority on my end, as well.

Version of Realm and tooling

Realm version(s): 5.12.0, but we've been seeing these native crashes since probably about 5.9.0 (rough guess)

Realm Sync feature enabled: No

Android Studio version: 3.4.2

Android Build Tools version: 28.0.3

Gradle version: 5.5

Which Android version and device(s): Again, we've been seeing this a lot. The specific log I posted is from a Samsung Galaxy S9, Android 9

[cmelchior]

It would be nice to see specifically what kind of query you are running. You can email the code privately to [email protected].

[DanylNysom]

Email sent.

Something that might be useful to anyone else who stumbles on this question is that the issue seems like it might be due to running out of memory. Almost all of the reports I'm seeing have <= 7% free RAM.

[iop280]

Seems like it would affect other issues as well.

[DanylNysom]

Do you have or need any more information @cmelchior?
I'm wondering if there actually is a memory leak, or if the reporting is somehow confused because of the bridge between Java and C++. You'd probably have a better idea of that than I would.

If this actually is a leak, then might it be in realm-core, and should I report this issue there?

Nothing has changed on our end.

[DanylNysom]

@cmelchior
Okay, we've fixed the memory leaks and that helped somewhat, but we're still seeing a lot of these crashes with no correlation to low RAM.

The crash shows up a few different ways (e.g. "tgkill", "SIGSEGV in libream-jni.so", "SIGABRT in libc.so") but from what I can tell they all happen at a call to RealmResults.size().

Our latest release uses realm-java 5.14.0

[DanylNysom]

@cmelchior any news? I've been trying (and failing) to repro in office, so any tips for things that might repro it at least would be helpful

[DanylNysom]

I've gotten the .realm file off of a customer's device that has been crashing very frequently, and attached it to the support ticket (ticket 4404). Unfortunately I've still been unable to repro the crash.

[cmelchior]

Hey @DanylNysom Sorry for the late response, we had an engineering offsite last week.
I have the query you sent to [email protected] and I'm planning to try to reproduce it this week. I'll keep you updated.

[cmelchior]

@DanylNysom I responded to your support tickets. I didn't find any concrete crashes but a few optimizations I think might help.