Realm-java: Realm.getGlobalInstanceCount(...) grows really fast AND throws OOM/too many open files exception

[ArilesGit]

Hello, and thank you in advance for the help ! :-)

Goal

Prevent crashes with better backgroung Realm instances management/multithreading.

Actual Results

5114-5337 E: ashmem_create_region failed for 'indirect ref table': Too many open files
5114-5337 W: Throwing OutOfMemoryError "Could not allocate JNI Env"
5114-6313 E: jni: ThrowingException 5, Too many open files in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101, .
5114-6313 E: Exception has been thrown: Unrecoverable error. Too many open files in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101
...

Steps & Code to Reproduce

I had this crash happen several times so tried to reproduce it this way:

I created "for loop" that gets a Realm instance and executes a transaction in a new thread

new Thread(() -> {
            try (Realm realm = Realm.getDefaultInstance()) {
                realm.executeTransaction(transaction);
            }).start();

I have a realm instance mRealm in an activity while I do this test. I only use a single RealmConfiguration.

Logs show that Realm.getGlobalInstanceCount(mRealm) grows really fast.
I also show logs about memory usage.

5114-5337 D: Memory > usedMB: 9 | maxHeapSize:  128 | availHeapSize:  119
5114-5337 D: Realm instance count > initial count: 1

5114-5338 D: Realm instance count > i: 1 | count: 4
5114-5338 D: Memory > usedMB: 9 | maxHeapSize:  128 | availHeapSize:  119

5114-5340 D: Realm instance count > i: 3 | count: 45
5114-5340 D: Memory > usedMB: 9 | maxHeapSize:  128 | availHeapSize:  119

5114-5342 D: Realm instance count > i: 5 | count: 101
5114-5342 D: Memory > usedMB: 10 | maxHeapSize:  128 | availHeapSize:  118

5114-5341 D: Realm instance count > i: 4 | count: 205
5114-5341 D: Memory > usedMB: 12 | maxHeapSize:  128 | availHeapSize:  116

>>>>> 5114-5124 I: Background partial concurrent mark sweep GC freed 44725(1658KB) AllocSpace objects, 0(0B) LOS objects, 30% free, 8MB/12MB, paused 11.471ms total 108.491ms

5114-5339 D: Realm instance count > i: 2 | count: 247
5114-5339 D: Memory > usedMB: 9 | maxHeapSize:  128 | availHeapSize:  119

5114-5337 E: ashmem_create_region failed for 'indirect ref table': Too many open files

5114-5337 W: Throwing OutOfMemoryError "Could not allocate JNI Env"
5114-6313 E: jni: ThrowingException 5, Too many open files in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101, .
5114-6313 E: Exception has been thrown: Unrecoverable error. Too many open files in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101

Version of Realm and tooling

Realm version(s): 5.7.0

Realm Sync feature enabled: No

Android Studio version: ? 3.2.1

Android Build Tools version: ? 28.0.3

Gradle version: ? 3.2.1

Which Android version and device(s): ? android 6.0 (noname tablet)

[cmelchior]

Somewhere in your app, you are spawning a lot of threads that all open Realms and are not closing them. Most likely you are manually starting a ton of threads at the same time in a loop or you have a misconfigured ThreadPool somewhere that isn't killing old threads.

Hopefully you have crash in the Java code opening the Realm so you can track it down?

[ArilesGit]

I use a helper class to execute all my transactions. I'm pretty sure nowhere else (except in activities) do I open a new Realm instance. I used try-with-resource, so all instances should be closed.

Am I answering your question ?

Thanks for the fast reply ! :-)

[cmelchior]

I'm not sure what to tell you except you have a bug somewhere. Realm.getGlobalInstanceCount(mRealm) returns the number of threads that have active Realms on them, so if the number is increasing it is because you spawn a lot of threads somewhere.

[ArilesGit]

Yeah, it looks like you are right, even thought I can't see where the bug is right now. I'll investigate the problem and keep you posted. Thanks again.

[Zhuinden]

new Thread(() -> {

Don't do that, you'll run out of allocatable stack space. Use a single-threaded executor thread pool.

[ArilesGit]

@Zhuinden thanks for the tip !

Actually, I had a fixed thread pool executor with a 100 threads capacity and unlimited queue size (Integer.MAX) but I encountered a bug where the executor was stuck and I couldn't understand. I mean, how could all the 100 block at the same time !?

I know it's hard to tell without having a look at the code but if you you happen the have a best practice or an article to suggest, I'll be thankful.

[cmelchior]

Realm transactions are blocking across all threads and processes (to ensure consistency), so if you are trying to parallelize 100 writes, they will all block each other, only resolving one at a time.

[ArilesGit]

Yes, that is what I expected but it seemed to me that the executor was completely stuck. But I may be wrong and I unfortunately (or is it fortunately ?) I can't reproduce the bug.

An example of what I'm trying to achieve is have an executor service execute all transactions. There is some data I need to upload. I use a flag (i.e: uploadAt as a date) so that a periodic job knows if it needs to upload a particular entry of some Realm table. I expect the executor to prevent concurrency problems and inconsistencies.

Is it a good idea to have the executor as a static final class member knowing that my app acts as a launcher ? I'm afraid of a potential blocking situation.

Do you think it's a good idea to have several executors, a low priority one for background writings, and another one, less commonly used ?

I hope I'm not asking too much from you. Thanks for your time.

[cmelchior]

Having the executor as a static final field is fine and a common pattern. Although it can make testing harder, that tradeoff is up to you though.

Have multiple worker queues for expressing priority is also pretty common, e.g. the high priority queue is always emptied before the lower priority one. There are many ways to implement that. In the context of Realm transactions, I'm not sure two executors are worth it though as they are blocking each other anyway.

You can also look into Work Manager from Architecture Components which are designed specifically to handle these kinds of situations: https://developer.android.com/topic/libraries/architecture/workmanager/

[ArilesGit]

In the context of Realm transactions, I'm not sure two executors are worth it though as they are blocking each other anyway.

Oh of course, I missed that !

I'll have a look at "Work Manager", thanks !

Edit:
I already use evernote's android-job for that part. Thanks anyway.

[Zhuinden]

That's why single threaded pool is safest. You can't run multiple transactions at the same time anyway, so you may as well just use 1 thread that you reuse.

[ArilesGit]

It makes sense to me now. Thanks @Zhuinden !
I'll try to implement it and experiment with it to see if the app gets stuck in an IRL scenario.

[Zhuinden]

I've used that in production. If you cancel the transaction in case of error (or use executeTransaction) then it won't freeze