Realm-java: Memory and file exceeds size issue

[ssunkar]

Goal

To allow app to work by solving realm memory issues

Steps & Code to Reproduce

06-12 13:45:51.895 16425-16425/? D/dalvikvm: Late-enabling CheckJNI
06-12 13:45:51.895 16425-16425/? D/dalvikvm: Try to disable coredump for pid 16425
06-12 13:45:51.895 16425-16425/? D/dalvikvm: Process 16425 nice name: com.tcrsoftware.android
06-12 13:45:51.895 16425-16425/? D/dalvikvm: Extra Options: not specified
06-12 13:45:51.935 16425-16425/com.tcrsoftware.android I/MultiDex: VM with version 1.6.0 does not have multidex support
06-12 13:45:51.935 16425-16425/com.tcrsoftware.android I/MultiDex: install
06-12 13:45:51.935 16425-16425/com.tcrsoftware.android I/MultiDex: MultiDexExtractor.load(/data/app/com.tcrsoftware.android-2.apk, false)
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android I/MultiDex: loading existing secondary dex files
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android I/MultiDex: load found 1 secondary dex files
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android I/MultiDex: install done
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android D/dalvikvm: DexOpt: couldn't find static field Landroid/os/Build;.SUPPORTED_ABIS
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android W/dalvikvm: VFY: unable to resolve static field 162 (SUPPORTED_ABIS) in Landroid/os/Build;
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android D/dalvikvm: VFY: replacing opcode 0x62 at 0x0006
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android D/dalvikvm: DexOpt: couldn't find static field Landroid/os/Build;.SUPPORTED_ABIS
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android I/dalvikvm: DexOpt: unable to optimize static field ref 0x00a2 at 0x0b in Lcom/getkeepsafe/relinker/SystemLibraryLoader;.supportedAbis
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android D/dalvikvm: Trying to load lib /data/app-lib/com.tcrsoftware.android-2/librealm-jni.so 0x43a301e0
06-12 13:45:51.945 16425-16425/com.tcrsoftware.android D/dalvikvm: Added shared lib /data/app-lib/com.tcrsoftware.android-2/librealm-jni.so 0x43a301e0
06-12 13:45:51.995 16425-16425/com.tcrsoftware.android D/REALM: jni: ThrowingException 11, mmap() failed: Out of memory size: 1207959552 offset: 0 in io_realm_internal_SharedGroup.cpp line 113, .
06-12 13:45:51.995 16425-16425/com.tcrsoftware.android D/REALM: Exception has been throw: Unrecoverable error. mmap() failed: Out of memory size: 1207959552 offset: 0 in io_realm_internal_SharedGroup.cpp line 113
06-12 13:45:51.995 16425-16425/com.tcrsoftware.android D/AndroidRuntime: Shutting down VM
06-12 13:45:51.995 16425-16425/com.tcrsoftware.android W/dalvikvm: threadid=1: thread exiting with uncaught exception (group=0x43712160)
06-12 13:45:51.995 16425-16425/com.tcrsoftware.android E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.tcrsoftware.android, PID: 16425
io.realm.exceptions.RealmError: Unrecoverable error. mmap() failed: Out of memory size: 1207959552 offset: 0 in io_realm_internal_SharedGroup.cpp line 113
at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(Native Method)
at io.realm.internal.SharedGroup.openSharedGroupOrFail(SharedGroup.java:95)
at io.realm.internal.SharedGroup.(SharedGroup.java:74)
at io.realm.internal.SharedGroupManager.(SharedGroupManager.java:49)
at io.realm.BaseRealm.(BaseRealm.java:86)
at io.realm.Realm.(Realm.java:135)
at io.realm.Realm.createAndValidate(Realm.java:233)
at io.realm.Realm.createInstance(Realm.java:214)
at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:126)
at io.realm.Realm.getInstance(Realm.java:178)
at com.tcrsoftware.android.MainApplication.onCreate(MainApplication.java:44)
at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1007)
at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4384)
at android.app.ActivityThread.access$1500(ActivityThread.java:138)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1296)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:149)
at android.app.ActivityThread.main(ActivityThread.java:5061)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:610)
at dalvik.system.NativeStart.main(Native Method)
06-12 13:46:42.055 16425-16425/? I/Process: Sending signal. PID: 16425 SIG: 9

Code Sample

MultiDex.install(this);

    RealmConfiguration config = new RealmConfiguration.Builder(mainContext).schemaVersion(0).migration(migration).build();
    Realm.setDefaultConfiguration(config);
    realm = Realm.getInstance(config);

```

Version of Realm and tooling

Realm version(s): ? version = "1.2.0"

Realm sync feature enabled: yes/no

Android Studio version: ? 2.2.3

Which Android version and device: ? Android N. Galaxy s7 and moto g5 plus

[Zhuinden]

According to the exception message, it says your Realm file grew to 1.125 GB in size, and cannot be opened in memory.

This is typically caused by if the following two conditions are met:

• opening Realm instances randomly on non-looping background threads inside thread pools (for example, AsyncTask or Schedulers.io())
• There is at least 1 local Realm instance that is not closed

For example, common cause is code like this (on background thread)

Realm.getDefaultInstance().where(...)....

instead of (on background thread)

try(Realm realm = Realm.getDefaultInstance()) {
    ....

It is also possible to reduce the amount of space that Realm has "retained from deletions" using the Realm.compactRealm() method, but that doesn't magically fix the misuse of Realm instances (meaning not closing them).

[ssunkar]

Yes. I'm calling AsyncTask for every 2 minutes. And realm instances are opened number of times and I might not closing it to avoid crashes relating to realm instances already closed.

Is there any way (tool or method) that checks the unnecessary instances to close

[ssunkar]

for example to my above comment:
I got this exception:
java.lang.IllegalStateException: This Realm instance has already been closed, making it unusable.
06-13 12:27:07.956 27123 27123 E AndroidRuntime: at io.realm.BaseRealm.checkIfValid(BaseRealm.java:444)
06-13 12:27:07.956 27123 27123 E AndroidRuntime: at io.realm.RealmResults.isLoaded(RealmResults.java:872)
06-13 12:27:07.956 27123 27123 E AndroidRuntime: at io.realm.RealmResults.iterator(RealmResults.java:273)
06-13 12:27:07.956 27123 27123 E AndroidRuntime: at java.util.AbstractCollection.toString(AbstractCollection.java:450)
06-13 12:27:07.956 27123 27123 E AndroidRuntime: at com.tcrsoftware.android.activities.TimeCardActivity.endTimeCard(TimeCardActivity.java:673)

The querywhich is used to fetch data:
final RealmResults timeCache = laborDB.getTimeCard();
for (final LaborMiscTime entry : timeCache) {
//
}

getTimeCard() method:
public RealmResults getTimeCard() {
realm = Realm.getDefaultInstance();
RealmResults laborMiscTimes = realm.where(LaborMiscTime.class).equalTo("IsActive", true).findAll();
realm.close();
return laborMiscTimes;
}

[ssunkar]

and my app size grows to 17.29 GB

[Zhuinden]

See https://realm.io/docs/java/latest/#closing-realm-instances and you can also see https://realm.io/docs/java/latest/#strategies-when-dealing-with-android-framework-threads for notes about using Realm in AsyncTask

Also:

If you need to use Realm in either of these methods you should open the Realm, perform your work and then close the Realm before exiting.

and

Realm uses an internal reference counted cache so that, after getting the first Realm instance, getting subsequent instances on the same thread is free. The underlying resources are released, though, only when all of the instances on that thread are closed.

Basically, open the Realm instance for duration of thread, and pass the Realm instance to the methods that need to operate on it.

---

But there's also a note on recommended lifecycle management for the UI Thread.

[ssunkar]

Thanks for the answer.
Can you please help me with this code

MainClass.java:
HandlerThread hThread = new HandlerThread("HandlerThread");
hThread.start();

    final Handler handler1 = new Handler(hThread.getLooper());

    deleteRealmLogsRunnable = new Runnable() {
        @Override
        public void run() {
            if (tcrLogDB == null) tcrLogDB = new TCRLogDB();
            RealmResults<TCRLog> tcrLogs = tcrLogDB.getLogs(); // this method is seen below
            tcrLogDB.deleteAllRealmLogs(tcrLogs);
            handler1.postDelayed(deleteRealmLogsRunnable, 120000);
        }
    };
    handler1.postDelayed(deleteRealmLogsRunnable, 120000);

LaborDB.java:
public RealmResults getLogs(){
realm = Realm.getDefaultInstance();
RealmResults tcrLogs = realm.where(TCRLog.class).findAll();
realm.close(); // below error occurs
return tcrLogs;
}

Error:
Method threw 'java.lang.IllegalStateException' exception.
This Realm instance has already been closed, making it unusable.

[Zhuinden]

I do not understand why you have so many handlers and threads and handlerthreads, and I don't know what thread you are calling these methods on :slightly_frowning_face:

[ssunkar]

the above mainclass.java (UI class) code is in one method which is called on button click.
once the button clicked the code runs for every 2 minutes

[Zhuinden]

The only advice I can give is to follow the documentation on how to manage Realm lifecycle on the background thread and on the UI thread.

[ssunkar]

Yes. I tried this way:
public RealmResults getLogs(){
realm = Realm.getDefaultInstance();
try {
RealmResults tcrLogs = realm.where(TCRLog.class).findAll();
return tcrLogs;
}finally {
realm.close();
}
}

but stil it returns the error saying : Unable to evaluate the expression Method threw 'java.lang.IllegalStateException' exception.
This Realm instance has already been closed, making it unusable.

[Zhuinden]

But that's not the recommended way.

Recommended way for background thread is

protected Void doInBackground(Void... params) {
    Realm realm = Realm.getDefaultInstance();
    try {
        RealmResults<TCRLog> logs = getLogs(realm);
        // do other things with Realm
    } finally {
        realm.close();
    }

    return null;
}

public RealmResults<TCRLog> getLogs(Realm realm) {
    return realm.where(TCRLog.class).findAll();
}

[ssunkar]

Sorry, I think I'm confusing you.

My requirement is I just need to fetch the data from realm by closing and opening its instance.

this is the code:
TCRLogDB tcrLogDB = new TCRLogDB();
RealmResults tcrLogs = tcrLogDB.getLogs();

TCRLogDB.class:
public RealmResults getLogs(){
realm = Realm.getDefaultInstance();
RealmResults tcrLogs = realm.where(TCRLog.class).findAll();
realm.close(); // below error occurs
return tcrLogs;
}

Unable to evaluate the expression Method threw 'java.lang.IllegalStateException' exception.
This Realm instance has already been closed, making it unusable.

There is no background thread involved here. Please explain

[ssunkar]

do i need to create realm instance in onCreate if calling from activity instead of creating instances in each method when calling from activity?

[zaki50]

You should not close Realm instance in getLogs() if you will use RealmResults outside of the method.

do i need to create realm instance in onCreate if calling from activity instead of creating instances in each method when calling from activity?

yes.

[ssunkar]

Thanks @zaki50
Is there any way I can close realm instance by creating them each in method instead of creating just one in onCreate.

Right now I'm not creating any realm instances in Activities. Strange thing is its working in some cases and not in some cases

[beeender]

@ssunkar

Is there any way I can close realm instance by creating them each in method instead of creating just one in onCreate.

no, you have to make sure the Realm instance is not closed if you plan to use any RealmObject/RealmResults/RealmList which are attached to the instance.

[ssunkar]

But why its working in some cases:
for example:

The folllowing is the code which is in one activity: (xxx.java extends AppCompatActivity)
LaborDB laborDB = new LaborDB();
final RealmResults timeCache = laborDB.getTimeCard();
for (final LaborMiscTime entry : timeCache) {
/// able to get records
}

//getTimeCard() method is in LaborDB class

public RealmResults getTimeCard() {
realm = Realm.getDefaultInstance();
RealmResults laborMiscTimes =
realm.where(LaborMiscTime.class).equalTo("IsActive", true).findAll();
realm.close();
return laborMiscTimes;
}

This is working perfectly without throwing any error related to realm instance.
Any idea what's the difference?
Here I'm not passing ream instance from the activty but still it works

[beeender]

We are using ref count to manage Realm instances internally. See below example:

Realm realm1 = Realm.getDefaultInstance();
// Since realm1 and realm2 are all on the same thread with same configuration, `getDefaultInstance()` will just return the SAME object.
Realm realm2 = Realm.getDefaultInstance();

realm1 == realm2 // TRUE!

realm1.close() // ref count -= 1
realm1.isClosed() // FALSE!! since `getDefaultInstance()` has been called twice on the same thread
realm1 == realm2 // still TRUE!!!
realm2.close(); //
realm2.isClosed() // true. Finally the ref count reaches 0, the instance is closed.

[zaki50]

Realm instance returned from Realm.getDefaultInstance() is reference counted in each thread.
When the Realm.getDefaultInstance() iscalled, the count is incremented and close() decrements it.
close() actually closes the Realm instance only when the counter reaches 0.

So, if you already called Realm.getDefaultInstance() and corresponding close() is not yet called, realm.close() in the getTimeCard() doesn't close actually.

[Zhuinden]

@ssunkar technically for UI thread access it is possible to globally count number of all open activities and keep 1 global ui thread realm instead of using onCreate/onDestroy, see https://stackoverflow.com/a/44337690/2413303

But you still need to pass Realm instance to methods instead of opening/closing in each method, especially if you need it on both UI thread and background thread.

@beeender really? I thought it provides different "local instance" and each gets closed individually. :open_mouth:

[beeender]

@Zhuinden

really? I thought it provides different "local instance" and each gets closed individually. 😮

That was the plan ... see https://github.com/realm/realm-java/issues/4573
But the semantics have been like this from day one, maybe some users have code relies on this behaviour, I am not sure it if is a good idea to refactor the semantics.

[ssunkar]

@zaki50 @beeender
now I understood that realm instances should be passed as parameter through activities or fragments.
I assume by the above discussion that I was lucky in some cases as there were some other realm instances which were not closed.

Now I'm trying to close all the instances from beginning of my code.

is it ok to not close just one instance to avoid the error "This Realm instance has already been closed, making it unusable" ???

keeping this just once instance causes memory leak exceptions??

[beeender]

By practise, have a Realm instance on UI thread won't be a big issue if you are always need it. And since there is a looper on UI thread, the Realm instance will be refreshed automatically.

On a background thread, yes, you have to close the Realm instance properly otherwise there will be leaks.

But those totally depend on your architecture. this may help https://realm.io/docs/java/latest/#controlling-the-lifecycle-of-realm-instances

[ssunkar]

Thanks for clear explaination.

So for a quick fix can i do this way?

The folllowing is the code which is in one activity: (xxx.java extends AppCompatActivity)
LaborDB laborDB = new LaborDB();
final RealmResults timeCache = laborDB.getTimeCard();
for (final LaborMiscTime entry : timeCache) {
/// able to get records
}
laborDB.closeRealm(); // closing realm instance after using its realmObject/realmResults

//getTimeCard() and closeRealm() method is in LaborDB class

LaborDB.java
public RealmResults getTimeCard() {
realm = Realm.getDefaultInstance();
RealmResults laborMiscTimes =
realm.where(LaborMiscTime.class).equalTo("IsActive", true).findAll();
realm.close();
return laborMiscTimes;
}
public void closeRealm() {
if (realm!=null) {
if (!realm.isClosed())
realm.close();
}
}

[Zhuinden]

No, the only quick fix would be to set ThreadLocal<Realm> at the start of the thread, close it at the end of the thread, and get the Realm instance from the ThreadLocal.

[ssunkar]

is it possible to use realmResults by the above step after closing its instance?

[Zhuinden]

RealmResults is a thread-local proxy view to the database.

So not unless all the data is copied before closing the Realm, which can be slow with large amount of data

[ssunkar]

can you help how to find which method producing this below warning:
06-20 15:09:59.911 28865-28874/com.tcrsoftware.android W/REALM: Remember to call close() on all Realm instances. Realm /data/data/com.tcrsoftware.android/files/default.realm is being finalized without being closed, this can lead to running out of native memory.

I have like hundreds of transactions running in my application. where to check the closing of the realm instance?

[Zhuinden]

Ctrl+Shift+F for Realm.getDefaultInstance() is a start, if the realm instance you obtain from it isn't close'd anywhere.

[ssunkar]

Is there anyway to copy RealmResults (to create detached copies) sams like RealmObjects?

so that even if we close realm instances, I can use it from the stores memory

[Zhuinden]

Yes, with realm.copyFromRealm(), but detaching means eager evaluation which can be slow with a large number of objects, and you lose the ability to add RealmChangeListener to your results (because it is detached)

[ssunkar]

1) so now I'm closing all the realm instances (after realm= realm.getdefaultinstance).
2) If i have to use RealmResults then I'm copying to List using copyFromrealm and then closing its instance.

Without this workaround I noticed warnings related to realm ( for ex: Remember to call close() on all Realm instances. Realm /data/data/com.tcrsoftware.android/files/default.realm is being finalized without being closed, this can lead to running out of native memory.).

But after the above steps I did not see any such warnings(I test for half day with number of transactions).

Does this solved my issue or is there any better way to test memory usage?

[beeender]

It should solve your issue I think if can you ensure all Realm instances which are used on the bg thread are closed properly.

the warning log will be printed if the Realm instance gets GCed before Realm.close() called on it.

[ssunkar]

ohk thanks.

I have problem converting List to RealmResults.

How to achieve this?

[Zhuinden]

Only a query executed against an open Realm returns a RealmResults.

A managed RealmList can also be converted to RealmResults using realmList.where().findAll().

[ssunkar]

@Zhuinden is there any way i can send my code for review.

The problem is as I am copying the realmresults (using copyFrpmrealm) to List, I'm unable to modify the obtained list in particular transaction.

so either I need to convert that List to realmresults or copyFromRealm should results realmresults again.

[Zhuinden]

An unmanaged object can be used to update existing RealmObjects if their primary key matches and insertOrUpdate is used (in a transaction, of course)

[kneth]

@ssunkar Did the comment above answer the question?

[ssunkar]

as of now there is no issue regarding implementation.
can I open this again if there is any issues regarding this?

[beeender]

@ssunkar Sure. Also please notice that we are going to release a new feature compactOnLaunch() which may help you on this situation. See https://github.com/realm/realm-java/pull/4857

Let us know if you still need help on this.