Realm-java: could not create WeakRealmNotifier ALooper message pipe: Too many open files

[kneth]

From https://github.com/realm/realm-object-store/issues/319 (see attached files there)

hello,
I try to evaluate different databases inside one android app- Therefore I created different scenarios. The scenarios are different in the size and count of objects I will store.

All calls to the realm database came from a class I called "RealmDataManager". Inside this I implement differend methods which I need for my evaluation.

I use a IntentService which gets started from my Mainactivity. Inside my IntentService I defined the different scenarios and what to do for each.

At the Moment I have one problem. After my 4th Szenario (from 90 szenario's) I got the following Output in Logcat,

01-06 13:51:54.957 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.959 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.963 14995-15012/de.gerding.dbcompare E/Parcel: dup() failed in Parcel::read, i is 0, fds[i] is -1, fd_count is 1, error: Too many open files
01-06 13:51:54.966 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.967 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.972 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.974 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.980 14995-15012/de.gerding.dbcompare E/Parcel: dup() failed in Parcel::read, i is 0, fds[i] is -1, fd_count is 1, error: Too many open files
01-06 13:51:54.982 14995-15012/de.gerding.dbcompare E/Surface: queueBuffer: error queuing buffer to SurfaceTexture, -22
01-06 13:51:54.982 14995-15012/de.gerding.dbcompare I/Adreno: QueueBuffer: queueBuffer failed
01-06 13:51:54.982 14995-15012/de.gerding.dbcompare W/OpenGLRenderer: swapBuffers encountered EGL_BAD_SURFACE on 0x7f82d04940, halting rendering...
01-06 13:51:54.984 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:54.985 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:55.014 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
01-06 13:51:55.016 14995-15287/de.gerding.dbcompare E/REALM: could not create WeakRealmNotifier ALooper message pipe: Too many open files
...

Can you help me to figure out what I should change to solve this?

It's not a big problem, it looks like it works although this logcat output gets spammed. But I am not sure if it works for a long time. In my mainactivity I show an ProgressDialog when I start my "MessService", and this ProgressDialog in style circle doesn't move anymore when I got the logcat spam.

I will attach you some of my classes.
RealmDataManager - where all calls to the database are implemented.

MessService - where all scenarios are defined and you can see how I use the RealmDataManager
I take some comments inside my MessService so that not alle scenarios are used at the moment because some scenarios needs long time and this is not fine for debugging.

RealmCANDaten - one of my two objecttypes stored in the database

RealmFahrt - the other objecttype stored in the database

If you need more information for reproducing please let me know that. I hope we can fix this issue together.

[kneth]

Which version of Realm Java do you use? In version 1.1.1 we reduced the number of files opened. Android has a upper limit of how many files, you can open at the same time (the exact number varies from version to version). It looks like you are opening many databases (I quickly looked at your code - I might have overseen something).

Btw, when doing benchmarks on Android, System.currentTimeMillis() is not always the best way to measure time. Especially when your benchmarks are short. The nano timer can be a better solution.

I image that you have an app for doing these benchmark. If you can share it (public github repo or send a zip file to [email protected]), we can reproduce it and get a better understand of these log messages.

[Zhuinden]

He has weak notifier, so that means it is at least 2.x and object store based notifications.

[Miger88]

Hello,
I am using realm version 2.1.0. I cannot publish my app because of copyrights. But I copied all relevant and write a small app with same problem. I hope this helps you to understand how I want to work. I can't send a mail with this project as zip because 50MB is to much for a mail. Because of this I put it in my private dropbox and you can download it from there.
You can download the zip file here.

Regards
Miger88

[Miger88]

@kneth I know that System.currentTimeMillis is not the most accurate. But it has less cpu usage so I decided to use this. I use this for all databases I evaluate. So the inaccuracy is the same for all tested databases.

[Zhuinden]

Does this only happen when using tests, using @Before and @After with Realm.getDefaultInstance()/realm.close()?

@beeender is it possible that close() doesn't unregister the pipe?

[Miger88]

What did you mean with @before/ @after?
All my tests are implemented in the MessService.java class.

In each Method of my RealmDataManager I call Realm.getDefaultinstance at the beginning and realm.close at the end.

[beeender]

@Zhuinden Yes, very much like there is a bug in cleaning up the pipes ... I can reproduce this issue, making a fix now.

[beeender]

The problem is https://github.com/realm/realm-object-store/blob/master/src/util/android/event_loop_signal.hpp#L41

When the Realm is created in an IntentService, an ALooper fd is created anyway. But since the looper in IntentService doesn't "loop", there is no chance to close the relevant pipes see https://github.com/realm/realm-object-store/blob/master/src/util/android/event_loop_signal.hpp#L124

[beeender]

The problem is caused by https://github.com/realm/realm-dotnet/pull/551 . Checking if the thread is an IntentService thread won't be an idea fix since it will still leak FD if the looper quits through Looper.quit(). For a generic solution, a lock might need to be introduced to event_loop_signal.hpp and also a Alooper ref needs to be acquired to ensure all relevant FDs are closed peacefully. @fealebenpae may have some input for this?

[beeender]

And more thing needs to be fixed is:
if the looper quits through Looper.quit(), all the shared_ptr left in the read fd need to be read and deleted. otherwise we are leaking shared_ptr.

[Miger88]

My MessService is an IntentService right. At the begin of our project we make small tests between AsyncTask, Thread and IntentService. At the end we could see that using an IntentService is better for performance then using AsyncTasks or Threads. This is the reason why the MessService is an IntentService.

Do you think you can fix this for IntentServices?

[beeender]

@Miger88 Yes, it can be fixed (and it should be.). I will try to prioritize this.

[fealebenpae]

@beeender I think the changes in https://github.com/realm/realm-dotnet/pull/551 can be reverted, because the EventLoopSignal class now has neither a copy, nor a move assignment operator. This means we should be able to close both ends of the pipe in the destructor instead of in the looper callback.

We can also ALooper_acquire the looper as you pointed out, so that we don't have to rely on ALooper_forThread in the destructor, which should give us additional flexibility.

Lastly, in the destructor, prior to closing the read end of the pipe, we can read() it to make sure we don't close with any dangling shared_ptrs that won't be consumed (and freed) by the looper callback.

I'm not too well-versed in the intricacies of the Android Looper and I've only tested my work on the EventLoopSignal (née WeakRealmNotifier) in a regular Activity, so I'm not sure if the changes I propose will actually be robust in the face of IntentService and Looper.quit(), but I'd be more than happy to team up with you if you feel you need my help.