Realm-java: Feature Request: Manual "Refresh" (synchronization of Realm instance to latest version on background thread)

[Zhuinden]

Goal

What do you want to achieve?

I feel a certain set of de-ja-vu, but I'd like to execute a periodic task in an IntentService and ensure that the Realm is always up to date for querying data without having to execute a transaction on non-looper (or IntentService) background thread.

Expected Results

Call realm.refresh() on a background non-looper (or IntentService) thread, and the Realm instance should be synchronized to its latest version.

Actual Results

This method was removed in 0.89.0 in favor of waitForChange(), but that method is difficult to work with waitForChange()/stopWaitForChange() because you need to use your own thread, and you cannot wake it up periodically while it waits for a Realm change.

Code Sample

http://stackoverflow.com/questions/39600315/realm-not-fetching-data
or
http://stackoverflow.com/questions/38833163/realmchangelistener-does-not-get-called-when-realm-gets-updated-in-notificationl
or
http://stackoverflow.com/questions/39613453/realm-and-the-lack-of-limit-how-a-to-do-a-stepped-delete-operation/39618296#39618296

Version of Realm and tooling

Realm version(s): 1.2.0

Android Studio version: 2.2.0

Which Android version and device: Nexus 5X, 7.0.0

[zaki50]

Hi @Zhuinden

How about waitForChange(long) that accepts timeout parameter.

As far as IntentService, it should not be used for a very long term task and thus open/close approach should work, but I think that allowing to use waitForChange(long) on the IntentService is a possible option.

[Zhuinden]

Eh, I think refresh is better, but ONLY on background threads.

Disallowed on looping autoupdating threads.

(I'm not even sure how that timeout would work, to be honest)

[zaki50]

I assume that waitForChange(0) will return immediately and if there are new commits from other threads, it will update all live objects to the latest, and triggers all listeners. Of course it only works on non-looper threads (and IntentService?).

[Zhuinden]

...ah, so when the timeout is over, then the Realm is refreshed?

In that case that knows more than I anticipated, in which case it would solve the problem.

[zaki50]

waitForChange(long) returns immediately when other threads change some data.
Even if no other threads make any commits, it will return with false when the timeout period is over.

[stk1m1]

@zaki50

waitForChange(long) returns immediately when other threads change some data.
Even if no other threads make any commits, it will return with false when the timeout period is over.

This sounds cool! 👍 and I think the logic is already there in core. What puzzles me is if waitForChnage(timeout) is just the same as refresh with timeout. (i.e. refresh(timeout) for a given time period?)

---

@Zhuinden
AFAIK, Realm.refresh() is removed due to its 1) redundant nature and 2) confusing semantic.

1. For the first, you simply run a transaction block which will update the Realm instance to the latest change anyway (correct me if I'm wrong).
2. For latter, refresh() might miss the latest change of underlying Realm database since it's non-blocking, racy operation. i.e. there is no guarantee a Realm is up to date after a refresh().

waitForChange(), on contrary, is designed to be more restrictive, but it mitigates the issues above. I don't deny that waitForChange() has its issues and isn't the most user friendly API though.
(You can read more about the origin of waitForChange() at #2319 and #2386.)

[Zhuinden]

@stk1m1 refresh() was removed because it kills asynchronous queries on looper threads.

Non-looper threads do not have asynchronous queries, because they don't have a looper. Currently the only way to refresh a background thread's Realm is to run your code in a transaction, but that might not always be exactly what you want.

Here is an exact code where I'm trying to run multiple transactions on a background thread, but I'd like to keep the Realm instance up to date (which would require a Looper even for a local commit):

Realm realm = null;
try {
    realm = Realm.getInstance(realmConfiguration);
    int logsPerRequest = 100;

    RealmRefresh.refreshRealm(realm);
    while(realm.where(LogMessage.class).count() > 0) {
        realm.executeTransaction(new Realm.Transaction() {
            @Override
            public void execute(Realm realm) {
                RealmResults<LogMessage> result = realm.where(LogMessage.class).findAll();
                final List<LogMessage> pageResult = result.subList(0, Math.min(logsPerRequest, result.size()));
                JsonObject json = new JsonObject();
                JsonElement array = gson.toJsonTree(pageResult, new TypeToken<List<LogMessage>>() {}.getType());
                json.add("events", array);

                RequestBody body = RequestBody.create(JSON, json.toString());
                Request request = new Request.Builder().url(extras.getString(URL, "")).post(body).build();

                //Try to sync
                final Response response = client.newCall(request).execute();
                if(response != null && response.isSuccessful() && response.body() != null) {
                    //Delete the synced log messages
                    for(int i = pageResult.size() - 1; i >= 0; i--) {
                        LogMessage message = pageResult.get(i);
                        message.deleteFromRealm();
                    }
                }
            }
        });
        RealmRefresh.refreshRealm(realm);
    }
} finally {
    if(realm != null) {
        realm.close();
    }
}

waitForChange() works fine for the external commit RealmDaemon, but in most cases, you have a TimerTask on a background thread, you can't just wait forever, while making empty transactions is wasteful, and realm.beginTransaction()/realm.cancelTransaction() is just overkill in my opinion.

So if there were a refresh(), the real question is whether it'd be blocked by on-going transactions (in same process and other process)

[stk1m1]

@Zhuinden

refresh() was removed because it kills asynchronous queries on looper threads.

I think this sounds closer to why refresh() was removed.

waitForChange() works fine for the external commit RealmDaemon, but in most cases, you have a TimerTask on a background thread, you can't just wait forever

Just to be clear, waitForChange() wasn't initially meant to block forever. It was intended that stopWaitForChange() could _temporarily_ stop, and one could wait again by calling waitForChange(). Only after long discussions and numerous test cases, it became clear the two were racy ops due to lack of abstraction on how to handle interruption to an active Thread and their families in Android.

The next thing for Realm update that's much more clear is, I for one, to execute a transaction. I'd agree that it's wasteful though.

[Zhuinden]

@stk1m1 Do I remember correctly that commitTransaction() does not update the Realm instance on non-looper background threads, only on threads that have a looper? Based on code that seems to be the behavior, but the code that I linked would retain old version for a long time because the thread cannot sync to latest.

[stk1m1]

commitTransaction() does not update the Realm instance on non-looper background threads

Hmm... I _believe_ a transaction updates your Realm to the latest change on either Looper or non-Looper thread. Otherwise you're in the world of pain. Even the deprecated refresh() could have updated Realm on non-Looper thread.

[Zhuinden]

I'll need to verify whether commitTransaction() updates the Realm instance on non-looper threads then o-o

[Zhuinden]

public class MainActivity
        extends AppCompatActivity {
    Realm realm;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        if(RealmConfigHolder.INSTANCE.getRealmConfiguration() == null) {
            RealmConfigHolder.INSTANCE.setRealmConfiguration(new RealmConfiguration.Builder(getApplicationContext()).deleteRealmIfMigrationNeeded().initialData(
                    new Realm.Transaction() {
                        @Override
                        public void execute(Realm realm) {
                            Doge doge = new Doge();
                            doge.setName("Doge");
                            realm.insert(doge);
                            doge.setName("Bark");
                            realm.insert(doge);
                            doge.setName("Boner");
                            realm.insert(doge);
                        }
                    })
                    .build());
        }

        new Thread(new Runnable() {
            @Override
            public void run() {
                Realm realm = null;
                try {
                    Thread.sleep(10000);
                    realm = Realm.getDefaultInstance();
                    RealmResults<Doge> _doges = realm.where(Doge.class).findAll();
                    Log.i("_DOGES", "Size: [" + _doges.size() + "]");
                    realm.executeTransaction(new Realm.Transaction() {
                        @Override
                        public void execute(Realm realm) {
                            RealmResults<Doge> doges = realm.where(Doge.class).findAll();
                            Log.i("DOGES", "Size: [" + doges.size() + "]");
                            Doge doge = new Doge();
                            doge.setName("Goofball");
                            realm.insert(doge);
                        }
                    });
                    RealmResults<Doge> doges = realm.where(Doge.class).findAll();
                    Log.i("DOGES", "Size: [" + doges.size() + "]");
                    for(Doge doge : doges) {
                        Log.i("DOGE", "[" + doge.getName() + "]");
                    }
                } catch(InterruptedException e) {
                    e.printStackTrace();
                } finally {
                    if(realm != null) {
                        realm.close();
                    }
                }
            }
        }).start();
        realm = Realm.getDefaultInstance();
    }

    @Override
    protected void onDestroy() {
        if(realm != null && !realm.isClosed()) {
            realm.close();
        }
        super.onDestroy();
    }
}

And

09-23 00:08:05.087 11210-11222/zhuinden.com.testrealmonbgthread I/_DOGES: Size: [3]
09-23 00:08:05.087 11210-11222/zhuinden.com.testrealmonbgthread I/DOGES: Size: [3]
09-23 00:08:05.091 11210-11222/zhuinden.com.testrealmonbgthread I/DOGES: Size: [4]
09-23 00:08:05.091 11210-11222/zhuinden.com.testrealmonbgthread I/DOGE: [Doge]
09-23 00:08:05.091 11210-11222/zhuinden.com.testrealmonbgthread I/DOGE: [Bark]
09-23 00:08:05.091 11210-11222/zhuinden.com.testrealmonbgthread I/DOGE: [Boner]
09-23 00:08:05.091 11210-11222/zhuinden.com.testrealmonbgthread I/DOGE: [Goofball]

Verifies that executing a transaction updates the Realm instance even on a non-looper background thread.

Well that's one mystery solved.

---

A more interesting mystery:

public class MainActivity
        extends AppCompatActivity {
    Realm realm;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        if(RealmConfigHolder.INSTANCE.getRealmConfiguration() == null) {
            RealmConfigHolder.INSTANCE.setRealmConfiguration(new RealmConfiguration.Builder(getApplicationContext()).deleteRealmIfMigrationNeeded().initialData(
                    new Realm.Transaction() {
                        @Override
                        public void execute(Realm realm) {
                            Doge doge = new Doge();
                            doge.setName("Doge");
                            realm.insert(doge);
                            doge.setName("Bark");
                            realm.insert(doge);
                            doge.setName("Boner");
                            realm.insert(doge);
                        }
                    })
                    .build());
        }

        new Thread(new Runnable() {
            @Override
            public void run() {
                Realm realm = null;
                try {
                    Thread.sleep(4000);
                    realm = Realm.getDefaultInstance();
                    RealmResults<Doge> _doges = realm.where(Doge.class).findAll();
                    Log.i("[1]_DOGES", "Size: [" + _doges.size() + "]");
                    realm.beginTransaction();
                    RealmResults<Doge> doges = realm.where(Doge.class).findAll();
                    Log.i("[1]DOGES", "Size: [" + doges.size() + "]");
                    Doge _doge = new Doge();
                    _doge.setName("Goofball");
                    realm.insert(_doge);
                    realm.commitTransaction();
                    doges = realm.where(Doge.class).findAll();
                    Log.i("[1]DOGES", "Size: [" + doges.size() + "]");
                    for(Doge doge : doges) {
                        Log.i("[1]DOGE", "[" + doge.getName() + "]");
                    }
                } catch(InterruptedException e) {
                    e.printStackTrace();
                } finally {
                    if(realm != null) {
                        realm.close();
                    }
                }
            }
        }).start();

        new Thread(new Runnable() {
            @Override
            public void run() {
                Realm realm = null;
                try {
                    realm = Realm.getDefaultInstance();
                    RealmResults<Doge> _doges = realm.where(Doge.class).findAll();
                    Log.i("[2]_DOGES_1", "Size: [" + _doges.size() + "]");
                    Thread.sleep(10000);
                    Log.i("[2]_DOGES_2", "Size: [" + _doges.size() + "]");
                    realm.beginTransaction();
                    RealmResults<Doge> doges = realm.where(Doge.class).findAll();
                    Log.i("[2]DOGES", "Size: [" + doges.size() + "]");
                    Doge _doge = new Doge();
                    _doge.setName("Ballgoof");
                    realm.insert(_doge);
                    realm.cancelTransaction();
                    doges = realm.where(Doge.class).findAll();
                    Log.i("[2]DOGES", "Size: [" + doges.size() + "]");
                    for(Doge doge : doges) {
                        Log.i("[2]DOGE", "[" + doge.getName() + "]");
                    }
                } catch(InterruptedException e) {
                    e.printStackTrace();
                } finally {
                    if(realm != null) {
                        realm.close();
                    }
                }
            }
        }).start();
        realm = Realm.getDefaultInstance();
    }

    @Override
    protected void onDestroy() {
        if(realm != null && !realm.isClosed()) {
            realm.close();
        }
        super.onDestroy();
    }
}

And

09-23 00:14:19.223 16969-16989/zhuinden.com.testrealmonbgthread I/[2]_DOGES_1: Size: [3]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]_DOGES: Size: [3]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]DOGES: Size: [3]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]DOGES: Size: [4]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]DOGE: [Doge]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]DOGE: [Bark]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]DOGE: [Boner]
09-23 00:14:23.183 16969-16988/zhuinden.com.testrealmonbgthread I/[1]DOGE: [Goofball]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]_DOGES_2: Size: [3]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]DOGES: Size: [4]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]DOGES: Size: [4]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]DOGE: [Doge]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]DOGE: [Bark]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]DOGE: [Boner]
09-23 00:14:29.227 16969-16989/zhuinden.com.testrealmonbgthread I/[2]DOGE: [Goofball]

Which essentially means that a retained version existed after the thread awakened, but cancelling a transaction also brought the Realm to be up to date.

---

In the end, the question is whether there is a point to a refresh that refreshes every sync result on a background thread even though cancelling an active transaction also updates the Realm; and that whether there is a point of syncing to the latest version even while a write transaction is active.

[Zhuinden]

An interesting fact from Realm-Cocoa:

If a thread has no runloop (which is generally the case in a background thread), then -[RLMRealm refresh] must be called manually in order to advance the transaction to the most recent state.

Realms are also refreshed when write transactions are committed (-[RLMRealm commitWriteTransaction]).

Failing to refresh Realms on a regular basis could lead to some transaction versions becoming “pinned”, preventing Realm from reusing the disk space used by that version, leading to larger file sizes.

---

Realm-Swift:

If a thread has no runloop (which is generally the case in a background thread), then Realm.refresh() must be called manually in order to advance the transaction to the most recent state.

Realms are also refreshed when write transactions are committed (Realm.commitWrite()).

Failing to refresh Realms on a regular basis could lead to some transaction versions becoming “pinned”,

---

Technically, Realm for iOS does provide refresh() for updating background thread, without having to cancel a transaction explicitly.

[leisim]

I have this problem on Android with versions becoming "pinned" and leading to big files . I have many background threads (non Looper). Is this maybe related to Realms not being refreshed? It seems that closing Realms does not free all resources :/

[Zhuinden]

@leisim depends on what you mean by large, having too many versions creates very large files, unusually large ones.

If your Realm is not encrypted, then that's what Realm.compactRealm() is for RealmConfigurations that belong to a Realm which has no open instances. If you can verify that your large Realm files are reduced when you use compactRealm() on them, then yes, this was your issue

[leisim]

@Zhuinden Large means >800mb...

I know but the problem is that my app runs 24/7 for at least a month without being restarted.

[Zhuinden]

That's rough. You'd still need to compact the Realm from time to time, as long as your Realm isn't encrypted. But that requires for there to be no open Realm instances

[Zhuinden]

@kneth I just had an idea, what would be great is if it was possible to have a single-execution listener for when Realm updates on the UI thread.

Although I guess this could be done if one retains a list of RealmChangeListener bound to the Realm instance itself, which removes itself from said list when it's done.

[kneth]

@Zhuinden Can you outline your idea in code? Just to get an idea of the API you envision.

[cmelchior]

@Zhuinden
This could would achieve want you wanted?

        realm.addChangeListener(new RealmChangeListener<Realm>() {
            @Override
            public void onChange(Realm element) {
                // Do work ...
                realm.removeChangeListener(this);
            }
        });

[Zhuinden]

@cmelchior this works assuming the reference to RealmChangeListener is strong reference.

Is it strong reference? I tend to forget these. (walks off to check)

EDIT: it is strong reference. I guess that solves this issue for the UI thread.

But I think I drifted off a bit because _this thread_ is about how Realm.getDefaultInstance() doesn't guarantee to be the latest version when used in thread pools on non-looping background threads, and that "periodic tasks" should refresh to the latest version of the Realm (without having to explicitly start a transaction to read the latest version)

In fact, I'm pretty sure I was thinking of the "timing issue" introduced by the daemon thread.

[cmelchior]

It is. It was originally a weak reference, but we changed it at some point pre 1.0

[Zhuinden]

@kneth @cmelchior I think my mind did a "derp" because I was thinking of "how would I fix the issue of the Realm not being up to date when onPostExecute() is called or an event is received on the UI thread via the handler", which is a totally different issue, and not the one about refreshing background threads :slightly_frowning_face:

Although that won't be solved by a RealmChangeListener on the Realm either, because the results "might have already been synced by the daemon". I'll take that over to its own thread

[Zhuinden]

In the meantime, http://stackoverflow.com/a/38839808/2413303 works

[Zhuinden]

Woo! :D