Realm-java: Exception when trying to start write transaction after committing a previous one.

[levibostian]

In my code base, I am running all of my Realm queries that display on the UI via an async read transaction and all background queries via realm sync queries. I am running all realm writes in the background thread using:

    protected fun performRealmTransaction(changeData: (realm: Realm, done: () -> Unit) -> Unit) {
        if (ThreadUtil.isOnMainThread()) {
            throw RuntimeException("Don't perform transaction from UI thread.")
        }
        val realm = Realm.getDefaultInstance()
        realm.beginTransaction()

        changeData(realm, {
            realm.commitTransaction()
            realm.close()
        })
    }

Using this in my code:

                    performRealmTransaction { realm, commitToRealm ->
                        // do work with realm instance. Query, update values, copy/update models. 
                        commitToRealm()
                    }

I use these transaction blocks to do all of my writes. This code works just fine where I am creating a new instance, having the chance to write to it, then committing synchronously and closing the realm instance when I am done.

Here is where I am having issues. I am wondering if Realm is really running commitTransaction() synchronously.

Whenever I have code like this:

fun writeDataToRealm(): Completable {
    return Completable.create { subscriber ->
        performRealmTransaction { realm, commitToRealm ->
            // update user of app's name to foo. 
            // realm.getUser() is a Kotlin extension I created to get the 1 and only UserModel of app.
            realm.getUser().name = "foo"
            commitToRealm()
           subscriber.onCompleted()
        }
    }
}

fun doSomethingElseWithRealmAfterWriteToItAlreadyOnce() {
    writeDataToRealm().subscribe({
        // Here, we can assume that writeDataToRealm is complete, user's name has been changed to "foo" and realm has committed successfully. 
        // But it doesn't. We actually throw an exception in this line of code below because we are now trying to create a *new* realm transaction. We get an exception saying Realm is already in a write transaction but that's false. We called `commitTransaction()` before we called `subscriber.onCompleted()` which got us to this point. 
        performRealmTransaction { realm, commitToRealm ->
            // this will not run because exception thrown while creating to create new realm transaction. 
            commitToRealm()
        }
    }, { error -> })
}

If I take this exact same code but add a small RxJava delay, it works successfully. No realm exception thrown:

fun writeDataToRealm(): Completable {
    return Completable.create { subscriber ->
        performRealmTransaction { realm, commitToRealm ->
            // update user of app's name to foo. 
            // realm.getUser() is a Kotlin extension I created to get the 1 and only UserModel of app.
            realm.getUser().name = "foo"
            commitToRealm()
           subscriber.onCompleted()
        }
    }
}

// *HERE* adding `.delay(300, TimeUnit.MILLISECONDS)` and it works. 
fun doSomethingElseWithRealmAfterWriteToItAlreadyOnce() {
    writeDataToRealm().delay(300, TimeUnit.MILLISECONDS).subscribe({
        // Here, we can assume that writeDataToRealm is complete, user's name has been changed to "foo" and realm has committed successfully. 
        // But it doesn't. We actually throw an exception in this line of code below because we are now trying to create a *new* realm transaction. We get an exception saying Realm is already in a write transaction but that's false. We called `commitTransaction()` before we called `subscriber.onCompleted()` which got us to this point. 
        performRealmTransaction { realm, commitToRealm ->
            // this will not run because exception thrown while creating to create new realm transaction. 
            commitToRealm()
        }
    }, { error -> })
}

If I give realm a small bit of time, no exceptions thrown. This has been my method of getting this working in various parts of my app. It fixes it, but this cannot be right. Must be a better way, something I am doing wrong.

Goal

What do you want to achieve?

I do not expect realm to throw an exception about already being in a realm write transaction. It should have ended via commitTransaction() and .close(). Realm is not in the middle of a write transaction. It ended.

Actual Results

IllegalStateException: The Realm is already in a write transaction in ...

Version of Realm and tooling

Realm version(s): 2.2.1

Realm sync feature enabled: no

Android Studio version: 2.2.3

Which Android version and device: Nougat, Marshmallow

[Zhuinden]

You're reusing the Realm instance multiple times for multiple transactions on a threadpool.

Realm instances are cached in the RealmCache for a thread, afterwards reference-counted.

So if you start opening multiple transactions on the same thread, then you'll be telling the same instance multiple times to start a transaction - which will result not in a "blocking", but in an IllegalStateException.

I think realm.executeTransaction() would be easier to manage.

[levibostian]

Thanks, @Zhuinden. realmIntance.close() must not really close the realm instance then? I do understand realm instances are cached on threads, but thought realm.close() would close that reference and decrement the count resulting in my 2nd write transaction getting a fresh new realm.

Either way, I will give realm.executeTransaction() a shot. I will see if that fixes my issue and allows me to remove the .delay() statements.

[dalinaum]

Hello @levibostian

realmIntance.close() must not really close the realm instance then? I do understand realm instances are cached on threads, but thought realm.close() would close that reference and decrement the count resulting in my 2nd write transaction getting a fresh new realm.

I think it depends on your scheduler. What scheduler do you use? How do you invoke writeDataToRealm and doSomethingElseWithRealmAfterWriteToItAlreadyOnce on your code?

[levibostian]

I use RxJava, Schedulers.io() to kick off the background task.

[kneth]

@leonardoxh If so, shouldn't you include the scheduler as a third parameter when calling delay?

[levibostian]

@kneth, I am trying to figure out a way to not require delay because it's a hack. RxJava seems to be setup correctly with the way I need it. In fact, the delay code I am using does not have the scheduler attached to it directly because the function that calls it contains the scheduler. It's already in the background thread.

[kneth]

@levibostian If you omit delay and try to read from your Realm in doSomethingElseWithRealmAfterWriteToItAlreadyOnce, can you then see the changes done in writeDataToRealm?

[levibostian]

@kneth, if the delay is there, no exception will be thrown, and yes, I can see the changes done in realm. If I omit delay, I am not sure if I can see the changes done in realm because the IllegalStateException: The Realm is already in a write transaction in ... exception is thrown.

[kneth]

@levibostian You don't need a transaction to query. You might also check if realm.isInTransaction() returns true.

[levibostian]

@kneth I understand I do not need a transaction to query. I am performing write transactions (hence receiving the IllegalStateException: The Realm is already in a write transaction in ... exception.

realm.executeTransaction() has been working better for me so far as far as removing this exception.

The reason for this GitHub issue is because I am receiving the exception: IllegalStateException: The Realm is already in a write transaction in ... and from the code samples I have above, it doesn't make sense to be receiving this issue because I have committed and closed the realm instance.

From my initial testing (cannot make any promises until I have run my Android app for a week or two with this code change) realm.executeTransaction() in place of realm.commitTransaction() and realm.close() has been working. From the Realm documentation and the point of realm.commitTransaction(), I would not expect this exception to be thrown. That is the point of this issue.

[kneth]

The reason I was suggesting the query and realm.isInTransaction() was to get an understanding on the the two "tasks" will be scheduled. commitTransaction() should not return before the write transaction is committed and the Realm is back to being in a read state (and ready to start a new write transaction).

[levibostian]

Ok, I see.

My codebase has changed a lot since I created this issue. I will have to go back and find a commit the code was at this state and then run some tests for you.

[kneth]

@levibostian Thanks. I'll leave the issue open and let you take the time you need to test.

[kneth]

@levibostian Did you have a chance to test?

[levibostian]

@kneth I am not seeing the exception anymore, but I am still having to use delay all over my code. Yesterday and today I had to do it more.

It seems from my testing that places where I am doing more changes to the realm in a transaction, I have to set the delay higher to give realm time to refresh compared to really small changes in other places of my app.

In this other issue's comment I realized that Realm takes some time to refresh on other threads.

What I do in my code is:

• User performs some action
• Using RxJava, perform a realm transaction in a background thread
• Immediately after the realm write transaction, I try to read these changes in the realm from the UI thread. This is where I am required to add the delay because the realm needs time to refresh and have my write transaction data available on the UI thread for me to read.

I need to read the data and show it to the user. Right now to solve this issue, I am adding delays and creating delayed Handlers all over my code to give realm time to refresh. It is (1) hacky, (2) delays the UI for the user (3) not 100% working as sometimes the realm takes longer to refresh then other times.

Is there a better way? Is there a way that I can be alerted on the UI thread when these changes are changed and ready for me? I know about realm change listeners, but that doesnt seem as though it will work for my needs because maybe a background task updated the realm model and called the realm listener, not the code that I called at that moment from the user's action.

Am I going about this wrong? It is annoying having to guess when realm is going to be refreshed with my new data.

[kneth]

@levibostian Thanks for the feedback/observations. We have recently (version 2.3.0) fixed another (unrelated) issue in the notification subsystem (see #4002). Can you try to upgrade?

[Zhuinden]

@kneth immediately reading on the UI thread won't work because it is not yet up to date https://github.com/realm/realm-java/issues/3427 you should rely on RealmChangeListeners

@levibostian yeah it is indeed rather unfortunate that Realm 2.x has changed how Realm change notifications work, but

if you can obtain a reference to the AndroidNotifier of the Realm from io.realm package via ((AndroidNotifier)realm.sharedRealm.realmNotifier).notifyCommitByOtherThread...

class AndroidNotifier implements RealmNotifier {
    private Handler handler;

    @Override
    public void notifyCommitByOtherThread() {
        if (handler == null) {
            return;
        }

        // Note there is a race condition with handler.hasMessages() and handler.sendEmptyMessage()
        // as the target thread consumes messages at the same time. In this case it is not a problem as worst
        // case we end up with two REALM_CHANGED messages in the queue.
        boolean messageHandled = true;
        if (!handler.hasMessages(HandlerControllerConstants.REALM_CHANGED) &&
                !handler.hasMessages(HandlerControllerConstants.LOCAL_COMMIT)) {
            messageHandled = handler.sendEmptyMessage(HandlerControllerConstants.REALM_CHANGED);
        }
        if (!messageHandled) {
            RealmLog.warn("Cannot update Looper threads when the Looper has quit. Use realm.setAutoRefresh(false) " +
                    "to prevent this.");
        }
    }

But I think unfortunately the handler is private; if you can somehow set a Handler which you have access to and send REALM_CHANGED notification to it after realm.commitTransaction(), then you'll be able to force the UI thread to update immediately after a commit.

I do not know if this causes any strange issues though

[levibostian]

@Zhuinden, @kneth when I perform a Realm write commit, the data is written to the realm database in memory immediately, correct?

If I wrote to thread A, immediately after the write commit I can read that written data on thread A because they are on the same thread.

Thread B, however, needs to wait X amount of time to read the data as it was written on thread A. It needs to wait to get notified.

The key word here is notified, right? I can read the data on thread B that was written on thread A because, it's just a database. The data is the same no matter what thread it wrote from. I can read the data from thread B but I need to wait to get notified from Realm.

The notification is delayed. Not the data? If this is true, @Zhuinden's suggestion sounds as though it may work. As it is private I do not feel comfortable trying it on my production app I am building.

Why is it that Realm is built this way? I am curious if I am the only one who has this need? Being able to write in a background thread and immediately reading that written data on the UI thread sounds like a pretty common use case to me. User does something, perform database interaction, update UI to reflect changes. But right now, I have to do: User does something, perform database interaction, delay half a second, update UI to reflect changes.

I am up for suggestions to trying a different pattern. If I am the only one with this use case, I feel as though I must be using Realm in a wrong way then others.

[Zhuinden]

User does something, perform database interaction, update UI to reflect changes.

This is what RealmChangeListeners were designed for in the first place: when there is a write to the database, it gives you the notification that tells you that you should be showing new data on the UI.

---

Technically what I said here would be very hacky, the only "reasonable" way about it would be to use Reflection to obtain the private Handler inside the notifier, but that's also not very reasonable either (a later Realm update could easily move the Handler elsewhere)

It is possible to force the local thread with mere package-private API, but that's not recommended on the UI thread imo (because it turns all async queries into sync queries).

---

But yes, the notification is delayed, because it relies on the external commit helper instead of directly sending a message to Android's handler.

This was a change in behavior I reported in 2.0.0, it breaks async tasks, but the consensus is that "RealmChangeListeners should be sufficient". ( https://github.com/realm/realm-java/issues/3427 )

Also, currently notification is delayed until all async queries are executed, so that you don't need to do isLoaded() checks on your RealmResults obtained by findAll*Async() when the RealmChangeListeners are called.

I wonder if this will change in 2.4 though.

[levibostian]

Thank you for that.

As it stands, I am going to continue to use delays in my app (for the use cases defined below) and cross my fingers. RealmChangeListeners are great and I use them as Observables, but does not do everything that I need. One use case in particular that I have in my head and where this whole discussion sprouted from mostly is when I create new data in realm. Change listeners say "when this data changes, notify me." But when I create new data, I cannot create a listener.

User enters data on a screen in my app, app creates new model in realm, UI shows that UI if successful write from the Realm data that was just created. I do create a change listener after the user creates the data for all future UI updates, but until the data is created I cannot do so.

So for updates to Realm, this is not much of an issue. Except when dealing with RecyclerViews. I don't like to call notifyDataSetChanged() every time the realm list is refreshed. I like to animate rows that update manually for a smoother UI/UX. Besides that, updates are pretty smooth. Creating data and working with RecyclerViews are 2 use cases I can think of now I am trying to have more control over in my app where: User performs action, I write to the realm database, if write successful, I need to reflect those changes to the UI.

Above 2 use cases are the places in my app that I have delays created waiting for the realm to refresh. These are the 2 use cases at the moment I have encountered. There may be more.

[Zhuinden]

RealmResults<T> automatically receives a notification to its RealmChangeListener if you write an object of its type into the Realm.

But yes, notifyDataSetChanged() is a problem. I have figured out a convenient way with Rx to create animatable results, but it requires copying the whole result set on any change, so it wouldn't work that well with larger data sets.

[levibostian]

Also, I should add. I run background tasks quite a bit with my app. Running tasks in the background that interact with the realm database and network. When the background job executes, it runs on a background thread and does everything it needs on the same thread so all of it's data is kept in sync together but at any time the user or another thread could change the realm data at any time. It is assumed that all writes will be persisted to the realm.

With this in mind, it is important that the realm data is safe when writes are performed and data will not clash.

---

@Zhuinden, Yes, I can create a listener on RealmResults I suppose. I could create a realm query immediately after the realm write specifying the exact ID of the realm object that I want then wait until the notification comes in. I don't want to query the whole data set, I would specify the exact model via equalTo() query params.

This seems to be the least messy solution so far besides delays. Least amount of code lines and smell. Not super convenient, but may have to do. I will look into creating some sort of generic call to make it much simpler to do...I will think about that one. In the meantime, delays still seem to be my best solution.

[Zhuinden]

All writes are persisted, but the notification to the UI thread is slightly delayed.

Hence why you have this problem.-

[levibostian]

Are notifications delayed to the UI thread only?

Could I do this: realm.copyToRealmOrUpdate(NewFooModel("joe")).addChangeListener(getChangeListenerInUIThread()) and get notified when that model is created even though at this point it's not created?"

[Zhuinden]

1.) non-looper background threads don't get notifications

2.) if there is no strong reference to your RealmResults or RealmObject then GC can eat it and then its change listener will no longer be called

btw I didn't mention it here yet, but with package-private api, it is possible to refresh a local thread - but it forces all async queries to be immediately loaded

[cmelchior]

@levibostian To clarify:

1) Each thread gets it's own version of Realm data when you open the instance the first time on that thread.
2) The version will be refreshed for a number of reasons:

a. If you start a write transaction (`realm.beginTransaction()`), the version will be updated to latest version, but notifications will not be triggered.

b. If you fully close a Realm on that thread and re-open it, it will be at the latest version.

c. If it is a Looper thread (like the UI thread), it will update when it receive a Looper to do so. That event will be sent from any other thread doing the write. This will be delayed for two reasons. The first is if there is a lot of events in the Looper message queue, and secondly, if you have any async queries running, those will be rerun before your thread is updated. Notifications will only be triggered when a looper event has come in and all queries rerun (this might change with #3834)

That is why we say that you _really_ should use RealmChangeListeners as the mean to get notified about changes, because trying to send your own events have a very high chance of doing it wrong.

[levibostian]

@cmelchior @Zhuinden appreciate it very very much. I understand the reasoning behind the change listeners now. Thank you.

Now is where I am thinking about the best method to be alerted as soon as data has been created in Realm. As mentioned before, I have change listeners setup via RxJava Observables for updates to realm, but as far as creations of new models, I am stuck on how to create listeners for that.

It sounds like what I need to do is create a change listener on a RealmResults list and then I will need to iterate the list on each update and look for the created model that was created? That does not seem right. How would you recommend being notified on the UI thread when a Realm model has been created?

Could I....

In the UI thread, create a realm query: realm.where(FooModel.class).equalTo("realm_id", newModelRealmId).findAll(), add a change listener to it, and wait until it notified me with a RealmResults list length > 0 indicating the model has been created?

Sounds like the best option thus far.

[Zhuinden]

In the UI thread, create a realm query: realm.where(FooModel.class).equalTo("realm_id", newModelRealmId).findAll(), add a change listener to it, and wait until it notified me with a RealmResults list length > 0 indicating the model has been created?

Sounds like the best option thus far.

As long as you keep the RealmResults in a queue and you remove it from the queue when the RealmChangeListener is called (and also remove the RealmChangeListener itself), that will work.

You can get some ideas from here

[levibostian]

Ok. I will try that this week and get back to you. I will see how I like it and give suggestions if it could be better.

Cannot thank the team enough for the help. Thank you, thank you. Chat soon.

[levibostian]

Created a solution:

        return Completable.create { subscriber ->
            val realm = Realm.getDefaultInstance()
            realm.where(FooModel::class.java).equalTo("realm_id", newRealmId).findAllAsync().asObservable()
                    .subscribe { data ->
                        if (data.count() > 0) {
                            realm.close()
                            subscriber.onCompleted()
                        }
                    }
        }.subscribeOn(AndroidSchedulers.mainThread()).observeOn(AndroidSchedulers.mainThread())

I need to run the app a couple dozen more times this week to see if this solution is consistently working, so far it has been with a dozen runs.

As I have been testing this issue the past couple of weeks, I have encountered https://github.com/realm/realm-java/issues/3802 consistently. I clean install my app, login, run this code above ~5 times successfully then the 6th time the BadVersionException is thrown. Uninstall, clean install and repeat. I am subscribed to https://github.com/realm/realm-java/pull/3834 to watch that issue's progress.

---

It would be awesome if I could create this "create listener" after I first create the model: realm.copyToRealmOrUpdate() and the above code runs. That would be ideal as an API standpoint. This works for now if it does end up working.

I have been working this afternoon to create a generic version of this above solution but I have ran into this issue: https://github.com/realm/realm-java/issues/3817 while doing so with what I believe to be the correct order of Android, Kotlin, Realm plugins. Will continue to work on it.

[cmelchior]

@levibostian If you are able to consistently reproduce the BadVersionException we would be very interested in knowing exactly how. How exactly is the code you refer to being run 5/6 times in a row?

[levibostian]

@cmelchior I have a write transaction that I perform after the user enters text into an EditText and pushes the save button.

I am running the write transaction only once each time they press the save button. When I say I am running my code 5/6 times and then I get the BadVersionException crash, I am pressing the button 5/6 times for 5/6 individual writes.

I start a Fragment, fill out info, press save, perform write transaction, press back, repeat. So I am starting with a fresh new Fragment each time.

[cmelchior]

Hi @levibostian Sorry for the late reply.
Are you able to share your project with us. I suspect you are seeing this due to some other factor not covered by the code you posted. If you want to do it privately, you can use [email protected] ?

[levibostian]

@cmelchior I emailed the email address. Correct, the BadVersionException is not from the code I am talking about in this issue specifically.

[levibostian]

I am talking with someone on the [email protected] email address about the BadVersionException issue.

About the original issue, I have created a solution that still works:

doRealmWriteTransactionInBackgroundThread().andThen(Completable.create { subscriber ->
            val realm = Realm.getDefaultInstance()
            realm.where(FooModel::class.java).equalTo("realm_id", newRealmId).findAllAsync().asObservable()
                    .subscribe { data ->
                        if (data.count() > 0) {
                            realm.close()
                            subscriber.onCompleted()
                        }
                    }
        }).subscribeOn(AndroidSchedulers.mainThread())

I do a realm write transaction in background and then immediately after I am doing a realm query on the UI thread waiting until the data.count > 0 which indicates that the data is available on the UI thread realm instance.

This has been working so far. Now I ask, is there a better way? It sounds as though there is not a better in the current Realm implementation. Would it be a good idea to add functionality to the library in the future for a RealmCreateListener to alert you when the data has been written and is available to the attached listener? Essentially doing what I am doing in code above but in the library and a better API?

Would that be of interest in the future sometime?

[Zhuinden]

I think you probably meant isLoaded()

[levibostian]

I read the docs for isLoaded. I think I know what you mean for using it:

doRealmWriteTransactionInBackgroundThread().andThen(Completable.create { subscriber ->
            val realm = Realm.getDefaultInstance()
            realm.where(FooModel::class.java).equalTo("realm_id", newRealmId).findFirstAsync().asObservable()
                    .subscribe { data ->
                        if (data.isLoaded() && data.isValid() && data != null) {
                            realm.close()
                            subscriber.onCompleted()
                        }
                    }
        }).subscribeOn(AndroidSchedulers.mainThread())

Is this the use you were talking about @Zhuinden? It didn't make the code much more simple but it was another way of writing it. I would have to test the above code to see if it works the same way as my previous example.

[Zhuinden]

I'm a bit skeptical. If findFirstAsync() returns an invalid value (object not found in database), your subscriber will never complete, and hte Realm instance will never be closed.

[bmeike]

Hey @levibostian,
I'd like to close this discussion. It is getting pretty diffuse. We are tracking the BadVersionException issue in private email. If there is anything else, at all, that we can help you with, I think it might be better to open a new, more focused, ticket. Please don't hesitate to do that!

[levibostian]

@Zhuinden, I do understand your point, however, (1) there is no other way to do it and (2) there should never be an instance that this situation would occur (I stress should, haha). If I am indeed creating a new object in realm and realm does not throw an exception, I can safety assume that this code will succeed, subscriber will complete, and realm instance will close.