Realm-java: Migration crashes on newly created object

[jpmcosta]

Goal

I'm trying to migrate my database. The migration creates new objects that hold lists of other objects, in order to speed-up search queries. It's related to #6601.

Actual Results

The migration unexpectedly crashes with:

Fatal Exception: java.lang.IllegalStateException
Object is no longer valid to operate on. Was it deleted by another thread?
io.realm.internal.UncheckedRow.nativeGetIndex

Steps & Code to Reproduce

This is the relevant part of the code:

val allItems = realm.where("Item").sort("updatedAt", Sort.ASCENDING).findAll()
val itemCount = allItems.size
if (0 < itemCount) {
    var id = NO_ID
    val maxItemCount = 5_000
    var fromIndex = 0
    do {
        val toIndex = min(itemCount, fromIndex + maxItemCount)
        val itemsSubList = allItems.subList(fromIndex, toIndex)
        val items = RealmList<DynamicRealmObject>().apply { addAll(itemsSubList) }
        val lastUpdatedAt = itemsSubList.last().getLong("updatedAt")

        val itemPage = realm.createObject("ItemPage", ++id)
        itemPage.setList("items", items) // Might crash here (most common).
        itemPage.setLong("lastUpdatedAt", lastUpdatedAt) // Might crash here (less common).

        fromIndex = toIndex
    } while (toIndex < itemCount)
}

Notes:

• this issue is in production
• I never run into this issue while testing; I've migrated databases with 5_000 and 50_000 items
• I'm using multi-process

Version of Realm and tooling

Realm version(s): 5.14.0

Realm Sync feature enabled: No

Android Studio version: 3.5

Android Build Tools version: 28.0.3

Gradle version: 5.4.1

Which Android version and device(s): 7.1, 8.1, and 9 (currently)

[jpmcosta]

I've also found a similar issue on StackOverflow:
https://stackoverflow.com/questions/56006227/realm-db-dynamicrealmobject-not-valid-during-migration

[cmelchior]

@jpmcosta Hmm, it is rather surprising that it is indeterministic which line crashes

Are you saying that you have stack traces for both a crash here:

itemPage.setList("items", items) // Might crash here (most common).

and here:

itemPage.setLong("lastUpdatedAt", lastUpdatedAt) // Might crash here (less common)

If that is true, that sounds really puzzling and would point to some kind of race condition, but since a migration runs inside a write transaction, no other changes should be allowed.

Are they crashing with the same error?

[jpmcosta]

@cmelchior these are the errors:

Fatal Exception: java.lang.IllegalStateException: Object is no longer valid to operate on. Was it deleted by another thread?
       at io.realm.internal.UncheckedRow.nativeGetIndex(UncheckedRow.java)
       at io.realm.internal.UncheckedRow.k + 2(UncheckedRow.java:2)
       at io.realm.DynamicRealmObject.setModelList + 137(DynamicRealmObject.java:137)

and

Fatal Exception: java.lang.IllegalStateException: Object is no longer valid to operate on. Was it deleted by another thread?
       at io.realm.internal.CheckedRow.nativeSetLong(CheckedRow.java)
       at io.realm.internal.UncheckedRow.b + 10(UncheckedRow.java:10)
       at io.realm.DynamicRealmObject.setLong + 148(DynamicRealmObject.java:148)

I've released an update where I would create the ItemPage objects before iterating the list, but that didn't solve the issue.

I've released a more recent update where I've changed allItems to:

val allItems = realm.where("Item").sort("updatedAt", Sort.ASCENDING).findAll().createSnapshot()

(it even speeds up the performance of the migration)

And I also catch a possible exception when setting the list and the long values, and retry to find or create the ItemPage. I still don't have much data about this update.

[jpmcosta]

Also, Item as a @LinkingObjects to its ItemPage.

[jpmcosta]

The crashes are still occurring with the createSnapshot() call. They seem to be happening less frequently. However, when they do happen, by catching the exception and retrying to find the object, I think I was able to prevent all migration issues.

These are the crashes that still happen. ItemPage's schema is also created in this migration:

Fatal Exception: java.lang.IllegalArgumentException
Illegal Argument: Field not found: lastUpdatedAt
io.realm.internal.CheckedRow.nativeGetColumnIndex (CheckedRow.java)
io.realm.internal.UncheckedRow.a (UncheckedRow.java:4)
io.realm.DynamicRealmObject.setLong (DynamicRealmObject.java:146)

[Zhuinden]

Sounds like in certain versions of the app, lastUpdatedAt does not exist and should have been added as part of the migration in case the field is not there.

[jpmcosta]

@Zhuinden thank you for your feedback. Unfortunately, I'm afraid it's not that simple.

lastUpdatedAt is always added and, after catching the exception, retrying to find the object, and re-applying the migration, it works.

[bmunkholm]

@jpmcosta If this is now resolved, will you close the issue?

[jpmcosta]

@bmunkholm I don't consider this issue is resolved. As @cmelchior stated, it seems there are racing conditions that are making migrations fail, but that shouldn't happen since we are inside a transaction.

Personally, I believe this issue is very serious and should have top priority. When migrations don't work, applications become dead. However, if you feel this is an edge-case and unimportant, maybe someone could add a "won't fix" label and close it.

[cmelchior]

No, this issue shouldn't be closed. There is clearly a bug that we need to find.

[bmunkholm]

Ahh, sorry @jpmcosta I just read your last two words: "it works" :-)

[Zhuinden]

There should be a block across all processes as the transaction over a dynamic realm is still a transaction.

It would be rather surprising if there truly is a race condition during initialization / migration...

I'd assume Realm would need a way to repro this in order to proceed further.