Realm-java: Batch or bulk updates

[cmelchior]

Feature request following this discussion: https://groups.google.com/forum/#!topic/realm-java/6NeywfWmCA4

Basically it allows setting a property on many objects in one operation just like this SQL:

UPDATE Foo SET isDeleted = 1 WHERE name = "123"

One option could be

RealmResults<Foo> foos = realm.where(Foo.class).equalsTo("name", "123");
realm.beginTransaction();
foos.updateAllObjects("isDeleted", 1);
realm.commitTransaction();

[kneth]

This should definitely be implemented mostly in C++ for better performance.

[bmunkholm]

The priority will be low as there are clear and easy workarounds for this. This is mainly a performance optimization.

[finnschiermer]

When it comes to code that makes changes in the database, the cost of the
commit that
will follow it will most likely dwarf all other contributions, so it might
not make a significant difference
in performance.

On Mon, Jan 19, 2015 at 9:41 AM, Brian Munkholm [email protected]
wrote:

The priority will be low as there are clear and easy workarounds for this.
This is mainly a performance optimization.

—
Reply to this email directly or view it on GitHub
https://github.com/realm/realm-java/issues/762#issuecomment-70460794.

[bmunkholm]

That all depends on the number of objects you want to update and if you persist to disk or possible in-memory. But commit to disk and updating few objects, you are right.

[finnschiermer]

The cost of commit does grow with the amount that is committed, the
break-even point
may be further out than we'd like.

/Finn

On Mon, Jan 19, 2015 at 11:16 AM, Brian Munkholm [email protected]
wrote:

That all depends on the number of objects you want to update and if you
persist to disk or possible in-memory. But commit to disk and updating few
objects, you are right.

—
Reply to this email directly or view it on GitHub
https://github.com/realm/realm-java/issues/762#issuecomment-70471435.

[njovy]

It would be great if a batch processing feature is implemented as this feature can add the IN clause.

[RobertoArtiles]

+1 Will really make things look prettier.

[basememara]

What exactly is the workaround? Is it this:

RealmResults<Foo> results = realm.where(Foo.class).equalTo("read", false).findAll();

for (int i = 0; i < results.size(); i++) {
    results.get(i).setRead(true);
}

I tried this and it doesn't update anything after 200 records for me, is there a limit or a soft fail if it reaches a memory threshold? I'm wrapping the above code in a realm.executeTransaction. A method for updateAllObjects makes total sense.

[saket]

@basememara It's a wrong idea to loop over a RealmResults while updating it. Since you're changing the RealmObjects (Foo in your case), the results is getting auto-updated on each update. Your indices will never be consistent!

[kneth]

@Saketme It depends on where you begin your transaction. As setRead() requires a transaction, the code could be

realm.beginTransaction();
RealmResults<Foo> results = realm.where(Foo.class).equalTo("read", false).findAll();

for (int i = 0; i < results.size(); i++) {
    results.get(i).setRead(true);
}
realm.commit();

[saket]

No, sorry not being clear earlier. I am talking about something else. This line of code get all Foo items that are "unread", right?

realm.where(Foo.class).equalTo("read", false).findAll();

Now, if you iterate over it using a for-i loop, setting each item to "read", it will cause the RealmResults list to auto-update itself on every iteration.

On every iteration, on every item that gets marked as "read", the RealmResults list will contain one less item, because it's only supposed to contain "unread" items.

As a result, the for loop will skip every alternate item.

[kneth]

@Saketme Well, you can do

realm.beginTransaction();
RealmResults<Foo> results = realm.where(Foo.class).equalTo("read", false).findAll();

for (int i = results.size() - 1; i >= 0; i--) {
    results.get(i).setRead(true);
}
realm.commit();

[saket]

Aha! That's a really nice idea :D Thanks!

[Zhuinden]

Seriously though, this is in iOS here:

https://realm.io/docs/swift/latest/#key-value-coding

Realm really will need something like this, because creating 3000 objects to update 3000 properties isn't very good for GC.

[cmelchior]

I tend to agree.
Having RealmCollection.setValue(String fieldName, Object value) would probably be relatively easy and straight forward. It would not support manipulating a RealmList but that is probably outside the scope of this issue anyway.

[cmelchior]

For reference: https://www.reddit.com/r/androiddev/comments/5npdk2/big_realmresult_how_to_iterate_through_all_of_it/, some users are hitting memory issues because of this, so we should probably prioritize getting it in relatively soon.

[Zhuinden]

Any chance of this one appearing any time soon?

It's kinda a necessity for working with larger data sets.

[Zhuinden]

@cmelchior Out of curiousity, then realmList.where().findAll().setValue(fieldName, value) wouldn't work either?

[kneth]

@cmelchior @Zhuinden Realm Core doesn't have a such a setter so if we were to implement setValue(), we have to loop. Of course we could move the looping to C++ to lower the memory pressure.

[Zhuinden]

@kneth but iOS has this 😕

Applying KVC to a collection is a great way to update objects in bulk without the overhead of iterating over a collection while creating accessors for every item.

RLMResults<Person*> *persons = [Person allObjects];
[[RLMRealm defaultRealm] transactionWithBlock:^{
  [[persons firstObject] setValue:@YES forKeyPath:@"isFirst"]; // <------
  // set each person's planet property to "Earth"
  [persons setValue:@"Earth" forKeyPath:@"planet"]; // <-------
}];

Creating a new object proxy for each updated element is bad for memory usage

[cmelchior]

@Zhuinden We just talked about this internally, and we want to bump the priority on this, but because we don't want to do the work twice, we would like to move this functionality to our Object Store. Because a generic setValue() method require some information about the underlying schema we depend on #4324. Fortunately, @kneth is resuming work on that this week.

[cmelchior]

Just to iterate the current design:

• [ ] Add a new setValue(String fieldName, Object value) method to RealmCollection.
• [ ] Determine how to handle unmanaged RealmLists's. We can only support them using reflection, so I would be tempted to just disallow that use case.
• [ ] Validation of fieldName vs. type of input should happen in Object Store. Currently blocked on #4324.
• [ ] Determine how to iterate all objects to update. It can be done in either Object Store or Core. Both kinda makes sense.

// Public API Example
RealmResults<Person> guests = realm.where(Person.class).equalTo("inviteSent", false).findAll();
realm.beginTransactions();
persons.setValue("inviteSent", true);
realm.commitTransaction();

[cireficc]

I am very excited for this feature, as I have a million and one for loops iterating to set a property. This will make my code just a little bit cleaner, which is always appreciated! Great work guys 👍

[ursusursus]

Very needed, my "move to trash" function which only sets a flag but has to iterate through large dataset takes even 1 minute possibly

[bmeike]

@cireficc You realize, of course, that this will simply put that loop in our code, instead of yours ;-P

[Zhuinden]

@bmeike if you can move that loop off the Java heap, that'd be great for larger data sets though :stuck_out_tongue_winking_eye:

[cmelchior]

Yes, that is the primary reason to move this into c++. It becomes zero-allocation on the Java side.

[cireficc]

@bmeike Exactly, then my code will be just a tad bit cleaner ;P

[Zhuinden]

Technically the real thing here is that updating would take O(N) memory which is bad, SQLite can do this with UPDATE without any allocation. So this will help remove a possible bottleneck in Realm if you ask me

[Zhuinden]

So, uh, is this still in RMP 2.0 milestone?

(and how did I unassign bmeike? that is odd. oh well)

[kneth]

@Zhuinden Maybe you are the magician?

[cmelchior]

@Zhuinden Yes, this is still scheduled for RMP 2.0 milestone, but not a high priority so if things get tight you have to wait for 4.1 😄

[VenomVendor]

Any updates, we are in 4.3.1?

[eugens-github]

It seems there is a snapshot concept, for at least a natural iterate/update scenarios: https://realm.io/docs/java/latest/#iterations-snapshots
So the gotcha "Never loop over RealmResults and update its items" is not a real problem if you do not access the result items by index.

[alexei-28]

In my Android application I has pojo:

public class Product extends RealmObject {
    @PrimaryKey
    private long id;
    private float price;
    private boolean favorite;
}

Suppose in Realm table I has 100 products. Nice.
Now I want to update property favorite = trueonly of 5 products with id= 1, 14, 38, 78, 90.

I need something like this:

UPDATE Product SET favorite = true WHERE id in [1, 14, 38, 78, 90]

How I can do this?

Thanks.

[cmelchior]

realm.beginTransaction();
RealmResults<Person> persons = realm.where(Product.class).in("id", new long[] { 1, 14, 38, 78, 90 }).findAll();
for (Person p : persons) {
  p.setFavorite(true);
}
realm.commitTransaction();

Should work

[cmelchior]

Support has been added in https://github.com/realm/realm-java/pull/5133. It will be part of 5.8.0

[Zhuinden]

Rejoice! 🎊 🎉 🎈 🍾 🎆

[cmelchior]

I thought you might like it 😉