Realm-java: Support partial updates of a RealmObject

[snowpong]

Goal

I want to update an object touching only the values that are actually set.

More specifically, we have a REST API that follows the expand fields
"design pattern", similar to Confluence here https://developer.atlassian.com/confdev/confluence-server-rest-api/expansions-in-the-rest-api . Basically it means that you can receive the same JSON object as a very terse object (maybe only an ID) or a fully expanded object (fields, like title, url , child objects etc.).

The returned objects are deserialized by Retrofit2 and GSON into POJOs that extends RealmObject, and we pass these to copyToRealmOrUpdate(..).

Expected Results

Existing objects are updated, preserving fields that are not set.

Actual Results

Existing objects are overwritten, fields are overwritten with null.

Comments

Now, I understand why after reading the fine-print in the documentation https://realm.io/docs/java/1.2.0/api/io/realm/Realm.html#copyToRealmOrUpdate-E-

...copying an object will copy all field values. Any unset field in the object and child objects will be set to their default value if not provided...

but your method name suggest otherwise. This is not an update, that is a set(..) or overwrite(..).

Is there any scenario where you actually update using this method?

It's almost as if Retrofit (or any converting from JSON to POJO) is discouraged with the current behavior/method since the complementing method https://realm.io/docs/java/1.2.0/api/io/realm/Realm.html#createOrUpdateObjectFromJson-java.lang.Class-org.json.JSONObject- has the behavior I'm after. Would it not make sense to provide a similar functionality when "updating" with POJOs to?

[Zhuinden]

Technically the object with the same primary key is updated to be the new object.

[kneth]

Naming is one of the hardest tasks in implementing any API. As @Zhuinden points out, objects with primary key benefit from this method. Maybe you can take another look at your model classes, and see if they have fields which can be used as primary key.

[snowpong]

@kneth You misunderstand. My model classes have a primary key. I was simply surprised that copyToRealmOrUpdate(..) overwrites all fields as that is not what I expected to happen given the method name.

To give some more context. Imagine the following REST API

1. GET projects/
2. GET project/{id/}
3. GET place/{id}

The first one returns an array of Projects that contain an array of Places who are just specified by an _ID_
The second one returns a Project with an array of Places that have an _ID, a name, and a location._
The third one returns a Place that has _an ID, a name, and a location and an array of images_

Now, If you use Retrofit and deserialize this into Realm Models / POJOs and call copyToRealmOrUpdate(..) on the different results you get from the above endpoints, you get into trouble. You have no idea what state your Places are in. Some could be complete, some could be partial, because every time you store the results from Retrofit into Realm you overwrite whatever other version is there.

I agree that naming is hard. You have two similar methods, one called copyToRealmOrUpdate(..)that takes a RealmModel, and another called createOrUpdateObjectFromJson(..) that takes a RealmModel.class and a JSONObject. One could be fooled that they behave similarly. Yet, one overwrites all fields, the other one does not. One behaves more like HTTP PUT the other one more like HTTP PATCH yet their both called _update_.

[cmelchior]

The problem is that in a POJO you cannot tell the difference between a field not being set and the field being set to the default value.

E.g

private String name = null;

By looking at that you don't know if nobody set the field yet, or if it was actually set to null. This is the reason why the methods accepting POJOs have to update all fields.

With JSON it is possible to express that distinction by just removing the field completely.

Note that when we support polymorphism ( #761 ) it would be possible to express it this way:

public class PartialPerson extends RealmObject {
  private long id;
  private String name;
}

public class FullPerson extends PartialPerson {
  private int age;
}

[snowpong]

@cmelchior You could tell if it's the default value (using for example https://google.github.io/guava/releases/19.0/api/docs/com/google/common/base/Defaults.html) and allow us to skip the update for those fields as a behavior?

GSON has a default rule about not serializing null fields. At the moment I'm considering serializing my POJOs back to JSON and calling createOrUpdateObjectFromJson(..) on them. However, that requires me to write a TypeAdapter for _all my models_ (which is quite a few) because of this shortcoming https://realm.io/docs/java/latest/#serialization

[cmelchior]

As a design guideline we try to stay as close to the Java language as possible, but I did not know about the Guava Defaults. I will take a look, thanks.

Having a specific override or annotation that makes Realm ignore default field might be an idea as well, although just having a Partial/Full model seems to be the most friction free approach that doesn't force a lot of new API surface to Realm.

But for solutions right now, it is probably easier to just ask Retrofit to return the JSON from your server and use that directly: http://stackoverflow.com/a/31112405/1389357

[snowpong]

@cmelchior Was that the correct reddit link? I get a comment that doesn't mention JSON nor Retrofit

[cmelchior]

Ups, fixed

[TR4Android]

@cmelchior I would also be very interested in a feature that doesn't overwrite the already existing values. Generally the following options come to mind (for the specific use case of my app):

• Usage of the provided JSON methods, however this doesn't really work for me as all variables are prefixed with a m (like mId) and other variable names are also different from its JSON representation. You might want to provide a @SerializedName annotation that allows to specify the variable name in its JSON representation to cover more use cases.
• Have the copyOrUpdate method ignore any fields that are the default value when updating objects. That kind of makes sense, since it already should be the default value from object creation and if it's different someone purposely changed it.
• Have the setters of a RealmObject track if they've been called and only update the corresponding fields if it was called. Since you're already modifying the code (I think) this might be an option. However this has the limitation that it only sensibly works if the setter methods are used, direct field access wouldn't work.

[snowpong]

@cmelchior Thanks for the updated link. Now, if I were to do as you suggest, and just ask Retrofit to return the JSON from the server and use that directly in a call to createOrUpdateObjectFromJson(..) then GSON is ignored right? You have your own internal parser for converting JSON into Realm Models I assume? So @SerializedName would be ignored for example.

[cmelchior]

Yes, that is correct.

[cmelchior]

After looking at Guava I don't think that is a solution, so to summarize:

Use case
In REST API's it is not uncommon to have two end points where one return the "full" object and the other a "partial" object. This makes sense when you want to limit the amount of data sent.

It should be possible to easily update RealmObjects using both end points

Current situation

• copyToRealm will copy _all_ fields, which means that you cannot use 1 RealmObject + GSON for both end points
• Using the JSON response directly means you will loose the field mapping features of GSON

Solutions on the way or with issues

• Two endpoints could be modeled by inheritence ( #761 )
• Realm offers their own @SerializedName annotation. This could also be useful when sharing schemas with other platforms.
• Being able to ignore fields when updating: https://github.com/realm/realm-java/issues/2288

Right now I'm leaning towards Inheritence probably being the simplest way of solving this problem as all the other solutions will add a lot of overhead to our API. Even if we add @SerializedName which is highly likely, then it doesn't really make the described usecase _simple_.

[snowpong]

@cmelchior In the inheritance solution, we would have to implement two models: MyPartialFooModel and MyFullFooModel. And I assume the full one extends the partial one.

Question 1: How would I query Realm to get any available FooModel, and how would it prefer the full one if it exists? I can't picture the query nor the return type. Is there an abstract Model as well?

Question 2: Would this solve the use case (mentioned elsewhere) where local fields (not coming from REST API JSON) are overwritten when updating? I guess we could have a PartialRestFooModel, FullRestFooModel, and FullRestFooModelWithLocalValues.

[TR4Android]

I agree with @snowpong, while inheritance is quite a powerful feature, the model classes would quickly get unwieldy if we need to adjust them for such use cases. Not to mention that it's not exactly intuitive (at least for me).

Since I'm using JSON for the data sync, having a @SerializedName annotation would be good enough for me. In order to actually work with the data though, I'd need an additional annotation, something like @SerializedObjectId to cover cases where the server only sends the id of the object involved, but the code contains an actual Java object. Something like this maybe:

public class TestObject extends RealmObject {
    @SerializedName("testObjectId") @PrimaryKey
    long mId;
    @SerializedRealmObjectId("relatedObjectId")
    RelatedObject mObject;
    // other variables
}

public RelatedObject extends RealmObject {
    @SerializedName("relatedObjectId") @PrimaryKey
    long mId;
    // other variables
}

This would allow me to properly process server output using the copyFromJson methods, e.g. for the following sample JSON (most online databases probably use a similiar structure);

{
    "relatedObjects": [{"relatedObjectId":1, ...}, {"relatedObjectId": 2, ...}, ...],
    "testObjects": [{"testObjectId": 1, "relatedObjectId": 2, ...}, {"testObjectId": 2, "relatedObjectId": 2, ...}, ...]
}

Might still be worth though looking more into ignoring fields as an additional pattern that developers can use to cover other cases. Thanks for all the consideration and work 👍

[kneth]

@TR4Android Thanks for your suggestions.

@snowpong We haven't really found a good answer to Question 1. The length of #761 illustrates that there is no simple solution ;-)

[kneth]

Can default values (see #777) help?

[snowpong]

@kneth My current solution when using GSON / RetroFit and handling partial/full updates to Realm is:

1. I added a boolean isFullVersion to the Realm Model (Java default is false)
2. I set isFullVersion to true only after doing a copyToRealmOrUpdate(..) from a full version
3. I only do copyToRealmOrUpdate(..) of the full version if: the ID doesn't exist in realm OR the ID exists but is not isFullVersion (a partial update created it) OR the reponse was from the network (not from cache) and is not 304.
4. My calls to copyToRealmOrUpdate(..) for partial versions happen indirectly when parent objects are copied into Realm. Those parents are only copied when: their ID doesn't exist OR the network replies success and not 304.

You can see parts of that implementation here https://github.com/Turistforeningen/SjekkUT/blob/master/android/app/src/main/java/no/dnt/sjekkut/network/StorePlaceCallback.java#L20 and https://github.com/Turistforeningen/SjekkUT/blob/master/android/app/src/main/java/no/dnt/sjekkut/network/StoreProjectCallback.java#L20

It's not perfect. But at least I ensure that full versions will overwrite partial versions and that we don't constantly write to Realm unless it's new data.

Regarding defaults in Realm: Since GSON already "supports" default values (GSON uses the default values set by you unless the JSON overwrites it) I'm not helped too much by it also working in Realm. My Realm Models are always created by GSON and then copied into Realm.

[kneth]

@snowpong Thanks for the details. Please consider to document it in a broader forum (medium.com, etc.).

[tmtrademarked]

I think this should be re-opened - the support for partial updates is still pretty essential for mobile apps. In our case, we have exactly the situation @cmelchior describes, where some of our endpoints return partial data about our objects.

It seems like the right way has to offer some sort of merge. This is too common a use case to not have native support from Realm.

[Zhuinden]

@tmtrademarked there is still no way to reliably tell the difference between when an element is specifically set to null, or when the received JSON model is "partial".

[tmtrademarked]

Sure, that's fair, but there are possible solutions to this. Here's a strawman proposal:

• Implement a function called "merge(value)" or similar.
• If the object has no primary key, merge rejects it - this is only meaningful for keyed objects.
• If the object isn't present, merge inserts it.
• If an object is present, merge will apply the updated fields from any non-null field of value.

This certainly doesn't solve every case - but it handles many, and would still be useful. If you want to be able to clear fields, you can use the existing insertOrUpdate. So merge would only be used for this kind of limited case.

I understand the concerns around expanding the API - but this is a serious pain point when using Realm. It differs from the semantics expected from other database systems (partial updates are super common in SQLite systems, for example). Partial data models are increasingly popular in mobile development, especially in places that use json-api or GraphQL. So I really think Realm needs to support this kind of functionality.

[zaki50]

Since null is not a special value, I'm against treating null as a marker for skipping.

However, partial update is very useful (and almost necessary) and I think we should seek for the good API to achieve that.

[mhd-adeeb-masoud]

I am having the same issue.
I am communicating with a DDP server in my app that only returns the changed fields and the id of the row. I just need a simple way of updating objects by passing column(s) name(s) and column(s) value(s).
If these "partial" values are converted to POJO, there is no way of telling which fields should be updated to null and which fields do not exist in the first place as @zaki50 stated.
Currently my code looks something like this:

        Realm realm=Realm.getDefaultInstance();
        JSONObject obj=new JSONObject(jsonValues);
        if (obj!=null) {
            realm.executeTransaction(realm1 -> {
                try {
                    RealmObject managedUser=realm1.where(User.class).equalTo("id",id).findFirst();
                    if ( managedUser==null)
                        return;
                    if (obj.has("fullName")) {
                        managedUser.setFullName(obj.getString("fullName"));
                    }
                    if (obj.has("firstName")) {
                        managedUser.setFirstName(obj.getString("firstName"));
                    }
                    if (obj.has("lastName")) {
                        managedUser.setLastName(obj.getString("lastName"));
                    }
                    if (obj.has("patientId")) {
                        managedUser.setPatientId(obj.getString("patientId"));
                    }
                } catch (JSONException e) {
                    e.printStackTrace();
                }
            });
            realm.close();
        }

[kneth]

@mhd-adeeb-masoud You could take a look at https://realm.io/docs/java/latest/#the-realm-mobile-platform :-)

[Zhuinden]

@mhd-adeeb-masoud with partial JSONs, that's pretty much the only way to do it. Otherwise you cannot differentiate between null and "non-existend JSON field".

[mhd-adeeb-masoud]

@Zhuinden I think so. But wouldn't it be more convenient if we have something like RealmObject.update(colomns,values) or maybe RealmObject.set(column,value)

Rather than having to write the java setter for each individual property.

[Zhuinden]

@mhd-adeeb-masoud nobody stops you from using reflection to invoke your setter by property name.

[beeender]

@mhd-adeeb-masoud I don't think you need obj.has() checks since Realm is doing that:

https://github.com/realm/realm-java/blob/master/realm/realm-annotations-processor/src/test/resources/io/realm/SimpleRealmProxy.java#L244

Means if your object has a primary key, and the JSON object has the same primary key, when you call createOrUpdateObjectFromJson the field doesn't exist in the json object will be ignored.

[Zhuinden]

@beeender assuming the incoming JSON perfectly matches the json parser code that Realm's schema mediator has.

[beeender]

@Zhuinden Uh ... yes ... non-matching field name is another topic :)

[mhd-adeeb-masoud]

@beeender thank you, that is exactly what I am trying to do...
I can happily convert the values I am getting into a JSON object that has exactly the same field names as the RealmObject since they are pretty much the same mostly. This makes perfect sense.