Google-cloud-go: datastore: When loading kinds into structs, ignore items where no struct field exist instead of generating an error

[lc-chrisbarton]

We process orders and we sometimes deprecate or are no longer interested in certain fields in an order. This means that during transaction processing we sometimes have entities in datastore which have properties which no longer have corresponding fields in the current version of our order struct. If we rely on the default loading this gives the following error "datastore: cannot load field "some_property_name" into a "some_struct": no such struct field". This behaviour seems odd to us, consider that normal JSON decoding in Go would ignore any fields in the incoming JSON if there was no destination field. The current behaviour means we must write custom loaders which is fine but more work. In any case we need to support deprecating fields in order to keep our API intuitive because that helps our clients and developers.

[jba]

The JSON convention seems justified to me because JSON is primarily an interchange format, not a storage format. If Datastore dropped unknown fields, then round-tripping would lose data. That seems too high a risk for what is essentially a convenience feature.

It would also break behavior for a stable package.

I suppose we could do it as an option.

/cc @zombiezen

@jskeet @garrettjonesgoogle How do the other typed languages handle deserializing unknown Datastore fields into objects?

[jskeet]

In C# we don't have any code to deserialize the result into objects anyway, so there's equivalent, as far as I can see.

[lc-chrisbarton]

Yes I guess you can't do it because updates are not fined grained and instead re-write the entire record. BTW before we started using a custom loader, in order to remove a field in datastore (thanks to v0.19.0) we would set the struct field to be a pointer, use omitempty and noindex and then re-write all old entities using a go script. That in time allows us to remove the field completely from the struct.

[zombiezen]

When this scenario occurs, the datastore package is documented to return an ErrFieldMismatch error, which you can then suppress.

[lc-chrisbarton]

Does the struct still get filled as much as it can when this error occurs?

[Thooms]

Seeing https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/datastore/load.go#L436-L438 I'd say so.

[bhardwajankur3]

I am also facing this issue. What is the ETA for the fix? Is there any temporary workaround for this?

[jba]

No priority on the issue means no ETA.

We're not even clear on what the right design is.

The workaround is to load the data into maps, and populate your struct from the maps. I know that is tedious, but as I mention above, the alternative opens the door to data loss.

[lc-chrisbarton]

One option is Firestore which supports fine grained updates. Also Datastore is getting the Firestore backend soon "Cloud Firestore in Datastore mode" and when it does updates can be fine grained if the api changes to support that.

https://cloud.google.com/datastore/docs/firestore-or-datastore

[zaddok]

I am facing this exact scenario as well. I'd love to see the datastore ignore field not in the struct.

If a field is not in the struct, its because I don't care about that field any more, but I don't want to just delete it.

[cnbuff410]

Yep, currently the error makes the data model update very inconvenient.

[jskeet]

For C# in Firestore, when deserializing to an attributed type, we added an UnknownPropertyHandling enum. with options of:

• Ignore
• Warn (via a warning delegate on FirestoreDb - not ideal, but we don't have a better standard warning mechanism)
• Throw an exception

The default is to warn.

Our Datastore code doesn't have any sort of "deserialize into user-specified objects" behavior, so the issue doesn't come up.

[uhayate]

I have solved this problem by customizing datastore.Keyloader.
You can see example: https://github.com/uhayate/datastore-keyloader-demo

[Blquinn]

Related issue is that if there is an additional field in the query response, which isn't in your destination struct, (select *) you will get a mapping error as well. Combined with the fact that the gql queries fail when we specify the columns, we're forced to check for the error an ignore it. This is also inconsistent with how other mapping libraries in std work. This particular issue happened to bite us in production because we didn't see the error until we hit an old record with an additional field in it.

[dogukankotan]

I have solved this problem by customizing datastore.Keyloader.
You can see example: https://github.com/uhayate/datastore-keyloader-demo

Thank you uhayate!

This is not a feature request. This is needed! We can always change Datastore Column types!

[saifabid]

I've found this to be a less than ideal workaround, but works since the remaining data still gets filled in the struct -

err := datastore.get()
if err != nil && !strings.Contains(err.Error(), "no such struct field") {
   return err
}