Google-cloud-go: firestore: Add function to convert firestore protobuf document to a struct

[bashtian]

Cloud Functions supports Firestore triggers and when a document is updated an event is send to the function with this format:

pb "google.golang.org/genproto/googleapis/firestore/v1"

// FirestoreEvent is the payload of a Firestore event.
type FirestoreEvent struct {
    OldValue   pb.Document `json:"oldValue"`
    Value      pb.Document `json:"value"`
    UpdateMask struct {
        FieldPaths []string `json:"fieldPaths"`
    } `json:"updateMask"`
}

There is currently no way to convert pb.Document to a normal Go struct. setFromProtoValue could be exported to make it possible or a new helper function based on this DataTo line.

[bashtian]

The data is send as JSON but unmarshalling it into a document does not work because OneOf fields are not supported. So this feature request not needed until we can unmarshal the data into a pb.Document.

Edit: the jsonpb package seems to work.

[jadekler]

cc @tbpg

[jadekler]

cc @benwhitehead

[tbpg]

Seems like a good idea to me.

[odeke-em]

Thank you for filing this issue @bashtian!

I don't understand what the feature request is about that isn't already supported. https://godoc.org/google.golang.org/genproto/googleapis/firestore/v1#Document is already a "normal" Go struct and moreover it can be JSON marshaled and then unmarshaled to a map or even another struct for which the JSON tags and fields could be attached.

Could you please show a use case/code sample for where converting to a "normal Go struct" from that pb.Document arises?

[bashtian]

Currently it's not possible to create a DocumentSnapshot from a firestore.Document. I made this change in my repo to create a new DocumentSnapshot:

https://github.com/bashtian/google-cloud-go/commit/d3311dabe766384e8cc050aec644950907b94360

With this change it's now possible to convert the protobuf data to a model struct in Cloud Function.

import (
    "github.com/golang/protobuf/jsonpb"
    firstorepb "google.golang.org/genproto/googleapis/firestore/v1"
)

type MyModel struct {
    Name string `firestore:"name"`
}

var client *firestore.Client

func (m *MyModel) UnmarshalJSON(b []byte) error {
    var doc firstorepb.Document
    u := jsonpb.Unmarshaler{}
    u.Unmarshal(bytes.NewReader(b), &doc)

    ds, err := client.NewDocumentSnapshot(&doc)
    if err != nil {
        return err
    }
    return ds.DataTo(m)
}

[tbpg]

Ahh. So, here is the flow:

1. Store some data in Firestore, going in as type MyModel.
2. A Firestore Cloud Function is triggered and the data comes out as firestorepb.Document, where all of the fields from MyModel are untyped in the Fields field.
3. You want to convert the firestorepb.Document to MyModel, just like how you can convert a DocumentSnapshot to MyModel with DataTo.

I think this is a great idea that is totally worth adding. I'm not totally sure about the API, though. The custom UnmarshalJSON method you gave is slick. It would be nice to not require users to write out that code every time, but I don't think we can somehow magically skip the firestorepb.Document.

So, it's a question how how best to go from firestorepb.Document -> MyModel. Your change to create the NewDocumentSnapshot seems like a pretty targeted fix. It's definitely smaller than adding DataTo, DataAt, etc. for firestorepb.Documents.

@odeke-em, @jadekler, or @BenWhitehead, any other ideas on a good API for this? Or do you think we should move forward with NewDocumentSnapshot?

---

Edit: of course, users could do the conversion directly in the Cloud Function. It's more direct and easier to explain, but makes the Cloud Function a little more clunky.

func HelloFirestore(ctx context.Context, e FirestoreEvent) error {
    var m MyModel
    ds, err := client.NewDocumentSnapshot(e.Value)
    if err != nil {
        return err
    }
    if err := ds.DataTo(&m); err != nil {
        return err
    }
    // Use m.
}

vs.

// HelloFirestore with the custom UnmarshalJSON.
func HelloFirestore(ctx context.Context, m MyModel) error {
    // Use m.
}

[bashtian]

The Firestore Protobuf document does not support UnmarshalJSON (https://github.com/golang/protobuf/issues/256), so we need to unmarshal it first with jsonpb. Using NewDocumentSnapshot without a custom UnmarshalJSON method would look like this:

// FirestoreEvent is the payload of a Firestore event.
type FirestoreEvent struct {
    OldValue   json.RawMessage `json:"oldValue"`
    Value      json.RawMessage `json:"value"`
    UpdateMask struct {
        FieldPaths []string `json:"fieldPaths"`
    } `json:"updateMask"`
}

func HelloFirestore(ctx context.Context, e FirestoreEvent) error {
    var doc firstorepb.Document
    u := jsonpb.Unmarshaler{}
    u.Unmarshal(bytes.NewReader(e.Value), &doc)

    var m MyModel
    ds, err := client.NewDocumentSnapshot(doc)
    if err != nil {
        return err
    }
    if err := ds.DataTo(&m); err != nil {
        return err
    }
}

If adding the dependency to jsonpb is not a problem, NewDocumentSnapshot could accept []byte or json.RawMessage to remove the Protobuf handling from the user code.

[BenWhitehead]

Based on this sample: Firestore Function String Uppercase I think it may be possible to get the decoding for "free" as long as you're okay with the intermediary struct FirestoreValue. Which then gives you struct traversal such as e.Value.Fields.Original.StringValue. One advantage here is you get full structs for the whole event.

[bashtian]

@BenWhitehead This example was my motivation to open this issue.
The example just maps the Protobuf JSON output of a firestore.Document to a custom struct. It's not a really usable solution for large structs with nested data, especially if you want to write the data again with the firestore client package. You will need a second struct for your model and keep these in sync.

[ghost]

Hey is there any update on this issue? We sometimes have nested data where firestore events look like this:

    "fields": {
      "myArray": {
        "arrayValue": {
          "values": [
            {
              "mapValue": {
                "fields": {
                  "a": {
                    "mapValue": {
                      "fields": {
                        "lastUpdatedAuthor": {
                          "stringValue": "John Doe"
                        },
                      }
                    }
                  },
                  "b": {
                    "mapValue": {
                      "fields": {
                        "lastUpdatedAuthor": {
                          "stringValue": "Jack"
                        },
                      }
                    }
                  }
                }
              }
            }, { ... }
          ]
        }
      }, { ... }
    }

Mapping these kind of events into our original Go struct becomes quite cumbersome :(.

[Dasio]

Any update to this? At least if https://github.com/bashtian/google-cloud-go/commit/d3311dabe766384e8cc050aec644950907b94360 was in upstream.

[maclir]

It would be nice to get an update on this :)
Quite a hassle to write data models for firestore triggers :/

[adrianduke]

This was painful to deal with on our project as we have large events coming in on the firestore trigger, so creating the protojson structs was a nightmare, instead I forked the firestore and and go sdk (as there are cross dependency of firestore on the google go cloud sdk) and added @bashtian's fix along with 1 additional modification so I could easily generate the incoming event protojson for testing (note the library has changed from jsonpb to protojson):

https://github.com/adrianduke/firestore/commit/cb0c15add69c5df169a623fe8b9ed70ad7018ea0

-func toProtoDocument(x interface{}) (*pb.Document, []*pb.DocumentTransform_FieldTransform, error) {
+func ToProtoDocument(x interface{}) (*pb.Document, []*pb.DocumentTransform_FieldTransform, error) {

This allowed me to create some helper functions:

import (
    "google.golang.org/protobuf/encoding/protojson"
    "google.golang.org/protobuf/types/known/timestamppb"
)

func firestoreCreateEventFromStepResult(s *pipeline.StepResult) (FirestoreEvent, error) {
    sFsValuePJBytes, err := firestoreValueToProtoJsonBytes(firestoreValueFromStepResult(s))
    if err != nil {
        return FirestoreEvent{}, err
    }

    return FirestoreEvent{
        Value:    sFsValuePJBytes,
        OldValue: json.RawMessage(`{}`),
    }, nil
}

func firestoreValueFromStepResult(s *pipeline.StepResult) FirestoreValue {
    v := FirestoreValue{
        CreateTime: time.Now(),
        Name: fmt.Sprintf(
            "projects/some-firestore-project-name/databases/(default)/documents/some-document/%s",
            s.ID,
        ),
        UpdateTime: time.Time{},
        Fields:     s,
    }

    return v
}

func firestoreValueToProtoJsonBytes(v FirestoreValue) ([]byte, error) {
    vProtoDoc, _, err := firestore.ToProtoDocument(v.Fields)
    if err != nil {
        return nil, err
    }
    vProtoDoc.Name = v.Name
    vProtoDoc.CreateTime = timestamppb.New(v.CreateTime)
    vProtoDoc.UpdateTime = timestamppb.New(v.UpdateTime)

    vBytes, err := protojson.Marshal(vProtoDoc)
    if err != nil {
        return nil, err
    }

    return vBytes, nil
}

type FirestoreValue struct {
    CreateTime time.Time            `json:"createTime"`
    Fields     *pipeline.StepResult `json:"fields"`
    Name       string               `json:"name"`
    UpdateTime time.Time            `json:"updateTime"`
}

Note I used the json.RawMessage version mentioned above for my incoming FirestoreEvent:

type FirestoreEvent struct {
    OldValue   json.RawMessage `json:"oldValue"`
    Value      json.RawMessage `json:"value"`
    UpdateMask struct {
        FieldPaths []string `json:"fieldPaths"`
    } `json:"updateMask"`
}

I couldn't get https://github.com/googleapis/google-cloud-go/issues/1438#issuecomment-530942781 working, but likely I did something wrong. It would be much nicer to have FirestoreEvent's that embed your native types or provider a wrapper around them so you don't need to worry about creating the incoming event message protojson for values.

You should be able to use my forked versions easily with the following go.mod replace directives:

replace cloud.google.com/go/firestore => github.com/adrianduke/firestore v1.2.8
replace cloud.google.com/go => github.com/adrianduke/google-cloud-go v0.59.3

[red010b37]

is there a reason we cant just a public function like this to the client?

// GenerateNewDocumentSnapshot generates a new document from a protobuf Document
func (c *Client) GenerateNewDocumentSnapshot(proto *pb.Document) (*DocumentSnapshot, error) {
    docRef, err := pathToDoc(proto.Name, c)
    if err != nil {
        return nil, err
    }
    doc, err := newDocumentSnapshot(docRef, proto, c, proto.UpdateTime)
    if err != nil {
        return nil, err
    }
    return doc, nil
}

[brensch]

this is a huge barrier to using firestore triggers in go. Is there any update on this?

[picaoao]

+1 it would be great to have a solution for the issue !

[corlinp]

Just ran into this. Makes working with firestore triggers in Go very frustrating. +1