Google-cloud-go: Struct can not contain a pointer field

[bfdy18]

Client

cloud.google.com/go v0.35.1

Describe Your Environment

OSX, Using Google Datastore simulator, go version go1.11.4 darwin/amd64

Expected Behavior

Using pointer to struct as member, example:

type PLSChildStruct struct {
    IntField   int      `datastore:"intField"`
    BasicSlice []string `datastore:"basicSlice"`
}

type PLSData struct {
    ID         string          `datastore:"id"`
    FlattenPtr *PLSChildStruct `datastore:"flattenPtrField,flatten"`
}

PLSData can be saved into datastore, when load back, encounter error:
datastore: cannot load field "flattenPtrField.basicSlice" into a "PLSData": no such struct field

If define PropertyLoadSaver,

func (r *PLSChildStruct) Load(ps []datastore.Property) error {
    return datastore.LoadStruct(r, ps)
}

func (r *PLSChildStruct) Save() ([]datastore.Property, error) {
    return datastore.SaveStruct(r)
}

PLSData can be loaded back, but PLSChildStruct.BasicSlice only has the last element, for example,
Save into datastore

PLSData{
    ID:         1,
    FlattenPtr: &PLSChildStruct{IntField: 6, BasicSlice: []string{"basic1", "basic2"}},
},

When loaded back, it becomes:

PLSData{
    ID:         1,
    FlattenPtr: &PLSChildStruct{IntField: 6, BasicSlice: []string{"basic2"}},
},

Is this by design that we can not use pointer to struct? If not, how to do it?

Thanks

[vvakame]

@jadekler I think this issue is not type: question.
bacause

PLSData can be saved into datastore, when load back, encounter error

We can put the entity, but can't get it.
I want to get entity or get an error when do put operation.

[poy]

@bfdy18

The flatten tag is invalid when the child struct has a slice.

We are going to add this to the docs and try to have it return an error to better guide.

[bfdy18]

@poy , I think it is more than that:

flatten can not be used with pointer to struct, with or without slice.

type PLSChildStruct struct {
    IntField   int      `datastore:"intField"`
    // BasicSlice []string `datastore:"basicSlice"`
}

type PLSData struct {
    ID         string          `datastore:"id"`
    FlattenPtr *PLSChildStruct `datastore:"flattenPtrField,flatten"`
}

Above example without slice, still fails.

If not using pointer, flatten will work well with child struct with slice,

type PLSChildStruct struct {
    IntField   int      `datastore:"intField"`
    BasicSlice []string `datastore:"basicSlice"`
}

type PLSData struct {
    ID         string          `datastore:"id"`
    Flatten PLSChildStruct `datastore:",flatten"` // not a pointer 
}

Above example works.

[poy]

@bfdy18 I'm having a bit of trouble recreating your error with a pointer:

type PLSChildStruct struct {
    IntField int `datastore:"intField"`
}

func (r *PLSChildStruct) Load(ps []datastore.Property) error {
    return datastore.LoadStruct(r, ps)
}

func (r *PLSChildStruct) Save() ([]datastore.Property, error) {
    return datastore.SaveStruct(r)
}

type PLSData struct {
    ID         string          `datastore:"id"`
    FlattenPtr *PLSChildStruct `datastore:"flattenPtrField,flatten"`
}

func main() {
    ctx := context.Background()

    // Create a datastore client. In a typical application, you would
    // create
    // a single client which is reused for every datastore operation.
    dsClient, err := datastore.NewClient(ctx, os.Getenv("DATASTORE_PROJECT_ID"))
    if err != nil {
        panic(err)
    }

    k := datastore.NameKey("Entity", fmt.Sprintf("id-%d", time.Now().UnixNano()), nil)
    k, err = dsClient.Put(ctx, k, &PLSData{
        ID: "1",
        FlattenPtr: &PLSChildStruct{
            IntField: 2,
        },
    },
    )
    if err != nil {
        panic(err)
    }

    var loaded PLSData
    if err := dsClient.Get(ctx, k, &loaded); err != nil {
        panic(err)
    }

    if err := json.NewEncoder(os.Stdout).Encode(&loaded); err != nil {
        panic(err)
    }
}

$ go run main.go
{"ID":"1","FlattenPtr":{"IntField":2}}

[bfdy18]

@poy , try this code, two of them failed. Define PropertyLoadSaver can help one of the failed case (without slice in child struct), not the other.

import (
    "testing"

    "cloud.google.com/go/datastore"
    "github.com/stretchr/testify/require"
)

func AssertEqual(t *testing.T, src, dst interface{}) {
    props, err := datastore.SaveStruct(src)
    require.NoError(t, err)

    datastore.LoadStruct(dst, props)
    require.NoError(t, err)

    require.Equal(t, src, dst)
}

func TestDatastoreSerialization(t *testing.T) {
    type ChildWithSlice struct {
        IntField   int
        SliceField []string
    }

    t.Run("no pointer, child has slice, passed as expected", func(t *testing.T) {
        type WithFlattenChild struct {
            StrField string
            Child    ChildWithSlice `datastore:",flatten"`
        }

        src := WithFlattenChild{
            StrField: "anything",
            Child: ChildWithSlice{
                IntField:   5,
                SliceField: []string{"s1", "s2"},
            },
        }

        var dst WithFlattenChild
        AssertEqual(t, &src, &dst)
    })

    t.Run("using pointer, child has slice, why failed?", func(t *testing.T) {
        type WithFlattenChild struct {
            StrField string
            ChildPtr *ChildWithSlice `datastore:",flatten"`
        }

        src := WithFlattenChild{
            StrField: "anything",
            ChildPtr: &ChildWithSlice{
                IntField:   5,
                SliceField: []string{"s1", "s2"},
            },
        }

        var dst WithFlattenChild
        AssertEqual(t, &src, &dst)
    })

    t.Run("using pointer, NO slice in child, why failed?", func(t *testing.T) {
        type ChildWithoutSlice struct {
            IntField int
            // SliceField []string
        }

        type WithFlattenChild struct {
            StrField string
            ChildPtr *ChildWithoutSlice `datastore:",flatten"`
        }

        src := WithFlattenChild{
            StrField: "anything",
            ChildPtr: &ChildWithoutSlice{
                IntField: 5,
            },
        }

        var dst WithFlattenChild
        AssertEqual(t, &src, &dst)
    })
}

Thanks

[poy]

@bfdy18 Thanks for the sample code. I'm still investigating, however I want to add one thing in case anyone else stumbles into this. Currently, your example is ignoring the error from datastore.LoadStruct(...). So if you change it to:

err = datastore.LoadStruct(dst, props)

You will get the following error:

datastore: cannot load field "ChildPtr.IntField" into a "ds.WithFlattenChild": no such struct field

I will continue to look into this and see if I can figure out why this error is occurring.

[bfdy18]

@poy thanks.
Also l suggest SaveStruct should return error too if LoadStruct does so.

[poy]

@bfdy18 It looks like it does:

props, err := datastore.SaveStruct(src)

[bfdy18]

@poy
I actually mean, if LoadStruct returns error on a kind of struct, this same struct should fail SaveStruct too. It is confusing that we can save a data into datastore but unable to load them back.

[poy]

@bfdy18 Ah, I see. I agree completely!

[poy]

This CL fixes the pointer issue:

https://code-review.googlesource.com/c/gocloud/+/38550

[poy]

The pointer fix has been merged: 2b3814dbf3ff7af232c7b874e69eab0b0d66f969

[poy]

CL to fix the docs:

https://code-review.googlesource.com/c/gocloud/+/38590