Google-cloud-go: datastore: retrieving an entity's key as a struct field

[luisfernrg]

I want to be able to query a my datastore and retrieve all of an entity's properties including it's key.

What I want to be able to have a struct like this

type User struct {
    Key          *datastore.Key `datastore:"__key__"`
    Username     string         `datastore:"username"`
    PasswordHash string         `datastore:"password_hash"`
    CreatedAt    time.Time      `datastore:"created_at"`
}

and be able to use a query like this

q := datastore.NewQuery("User").Filter("username =", "gopher").Limit(1)

and be able to get what ever user that is return and store it in a User struct. Like how it's done when using datastore.Get but instead of getting the entity with they key i want to be able to do so with other properties such as username or even email if I had a property like that.

at the moment I'm using the datastore.GetAll function and ranging over they keys that is return from that function and with []User and storing the key with it's user
example:

        var user []User
    q := datastore.NewQuery("User").Filter("username =", "gopher").Limit(1)
    keys, err := datastore.GetAll(ctx, q, &user)
    if err != nil {
        // Handle error
    }
    for k,v := range keys {
        user[k].Key = v
    }

is there a better way of doing this or is this the correct way?

[derekperkins]

You can use a PropertyLoadSaver

[adams-sarah]

Your problem is not in how you are querying the data, but in how you are storing it.

The datastore:"__key__" struct field tag should only be used for embedded structs, to mark the field as a Key for an Entity Value. When it is found at the top level, the entire field is ignored.

Eg. You should use the tag for something like:

type User struct {
    Name string
    Pets []struct{
        Breed string
        Key *datastore.Key `datastore:"__key__"`
    }
}

In your case, your struct should just look like:

type User struct {
    Key          *datastore.Key
    Username     string         `datastore:"username"`
    PasswordHash string         `datastore:"password_hash"`
    CreatedAt    time.Time      `datastore:"created_at"`
}

Where the Key field is treated like a normal property on the entity, an equal to Username, etc.

Let me know if you have further questions!

[luisfernrg]

@adams-sarah
I'm using this struct now

type User struct {
    Key          *datastore.Key
    Username     string         `datastore:"username"`
    PasswordHash string         `datastore:"password_hash"`
    CreatedAt    time.Time      `datastore:"created_at"`
}

I'm querying like this but when I fill my struct with the fields they all get filled with the correct data except Key. Should this be working or is it not done this way?

func (app *Application) GetUser(w http.ResponseWriter, r *http.Request) {
    username := r.FormValue("username")

    var users []User
    q := datastore.NewQuery("User").Filter("username =", username).Limit(1)
    _, err := app.dsc.GetAll(app.ctx, q, &users)
    if err != nil {
        log.Println(err)
        return
    }
    log.Println(users)
}

[adams-sarah]

Hm ok, so again the problem is in how you are storing the data, and not in how you are querying it.

So your initial Put would need to look something like:

user := User{/* blah */}
// create a complete key
key := datastore.IDKey("User", 12345, nil)
// add key to the user struct value
user.Key = key
// Put User
_, err = client.Put(ctx, key, &user)
if err != nil {
    // do something
}

// and now you can do your query

Remember that the Key field on User is exactly like all the other fields. Nothing special. You have to populate the field yourself before you save the User.

I suspect that the reason you are not getting Keys back in your query is because they are empty in the entity in datastore.

I'm sorry this is not more intuitive. I'm thinking about ways we can make the datastore:"__key__" field tag work more magically for the top level struct (as you initially thought it would).

Happy to answer any more questions you might have.

[luisfernrg]

@adams-sarah
Oh my bad I completely miss read that part where you said Your problem is not in how you are querying the data, but in how you are storing it. in your first response I understand now thank you. Is there any to do this but with a incomplete key so that the Cloud Datastore will automatically generate a numeric ID for the entity's key and that I still will be able to query out?

[adams-sarah]

The only thing you can do is put the returned Key from Put into your struct and Put it again, eg:

key := datastore.IncompleteKey("User", nil)
key, err = client.Put(ctx, key, &user)
if err != nil {
    // do something
}
user.Key = key
_, err = client.Put(ctx, key, &user)
if err != nil {
    // do something
}

but this is really awful.

On the other hand, maybe it's not too terribly difficult for you to come up with your own unique ids? Could you use the 'username' as a string key? Are your usernames unique? Eg:

key := datastore.NameKey("User", user.Username)

Else, there are some pretty simple algorithms for coming up with a unique numeric ID. Just let me know if you'd like some help with this :)

[luisfernrg]

@adams-sarah
I would prefer using unique numeric IDs and I would most definitely appreciate some help on how I should go about this

Btw I also really appreciate taking your time to help! :D

[luisfernrg]

@adams-sarah
Could some thing like a UUID v4 work well maybe with the dashes removed as well.
https://github.com/google/uuid
Or are there other solid ways of generating unique ID's which are numeric?

[adams-sarah]

Sorry for my delay. Yea a UUID would work great.

[luisfernrg]

@adams-sarah
no worries thank you for the help I appreciate it! I was looking at https://github.com/rs/xid or even https://github.com/oklog/ulid but in the end I decided to go with UUID's :D

[suau]

@adams-sarah
I still don't quite get it.
I'm seeing the opposite:

• top level __key__ fields will get populated properly using GetAll or Iterator
• nested structs (Entity Values) will not automatically get a valid key assigned, even not when a Incomplete keys has been put in there
• manually assigning a NameKey does work, but has no effect

so the manual NameKey will get stored, but what's the usecase here ? I tried to query for Inner Entities using the supplied Inner-Entity-Kind, but no results. What advantages do I get from using datastore:"__key__" ?

[adams-sarah]

@suau See the godoc here: https://godoc.org/cloud.google.com/go/datastore#hdr-Key_Field
and the section below, "Structured properties"

I'm not sure what you mean by the second two bullet points.

There are few advantages to using datastore:"__key__" on a nested struct field. You cannot really query using them, except as any other field (key would be "__key__" in the query). The main advantage is struct re-usability, and package interoperability with datastore clients written in other languages.
Again, on a nested struct, a *Key field tagged with datastore:"__key__" is just treated as another field with the name, "__key__".

[suau]

@adams-sarah

a *Key field tagged with datastore:"__key__" is just treated as another field with the name, "__key__"

that makes a lot more sense now. Reading the docs:

If the Inner struct contains a *Key field with the name "__key__", like so:

type Inner struct {
  W int32
  X string
  K *datastore.Key `datastore:"__key__"`
}


type Outer struct {
  I Inner
}

then the value of K will be used as the Key for Inner, represented as an Entity value in datastore.

I was under impression the tagged field would get some special treatment when used in a inner struct.
(With my two other bullet points, I assumed the embedded entities would automatically get an IDKey assigned like top-level entities do)

The struct re-usability comes in really handy when storing entities and denormalized groups of those entities, I totally overlooked that, although I have lots of those almost-identical-structs.
Thanks for pointing that out !

So it's just a package level trick to ignore __key__ on top level entities, but not on embedded entities. Thanks for clearing that up.

[adams-sarah]

Yea. Sorry this is so confusing. We went through a lot of iterations to try to make it better, but still it's not super intuitive.

That is all correct, except the last bit you said.
The __key__ struct field tag on a top-level struct is ignored on PUT requests (any writes). But on GET (any reads), the key of the entitiy is magically loaded into the struct for you. This is helpful especially if you query for many entities. Then you can tell which one is which.

If you can think of ways to make this more clear in the docs, we'd love a contribution!!

[murraycu]

The __key__ struct field tag on a top-level struct is ignored on PUT requests

Is this still true? I'm seeing this error when I try this with datastore.Put():

API error 1 (datastore_v3: BAD_REQUEST): cannot store entity with reserved property name '__key__'

[adams-sarah]

@murraycu yep, should be. we've got a few tests which exercise this behavior. one here (actually hits the datastore server too): https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/datastore/integration_test.go#L103-L111

Can you paste your code?
Perhaps you are tagging it datastore:"key" and not (what it should be) datastore:"__key__" ?

[murraycu]

Thanks for checking. Here is my code that uses that datastore tag: https://github.com/murraycu/go-bigoquiz-server/blob/stats_key/src/user/stats.go#L10 and here is where I use datastore.Put(): https://github.com/murraycu/go-bigoquiz-server/blob/stats_key/src/db/db.go#L210

And this is the error message that I then see when calling Put() with one of those entities:

API error 1 (datastore_v3: BAD_REQUEST): cannot store entity with reserved property name '__key__'

[adams-sarah]

Ah, so you are importing the appengine datastore library (google.golang.org/appengine/datastore).
This issue tracker is for the cloud datastore libraray (cloud.google.com/go/datastore).

The two packages have diverged since the original fork :(. Sorry for the confusion.

However, you should be able to use cloud.google.com/go/datastore on appengine, and the migration should be fairly painless. Maybe that's a good option for you?

cc @zombiezen (as an fyi) re: documentation / the 'which library do i use' issue

[adams-sarah]

As an FYI, the appengine datastore issue tracker lives at github.com/golang/appengine

[murraycu]

Oh, thanks, and sorry. That is very confusing. Google's AppEngine document talks about the "Google Cloud Datastore API", with example code that imports google.golang.org/appengine/datastore:
https://cloud.google.com/appengine/docs/standard/go/datastore/creating-entities
None of the obvious documentation for each API seems to mention that there is another API that is similar but different.

(Sorry for polluting this issue. I'll gladly remove this comment and put it wherever else it really belongs.)

[adams-sarah]

No no, please keep it.
Thanks for reporting the issue. Yea I agree about documentation being confusing re: all the go cloud client libs. This is actually a common complaint we get.
Currently trying to collect comments like yours so as to convince leadership that this is a big issue that needs attention.
So yea, please don't delete!

[kasireddy002]

Hi everyone,

i am writing a service for search functionality .
when i pass the USERNAME in the body to get the data based on that USERNAME .
but i am unable to get the values.

func searchValue (res http.ResponseWriter, req *http.Request){
log.Printf("%#v Getting values url - x ", "called search")
patient := &PatientData{}
if err := json.NewDecoder(req.Body).Decode(patient); err != nil {
panic(err)
}
ctx := appengine.NewContext(req)
ctx, err := appengine.Namespace(ctx, NAMESPACENAME)
if err != nil {
panic(err)
}
m := patient.Phone
i, err := strconv.ParseInt(m, 10, 64)
log.Printf("%#v Getting values m ", m)
log.Printf("%#v Getting values url -yyy ",i)

key := datastore.NewKey(ctx, "Patient","" , i, nil)
log.Printf("%#v Getting values url - key ", key)
   err = datastore.Get(ctx, key, patient)
    if err := json.NewEncoder(res).Encode(patient); err != nil {
        panic(err)
    }       
}

In these code i am passing PHONE to get the full struct values.
i don't want to use the newkey to generate the keyname and to get values.
please i need a help .

[zombiezen]

Hey @kasireddy002, your question is a different one from this issue. I see you opened #831; let's continue the conversation there.

[ffeu]

Hi @adams-sarah , just making a comment:

However, you should be able to use cloud.google.com/go/datastore on appengine, and the migration should be fairly painless. Maybe that's a good option for you?

To me, the migration was simple and it worked in GAE standard. I was even able to do what @luisfernrg tried to do at the first place in this issue - it worked!

However, after I done that, I was not able anymore to run it locally (with dev_appserver.py).
Maybe it works locally with GAE flex compatible code, but apparently it does NOT with GAE standard compatible code (which means using appengine.NewContext instead of context.Background).

So, back to .../appengine/datastore ...