Google-cloud-ruby: Getting "no matching index found" when running example

[r-gochain]

datastore = Google::Cloud::Datastore.new project: project_id => #<Google::Cloud::Datastore::Dataset:0x000055fbec9d62f8 @service=Google::Cloud::Datastore::Service(project_name)> [17] pry(main)> task = datastore.entity "Task", "sampleTask" do |t| [17] pry(main)* t["type"] = "Personal" [17] pry(main)* t["done"] = false [17] pry(main)* t["priority"] = 4 [17] pry(main)* t["description"] = "Learn Cloud Datastore" [17] pry(main)* end => #<Google::Cloud::Datastore::Entity:0x000055fbec7c39c0 @_exclude_indexes={}, @key=#<Google::Cloud::Datastore::Key:0x000055fbec7c3600 @kind="Task", @name="sampleTask", @namespace=nil, @project=nil>, @properties= #<Google::Cloud::Datastore::Properties:0x000055fbec7c3998 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>> [18] pry(main)> [19] pry(main)> # Save the new task [20] pry(main)> datastore.save task => [#<Google::Cloud::Datastore::Entity:0x000055fbec7c39c0 @_exclude_indexes={}, @key=#<Google::Cloud::Datastore::Key:0x000055fbec7c3600 @kind="Task", @name="sampleTask", @namespace=nil, @project=nil>, @properties= #<Google::Cloud::Datastore::Properties:0x000055fbec7c3998 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>>] [21] pry(main)> [22] pry(main)> # Run a query for all completed tasks [23] pry(main)> query = datastore.query("Task"). [23] pry(main)* where("done", "=", false) => #<Google::Cloud::Datastore::Query:0x000055fbec4eb0b8 @grpc= <Google::Datastore::V1::Query: projection: [], kind: [<Google::Datastore::V1::KindExpression: name: "Task">], filter: <Google::Datastore::V1::Filter: composite_filter: <Google::Datastore::V1::CompositeFilter: op: :AND, filters: [<Google::Datastore::V1::Filter: composite_filter: nil, property_filter: <Google::Datastore::V1::PropertyFilter: property: <Google::Datastore::V1::PropertyReference: name: "done">, op: :EQUAL, value: <Google::Datastore::V1::Value: boolean_value: false, integer_value: 0, double_value: 0.0, key_value: nil, entity_value: nil, geo_point_value: nil, array_value: nil, timestamp_value: nil, null_value: :NULL_VALUE, meaning: 0, string_value: "", blob_value: "", exclude_from_indexes: false>>>]>, property_filter: nil>, order: [], distinct_on: [], start_cursor: "", end_cursor: "", offset: 0, limit: nil>> [24] pry(main)> tasks = datastore.run query Google::Cloud::FailedPreconditionError: 9:no matching index found. from /home/user/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/google-cloud-datastore-1.4.0/lib/google/cloud/datastore/service.rb:191:in `rescue in execute'

[blowmage]

This error is coming from the Datastore API. The Indexes guide states:

By default, Cloud Datastore automatically predefines an index for each property of each entity kind. These single property indexes are suitable for simple types of queries.

Your query is using a built-in index so it should be available. How long did you wait between creating the first entity with that property and running the query? It may take a few seconds for the built-in index to become available.

[r-gochain]

It works perfectly from UI

[r-gochain]

So at least 10+ minutes

[r-gochain]

Just recreated a project from the scratch and got the same

Google::Cloud::FailedPreconditionError: 9:no matching index found.

[blowmage]

I suspect there may be something else going on with your setup as I am unable to reproduce the error you are reporting. I just ran the following Ruby code in a new IRB session without any issues

require "google/cloud/datastore"
datastore = Google::Cloud::Datastore.new
task = datastore.entity "NeverUsedBeforeTask", "sampleTask" do |t|
  t["type"] = "Personal"  
  t["done"] = false  
  t["priority"] = 4  
  t["description"] = "Learn Cloud Datastore"  
end  
datastore.save task
query = datastore.query("NeverUsedBeforeTask").where("done", "=", false)  
tasks = datastore.run query

I pasted the ruby code into the console so it ran as fast as possible. Here is the output:

$ irb
irb(main):001:0> require "google/cloud/datastore"
=> true
irb(main):002:0> datastore = Google::Cloud::Datastore.new
=> #<Google::Cloud::Datastore::Dataset:0x00007ff94e8b8db0 @service=Google::Cloud::Datastore::Service(my-project-id)>
irb(main):003:0> task = datastore.entity "NeverUsedBeforeTask", "sampleTask" do |t|
irb(main):004:1*   t["type"] = "Personal"  
irb(main):005:1>   t["done"] = false  
irb(main):006:1>   t["priority"] = 4  
irb(main):007:1>   t["description"] = "Learn Cloud Datastore"  
irb(main):008:1> end  
=> #<Google::Cloud::Datastore::Entity:0x00007ff94e872748 @properties=#<Google::Cloud::Datastore::Properties:0x00007ff94e872720 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>, @key=#<Google::Cloud::Datastore::Key:0x00007ff94e8723d8 @kind="NeverUsedBeforeTask", @name="sampleTask", @project=nil, @namespace=nil>, @_exclude_indexes={}>
irb(main):009:0> datastore.save task
=> [#<Google::Cloud::Datastore::Entity:0x00007ff94e872748 @properties=#<Google::Cloud::Datastore::Properties:0x00007ff94e872720 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>, @key=#<Google::Cloud::Datastore::Key:0x00007ff94e8723d8 @kind="NeverUsedBeforeTask", @name="sampleTask", @project=nil, @namespace=nil>, @_exclude_indexes={}>]
irb(main):010:0> query = datastore.query("NeverUsedBeforeTask").where("done", "=", false)  
=> #<Google::Cloud::Datastore::Query:0x00007ff94c06d950 @grpc=<Google::Datastore::V1::Query: projection: [], kind: [<Google::Datastore::V1::KindExpression: name: "NeverUsedBeforeTask">], filter: <Google::Datastore::V1::Filter: composite_filter: <Google::Datastore::V1::CompositeFilter: op: :AND, filters: [<Google::Datastore::V1::Filter: composite_filter: nil, property_filter: <Google::Datastore::V1::PropertyFilter: property: <Google::Datastore::V1::PropertyReference: name: "done">, op: :EQUAL, value: <Google::Datastore::V1::Value: boolean_value: false, integer_value: 0, double_value: 0.0, key_value: nil, entity_value: nil, geo_point_value: nil, array_value: nil, timestamp_value: nil, null_value: :NULL_VALUE, meaning: 0, string_value: "", blob_value: "", exclude_from_indexes: false>>>]>, property_filter: nil>, order: [], distinct_on: [], start_cursor: "", end_cursor: "", offset: 0, limit: nil>>
irb(main):011:0> tasks = datastore.run query
=> [#<Google::Cloud::Datastore::Entity:0x00007ff94e848830 @properties=#<Google::Cloud::Datastore::Properties:0x00007ff94e8482e0 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>, @key=#<Google::Cloud::Datastore::Key:0x00007ff94e848600 @kind="NeverUsedBeforeTask", @name="sampleTask", @project="my-project-id", @namespace="">, @_exclude_indexes={"type"=>false, "done"=>false, "priority"=>false, "description"=>false}>]

I switched projects and got the same results. Is there anything else you can tell us about your setup?

[r-gochain]

That's weird, I cannot imagine what could be wrong with the setup.

[quartzmo]

I'll try to reproduce as well.

[blowmage]

I created a new project named temporary-datastore-test, created service account credentials and downloaded the JSON file, and then visited the following URL to set up the database.

https://console.cloud.google.com/datastore/setup?project=temporary-datastore-test

I did not create an entity at that URL, I only created the database. Then I ran the previous Ruby code as soon as the web UI refreshed after the database was created. Here is the output:

$ DATASTORE_PROJECT=temporary-datastore-test DATASTORE_CREDENTIALS=/downloads/temporary-datastore-test.json irb
irb(main):001:0> require "google/cloud/datastore"
=> true
irb(main):002:0> datastore = Google::Cloud::Datastore.new
=> #<Google::Cloud::Datastore::Dataset:0x00007fe77013b910 @service=Google::Cloud::Datastore::Service(temporary-datastore-test)>
irb(main):003:0> task = datastore.entity "NeverUsedBeforeTask", "sampleTask" do |t|
irb(main):004:1*   t["type"] = "Personal"  
irb(main):005:1>   t["done"] = false  
irb(main):006:1>   t["priority"] = 4  
irb(main):007:1>   t["description"] = "Learn Cloud Datastore"  
irb(main):008:1> end  
=> #<Google::Cloud::Datastore::Entity:0x00007fe770118c30 @properties=#<Google::Cloud::Datastore::Properties:0x00007fe770118c08 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>, @key=#<Google::Cloud::Datastore::Key:0x00007fe770118708 @kind="NeverUsedBeforeTask", @name="sampleTask", @project=nil, @namespace=nil>, @_exclude_indexes={}>
irb(main):009:0> datastore.save task
=> [#<Google::Cloud::Datastore::Entity:0x00007fe770118c30 @properties=#<Google::Cloud::Datastore::Properties:0x00007fe770118c08 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>, @key=#<Google::Cloud::Datastore::Key:0x00007fe770118708 @kind="NeverUsedBeforeTask", @name="sampleTask", @project=nil, @namespace=nil>, @_exclude_indexes={}>]
irb(main):010:0> query = datastore.query("NeverUsedBeforeTask").where("done", "=", false)  
=> #<Google::Cloud::Datastore::Query:0x00007fe7701117a0 @grpc=<Google::Datastore::V1::Query: projection: [], kind: [<Google::Datastore::V1::KindExpression: name: "NeverUsedBeforeTask">], filter: <Google::Datastore::V1::Filter: composite_filter: <Google::Datastore::V1::CompositeFilter: op: :AND, filters: [<Google::Datastore::V1::Filter: composite_filter: nil, property_filter: <Google::Datastore::V1::PropertyFilter: property: <Google::Datastore::V1::PropertyReference: name: "done">, op: :EQUAL, value: <Google::Datastore::V1::Value: boolean_value: false, integer_value: 0, double_value: 0.0, key_value: nil, entity_value: nil, geo_point_value: nil, array_value: nil, timestamp_value: nil, null_value: :NULL_VALUE, meaning: 0, string_value: "", blob_value: "", exclude_from_indexes: false>>>]>, property_filter: nil>, order: [], distinct_on: [], start_cursor: "", end_cursor: "", offset: 0, limit: nil>>
irb(main):011:0> tasks = datastore.run query
=> [#<Google::Cloud::Datastore::Entity:0x00007fe770108dd0 @properties=#<Google::Cloud::Datastore::Properties:0x00007fe770108470 @hash={"type"=>"Personal", "done"=>false, "priority"=>4, "description"=>"Learn Cloud Datastore"}>, @key=#<Google::Cloud::Datastore::Key:0x00007fe770108b78 @kind="NeverUsedBeforeTask", @name="sampleTask", @project="temporary-datastore-test", @namespace="">, @_exclude_indexes={"type"=>false, "done"=>false, "priority"=>false, "description"=>false}>]

[r-gochain]

Seems like we found the root of the issue.

[quartzmo]

I could not reproduce in my own project, using @blowmage's snippet in this comment. I get the same results as @blowmage posted there.

[treeder]

Ok, we figured out how to work around this. We were creating the databases in firebase which leads you to cloud console for service accounts and says that you can use datastore libraries, and lastly it says that it also automatically creates indexes:

So we tried creating the database in a new project in the cloud console instead and low and behold it worked. So it looks like there's a disconnect between creating it in the firebase console vs the cloud console.

[quartzmo]

@treeder So would you consider this a problem with the documentation? Can you provide additional links/screenshots and a recommendation for changes? We do not have the ability to change console or cloud.google.com docs ourselves, but we may be able to get the attention of someone who can.

[treeder]

Well, I would just expect it to work. I assume these indexes should be there when created in Firebase since it says that they should be per the screenshot I posted above. I guess I'm also assuming that these libraries should work with Firestore too since from what I've read, Firestore is backed by Datastore?

Maybe those are wrong assumptions though... ? 😄

[quartzmo]

Fair enough, let's leave open the possibility that it could be a bug in the service. We will do what we can to raise this issue to Google, but please also pursue any other support channels you may have.

Both scenarios are out of scope for this library, so I'm closing this issue now. Feel free to reopen this issue or new ones if you discover anything we can do to improve this library and/or google-cloud-firestore, which we hope you might also find useful.

[blowmage]

I do not know much about the Firebase console, or how it connects to Datastore. I don't believe that the newer Firestore service is Backed by Datastore however. Firestore is implemented on top of Spanner, and Datastore pre-dates Spanner. Also, this repo maintains two separate projects for acceptance tests because Datastore and Firestore cannot be enabled on the same project. We have been told a project much choose one or the other.

[blowmage]

@pcostell Do you have any insight into this behavior? It looks like databases created by Firebase don't have the built-in indexes enabled. Is this behavior known?

[pcostell]

Databases created through Firebase create Cloud Firestore projects. Cloud Firestore and Cloud Datastore have some interoperability, however they have different query semantics (and thus different indexes). Like Cloud Datastore, Cloud Firestore has automatic indexes, but they are a different set of indexes, which is why your Cloud Datastore queries (with Cloud Datastore semantics) are missing an index.

If you want to use Cloud Firestore, you should use it with the Cloud Firestore client libraries. Otherwise, if you want to use Cloud Datastore client libraries, you should make sure to create a Cloud Datastore project at cloud.google.com/console.

[quartzmo]

@pcostell From the above comment:

We were creating the databases in firebase which leads you to cloud console for service accounts and says that you can use datastore libraries

Is there perhaps a documentation problem in this flow?

[treeder]

We found this lib and are giving it a try: https://github.com/firebase/firebase-admin-go You can probably chock this up to bad assumptions on our part. And the fact that this library partially works with Firestore (inserts, queries without filters, etc) which added to the confusion.

[quartzmo]

@treeder We let some Google folks know about this, thanks so much for opening this issue!

[frankyn]

Hi @treeder,

Thanks for filing the issue could you help me understand how you came across information stating you could Datastore client libraries to access Firestore?

After chatting with @pcostell, there does exist some interoperability but not enough to recommend that you use the Datastore client. Please continue to use Ruby Firestore client libraries.

Thanks!