Google-cloud-php: Add ability to exclude list values from indexes

[GuillaumeMiralles]

Hi,

I have already asked this in the issue #3619 , but as I need an answer that I cannot find elsewhere, I open a new issue. Sorry in advance for asking this here again.

How is that supposed to be done to prevent this error Exclude from indexes cannot be set on a list value? Am I supposed to do the same thing than embedded entities? Something like

$ds = new DatastoreClient;

$data = [1, 2, 3];
$array = $ds->entity(null, $data, [
    'excludeFromIndexes' => array_keys($data),
]);

I have tried this, and it doesn't work, so I'm wondering if there is any solution to exclude an entire list from indexes.

Thanks in advance

[jdpedrie]

That message comes from the API, not from the client. I believe that Datastore doesn't support excluding specific items in a list from indexes. I haven't been able to find that in the documentation though, so I'll raise it internally and see if I can get the definitive answer from a Googler who knows more about Datastore than I do.

[GuillaumeMiralles]

Thanks for your time @jdpedrie .
My question is not only to exclude from index specific items, but also how to exclude the entire list itself from index.

Let's say I have an Entity User with an attribute tokens which is a string array, I cannot exclude from index the attribute tokens, and I get the error given above.

Since the error says that it is not possible to exclude from index a list value, then I was guessing that maybe the way to do is to exclude all the specific items in it, but maybe my supposition is wrong

[bshaffer]

@googleapis/storage-dpe

[BenWhitehead]

/cc @googleapis/firestore-dpe

[BenWhitehead]

Hi @GuillaumeMiralles,

I apologies if I'm mis-understanding anything my PHP isn't very good, and I'm coming off a fever.

When excluding things from indexing the thing that needs to be excluded is the "value", unfortunately in our apis what is and isn't a value is often made opaque. So in this case, each value of the array of values needs to be marked excluded.

Here is a StackOverflow post that asks a similar question (albeit for NodeJS instead of PHP) but it illustrates the fact that each value needs the exclude marker on it.

In the GCP console, when looking at an entity in the Datastore Entity viewer you can also click on an array property to inspect whether the values are excluded or not.

Hope this helps, if anyone else with PHP knowledge knows how this can be performed I welcome their insight as I do not know php well enough to be able to figure it out.

[GuillaumeMiralles]

Hi @BenWhitehead !

This is indeed what I intend to be done. Since I cannot exclude an entire list from index, I was looking to exclude each values in the list from index instead. But the thing is, there is no clear way of doing this in PHP.

I managed to do it for embedded entities thanks to @jdpedrie help, but for values inside a list, I didn't find how this can be done.

So let's take my previous example, an Entity User with a property tokens that is a list with no keys, such as your example ["a", "b"], how to remove those specific items from index in PHP?

I have seen in the nodeJS documentation something similar to this:

$ds = new DatastoreClient;

$entity = $ds->entity('User', [
    'firstName' => 'Bob',
    'dateOfBirth' => new DateTime('January 31, 1969'),
    'tokens' => ["a", "b"],
], [
    'excludeFromIndexes' => [
        'dateOfBirth',
        'tokens[]',
        'tokens[].0'
    ]
]);

src: https://googleapis.dev/nodejs/datastore/latest/Datastore.html#save-examples

[crwilcox]

Relevant node code from link above:

//-
// Use an array, `excludeFromIndexes`, to exclude properties from indexing.
// This will allow storing string values larger than 1500 bytes.
//-
const entity = {
  key: datastore.key('Company'),
  excludeFromIndexes: [
    'description',
    'embeddedEntity.description',
    'arrayValue[]',
    'arrayValue[].description'
  ],
  data: {
    description: 'Long string (...)',
    embeddedEntity: {
      description: 'Long string (...)'
    },
    arrayValue: [
      'Long string (...)',
      {
        description: 'Long string (...)'
      }
    ]
  }

Looking at the node code your sample looks pretty similar. The tokens[] should exclude the top-level unnamed values in an array. There is a potential hangup if the php library (like Ben, not a language expert) is passed as tokens or tokens[]. For instance, if you retrieve the excluded index list in the python library longStringArray[] is just longStringArray.

A more complete example can be found in the system tests for the node.js library. https://github.com/googleapis/nodejs-datastore/blob/cf88927998ada76614fbe500c6e59e0d81c1b78b/system-test/datastore.ts#L78

The system test code code excludes the indexes for {'longStringArray', 'longString'}, along with metadata. The fields excluded couldn't be indexed as they are too long, so they are being excluded.

[GuillaumeMiralles]

Sorry my previous message might be confusing. The example I did was made for PHP, but using the nodeJS syntax to exclude list or items of list from index to show what I want to do. But then my question is how to do this in full PHP syntax.

I did retry the syntax like tokens[], tokens[].0 or tokens.0 but this doesn't do anything. I still get the json with no exclude from index.

{
  "values": [
    {
      "stringValue": "a"
    }
  ]
}

[jdpedrie]

Thanks @bshaffer @crwilcox @BenWhitehead for weighing in.

@GuillaumeMiralles looks like we have some missing functionality. I'll mark this as a feature request and get it added for you as soon as I can.