Google-cloud-python: Firestore: ArrayUnion not working in latest version

[GuanacoDevs]

Environment details

1. google-cloud-firestore==0.32.1
2. google-cloud-function

Steps to reproduce

1. When using ArrayUnion with google-cloud-firestore==0.32.1, the function crashes with the following message:

TypeError: ('Cannot convert to a Firestore Value', <google.cloud.firestore_v1beta1.transforms.ArrayUnion object at 0x2a79c19b2fd0>, 'Invalid type', <class 'google.cloud.firestore_v1beta1.transforms.ArrayUnion'>)

Code example

Virtually the same code from the snippets

from google.cloud.firestore_v1beta1 import ArrayUnion
events_ref.update({'events': ArrayUnion([event_id])})

Stack trace

Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 383, in run_background_function
    _function_handler.invoke_user_function(event_object)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 214, in call_user_function
    event_context.Context(**request_or_event.context))
  File "/user_code/main.py", line 62, in user_delete_anonymous
    if check_event(context.event_id):
  File "/user_code/main.py", line 27, in check_event
    events_ref.update({'events': ArrayUnion([event_id])})
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/document.py", line 379, in update
    batch.update(self, field_updates, option=option)
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/batch.py", line 112, in update
    reference._document_path, field_updates, option
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/_helpers.py", line 822, in pbs_for_update
    update_pb = extractor.get_update_pb(document_path)
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/_helpers.py", line 459, in get_update_pb
    name=document_path, fields=encode_dict(self.set_fields)
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/_helpers.py", line 215, in encode_dict
    return {key: encode_value(value) for key, value in six.iteritems(values_dict)}
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/_helpers.py", line 215, in <dictcomp>
    return {key: encode_value(value) for key, value in six.iteritems(values_dict)}
  File "/env/local/lib/python3.7/site-packages/google/cloud/firestore_v1/_helpers.py", line 200, in encode_value
    "Cannot convert to a Firestore Value", value, "Invalid type", type(value)
TypeError: ('Cannot convert to a Firestore Value', <google.cloud.firestore_v1beta1.transforms.ArrayUnion object at 0x2a79c19b2fd0>, 'Invalid type', <class 'google.cloud.firestore_v1beta1.transforms.ArrayUnion'>)

Changing to google-cloud-firestore==0.31.0 in my requirements.txt fixes the issue, tested 0.32.0 with the same crash.

Best Regards

[tseaver]

I am not able to reproduce the issue:

>>> from google.cloud import firestore
>>> firestore.__version__
'0.32.1'
>>> from google.cloud.firestore_v1beta1 import ArrayUnion, Client
>>> client = Client.from_service_account_json('/path/to/service_account.json')
>>> collection = client.collection('gcp-7720')
>>> events_ref = collection.document('testing')
>>> events_ref.set({'events': ['a', 'b', 'c']})
update_time {
  seconds: 1555523680
  nanos: 287775000
}

>>> events_ref.get().to_dict()
{'events': ['a', 'b', 'c']}
>>> events_ref.update({'events': ArrayUnion(['d'])})
update_time {
  seconds: 1555523697
  nanos: 585277000
}
transform_results {
  null_value: NULL_VALUE
}

>>> events_ref.get().to_dict()
{'events': ['a', 'b', 'c', 'd']}

Can you please show how your code is creating / initializing events_ref?

[GuanacoDevs]

I did not tried this locally from my computer, the error happens when the function gets triggered for a onCreate event in google cloud functions. I did deploy the function using 0.32.1, 0.32.0 and 0.31.0 in the requirements, the error is not thrown with 0.31.0

I'm using this function to deal with event replication, the relevant code:

import logging

from google.cloud import firestore
from google.cloud.firestore_v1beta1 import ArrayUnion

db = firestore.Client()

def check_event(event_id: str):
    events_ref = `db.document('utilities/events')`
    events = events_ref.get().to_dict()
    if event_id not in events:
        logging.info('This is a new event, adding it to the list.')
        events_ref.update({'events': ArrayUnion([event_id])})

requirements.txt

firebase-admin==2.16.0 
google-cloud-firestore==0.32.1

[tseaver]

I still cannot see how to reproduce your traceback.

It might be unrelated, but your if statement is not testing what I think you mean: events is a dict, whose only key (as far as I can tell from your example) is events. Maybe the following is what you want?

def check_event(event_id: str):
    events_ref = `db.document('utilities/events')`
    event_snapshot = events_ref.get()
    event_list = event_snapshot.get('events')

    if event_id not in event_list:
        logging.info('This is a new event, adding it to the list.')
        events_ref.update({'events': ArrayUnion([event_id])})

[GuanacoDevs]

Maybe the issue only happened when deploy to google cloud, and that's already updated?
The code works well when deploying using 0.31.0, I'll test again maybe is already fixed in the cloud and close the issue.
There was a user on SO who had the same issue.
events is a document that is meant to contain an array of event_id, bellow is the full check_event I posted what I thought was relevant, but you are right from what I posted.

def check_event(event_id: str):
    events_ref = db.document('utilities/events')
    events = events_ref.get().to_dict()
    if not events:
        events_ref.set({'events': [event_id]})
        logging.info('Adding event to events list.')
        return False
    elif event_id in events['events']:
        logging.warning('Event already in the list, exiting this instance.')
        return True
    else:
        logging.info('This is a new event, adding it to the list.')
        events_ref.update({'events': ArrayUnion([event_id])})
        return False

[GuanacoDevs]

I just tested deploying with 0.32.1 and still the same error when the function is triggered.

[tseaver]

One thing I see in that traceback is that the code in the frames is from firestore_v1, but you are trying to save the ArrayUnion type from firestore_v1beta1. Can you change the import?

[GuanacoDevs]

Sorry @tseaver you lost me there, I'm using code from the snippets.
How the import should be?

[tseaver]

from google.cloud.firestore_v1 import ArrayUnion

That type should be importable from google.cloud.firestore, but it is not at the moment.

[OlegNovosad]

Please update your docs here: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/firestore/cloud-client/snippets.py with proper import. This snippet is already referenced multiple time on different resources and took me couple hours to get to this "update import" :-)

[tseaver]

@OlegNovosad Thanks for the prod. @BenWhitehead updated the import yesterday in https://github.com/GoogleCloudPlatform/python-docs-samples/pull/2137.

[mahi1518]

https://github.com/googleapis/google-cloud-python/issues/7720#issuecomment-484654762
Thanx @tseaver it work