Google-cloud-python: Client hangs when using from_service_account_json from the datastore library

[ajudi]

I can't get anything from the datastore or put stuff in the data store from the python lib when using the service account.

1. Linux, Ubuntu 16.04
2. Python 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609] on linux2
3. Name: google-cloud
   Version: 0.27.0
   Summary: API Client library for Google Cloud
   Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
   Author: Google Cloud Platform
   Author-email: [email protected]
   License: Apache 2.0
   Location: /home/tom/.virtualenvs/test/lib/python2.7/site-packages
   Requires: google-cloud-bigtable, google-cloud-core, google-cloud-language, google-cloud-pubsub, google-cloud-error-reporting, google-cloud-videointelligence, google-cloud-speech, google-cloud-translate, google-cloud-storage, google-cloud-runtimeconfig, google-cloud-datastore, google-cloud-vision, google-cloud-monitoring, google-cloud-bigquery, google-cloud-resource-manager, google-cloud-dns, google-cloud-logging, google-cloud-spanner
   4.

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-cc19367a0521> in <module>()
----> 1 client.get(key)

NameError: name 'key' is not defined

In [4]: key = client.key('test')

In [5]: client.get(key)
---------------------------------------------------------------------------
GatewayTimeout                            Traceback (most recent call last)
<ipython-input-5-cc19367a0521> in <module>()
----> 1 client.get(key)

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/client.pyc in get(self, key, missing, deferred, transaction)
    307         """
    308         entities = self.get_multi(keys=[key], missing=missing,
--> 309                                   deferred=deferred, transaction=transaction)
    310         if entities:
    311             return entities[0]

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/client.pyc in get_multi(self, keys, missing, deferred, transaction)
    354             missing=missing,
    355             deferred=deferred,
--> 356             transaction_id=transaction and transaction.id,
    357         )
    358 

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/client.pyc in _extended_lookup(datastore_api, project, key_pbs, missing, deferred, eventual, transaction_id)
    136         loop_num += 1
    137         lookup_response = datastore_api.lookup(
--> 138             project, read_options, key_pbs)
    139 
    140         # Accumulate the new results.

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.pyc in lookup(self, *args, **kwargs)
    113         """
    114         with _catch_remap_gax_error():
--> 115             return super(GAPICDatastoreAPI, self).lookup(*args, **kwargs)
    116 
    117     def run_query(self, *args, **kwargs):

/usr/lib/python2.7/contextlib.pyc in __exit__(self, type, value, traceback)
     33                 value = type()
     34             try:
---> 35                 self.gen.throw(type, value, traceback)
     36                 raise RuntimeError("generator didn't stop after throw()")
     37             except StopIteration, exc:

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.pyc in _catch_remap_gax_error()
     80         else:
     81             new_exc = error_class(exc.cause.details())
---> 82             six.reraise(error_class, new_exc, sys.exc_info()[2])
     83 
     84 

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.pyc in _catch_remap_gax_error()
     72     """
     73     try:
---> 74         yield
     75     except GaxError as exc:
     76         error_code = exc_to_code(exc.cause)

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/datastore/_gax.pyc in lookup(self, *args, **kwargs)
    113         """
    114         with _catch_remap_gax_error():
--> 115             return super(GAPICDatastoreAPI, self).lookup(*args, **kwargs)
    116 
    117     def run_query(self, *args, **kwargs):

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/cloud/gapic/datastore/v1/datastore_client.pyc in lookup(self, project_id, read_options, keys, options)
    202         request = datastore_pb2.LookupRequest(
    203             project_id=project_id, read_options=read_options, keys=keys)
--> 204         return self._lookup(request, options)
    205 
    206     def run_query(self,

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/gax/api_callable.pyc in inner(request, options)
    450                 func, this_settings.timeout, **this_settings.kwargs)
    451         api_call = _catch_errors(api_call, gax.config.API_ERRORS)
--> 452         return api_caller(api_call, this_settings, request)
    453 
    454     if settings.page_descriptor:

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/gax/api_callable.pyc in base_caller(api_call, _, *args)
    436     def base_caller(api_call, _, *args):
    437         """Simply call api_call and ignore settings."""
--> 438         return api_call(*args)
    439 
    440     def inner(request, options=None):

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/gax/api_callable.pyc in inner(*args, **kwargs)
    374         """Wraps specified exceptions"""
    375         try:
--> 376             return a_func(*args, **kwargs)
    377         # pylint: disable=catching-non-exception
    378         except tuple(to_catch) as exception:

/home/tom/.virtualenvs/test/local/lib/python2.7/site-packages/google/gax/retry.pyc in inner(*args)
    141                         timeout * timeout_mult, max_timeout, deadline - now)
    142 
--> 143         raise exc
    144 
    145     return inner

GatewayTimeout: 504 Deadline Exceeded

5.

client = datastore.Client.from_service_account_json('./serviceAccount.json')
key = client.key('test') 
client.get(key)

Tested from the node version of the datastore library and it does work there so doesn't seem to be an auth issue.

[dhermes]

@ajudi Can you try this with HTTP and see if you get a similar error? You can do this by setting the env. variable GOOGLE_CLOUD_DISABLE_GRPC=True.

[ajudi]

@dhermes, thanks! This seems to solve the issue but do you know why GRPC wouldn't be working for me. Seems to be installed and importable. As another note the datastore also works when using the local emulator without the env_var set.

[dhermes]

@ajudi I was "hoping" it'd be the same error with a better error message.

[mlaferrera]

I've been running into this same issue. Settings the above ENV variable does seem to work. One thing that I did notice in my testing is if I run client.get(key) within the python interactive shell it will hang, then if I hit ctrl-c and re-run client.get(key) without instantiating client again, it will successfully run.

[dhermes]

@mlaferrera That makes it sound like a gRPC issue (which is what I think may be happening with @ajudi). Do you mind sharing what OS you are running on? And what version of Python?

[mlaferrera]

Certainly. I was also attempting to test on python 3.5.2 and was experiencing the same issue.

Linux testbox 4.4.52+ #1 SMP Thu Jul 13 11:47:20 PDT 2017 x86_64 Linux

Python 3.6.1 (default, May  2 2017, 15:16:41)
[GCC 6.3.0] on linux

[dhermes]

@mlaferrera What version of grpcio do you have installed? Can you consistently reproduce this behavior?

[mlaferrera]

Yes, this is consistently reproducible when datastore.Client() is instantiated. However, this was previously working fine in a production environment. From my logs, it appears the problem started at 18:49:25.000 on 9/27/2017.

Below are the python libs installed:

$ pip freeze
Babel==2.3.4
billiard==3.5.0.2
blinker==1.4
certifi==2017.7.27.1
chardet==3.0.4
click==4.1
colorlog==2.10.0
contextlib2==0.5.5
dill==0.2.6
enum34==1.1.6
future==0.16.0
gapic-google-cloud-datastore-v1==0.14.1
gapic-google-cloud-logging-v2==0.90.1
gapic-google-cloud-pubsub-v1==0.14.1
google-auth==0.6.0
google-auth-httplib2==0.0.2
google-cloud==0.22.0
google-cloud-bigquery==0.22.1
google-cloud-bigtable==0.22.0
google-cloud-core==0.22.1
google-cloud-datastore==0.22.1
google-cloud-dns==0.22.0
google-cloud-error-reporting==0.22.0
google-cloud-happybase==0.22.0
google-cloud-language==0.22.2
google-cloud-logging==0.22.0
google-cloud-monitoring==0.22.0
google-cloud-pubsub==0.22.0
google-cloud-resource-manager==0.22.0
google-cloud-runtimeconfig==0.22.0
google-cloud-storage==0.22.0
google-cloud-translate==0.22.0
google-cloud-vision==0.22.0
google-gax==0.15.4
googleapis-common-protos==1.5.1
grpc-google-cloud-datastore-v1==0.14.0
grpc-google-cloud-logging-v2==0.90.0
grpc-google-cloud-pubsub-v1==0.14.0
grpc-google-iam-v1==0.11.1
grpcio==1.1.0
httplib2==0.10.3
idna==2.6
itsdangerous==0.24
jsonschema==2.6.0
oauth2client==3.0.0
ply==3.8
protobuf==3.2.0
psq==0.5.0
pyasn1==0.2.2
pyasn1-modules==0.0.8
requests==2.18.4
retrying==1.3.3
rsa==3.4.2
six==1.10.0
tornado==4.2
urllib3==1.22
Werkzeug==0.11.15

[dhermes]

@mlaferrera I was hoping grpcio==1.6.0 would be the culprit, but I'm equally surprised to see grpcio==1.1.0. How are you installing the packages?

[mlaferrera]

Well that is silly on my part. I was running an old image that initially had the issue as of the above time. I rebuilt the image with grpcio==1.6.3 via pip install google-cloud==0.27.0.

Without the ENV variable GOOGLE_CLOUD_DISABLE_GRPC=True set, I then ran:

>>> from google.cloud import datastore
>>> client = datastore.Client()
>>> key = client.key(key, value)
>>> client.get(key)
<Entity('mykey', 'myvalue')...

Success! With grpcio==1.6.3 it does not appear to require the ENV set.

However, if I revert to the old image (with the previously posted libs) and run the same thing, it hangs on client.get(key). Hitting ctrl-c then running client.get(key) runs successfully. If I set the ENV variable and do the same, client.get(key) runs successfully every time.

Sorry for the confusion.

[dhermes]

@mlaferrera FWIW the ENV var actually changes the transport from HTTP 1.1 to gRPC (which uses HTTP 2). You can "see" this by viewing client._datastore_api.

ASIDE: I would recommend that you use grpcio==1.4.0 due to https://github.com/grpc/grpc/issues/12455

[tseaver]

Given that an update fixed the OPs problem, and that grpcio 1.7.0 is out with a fix for the grpc/grpc#12455 issue, I'm going to close this one.

[titocosta]

Having the same issue on MacOs. Google Cloud Datastore client hangs on queries and gets.
Setting os.environ["GOOGLE_CLOUD_DISABLE_GRPC"] = "True" did not fix it for me.

I have grpcio 1.17.1 installed.

[titocosta]

I fixed it as follows:

1. Used export GOOGLE_CLOUD_DISABLE_GRPC=True on bash before invoking python
2. I solved the authentication problem with $ gcloud auth application-default login as described here