Google-cloud-python: When I use grpc for datastore_api to get entities in GAE flex, there is no response.

[getumen]

Yesterday, I update my project's environment from vm: true to env: flex and datastore api version from 0.22.* to 1.0.0.
In local test, there is no problem and I can fetch entities with datastore_api=make_datastore_api(self)(google-cloud-python/datastore/google/cloud/datastore/client.py)
However, in GAE flex, when I use grpc for datastore_api to get entities in GAE flex, there is no response.
I can get entities in GAE flex when I use HTTPDatastoreAPI as datastore_api.
How can I use grpc for datastore_api in GAE flex?

1. OS type and version
   Local: Ubuntu 16.04
   Cloud: Google App Engine Flexible Environment
2. Python version and virtual environment information python --version
   Python 3.5.3
3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

cachetools==2.0.0
certifi==2017.4.17
chardet==3.0.3
click==6.7
colorama==0.3.9
defusedxml==0.5.0
dill==0.2.6
Django==1.10
django-allauth==0.27.0
django-bmemcached==0.2.3
django-bootstrap3==7.1.0
django-password-validation==0.1.1
django-recaptcha==1.3.0
docopt==0.6.2
enum34==1.1.6
feedparser==5.2.1
future==0.16.0
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
gapic-google-cloud-pubsub-v1==0.15.4
gapic-google-cloud-spanner-admin-database-v1==0.15.3
gapic-google-cloud-spanner-admin-instance-v1==0.15.3
gapic-google-cloud-spanner-v1==0.15.3
gapic-google-cloud-speech-v1beta1==0.15.3
gapic-google-cloud-vision-v1==0.90.3
geoip2==2.5.0
gevent==1.2.2
gitsome==0.7.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud==0.25.0
google-cloud-bigquery==0.24.0
google-cloud-bigtable==0.24.0
google-cloud-core==0.24.1
google-cloud-datastore==1.0.0
google-cloud-dns==0.24.0
google-cloud-error-reporting==0.24.2
google-cloud-happybase==0.24.0
google-cloud-language==0.24.1
google-cloud-logging==1.0.0
google-cloud-monitoring==0.24.0
google-cloud-pubsub==0.25.0
google-cloud-resource-manager==0.24.0
google-cloud-runtimeconfig==0.24.0
google-cloud-spanner==0.24.2
google-cloud-speech==0.24.0
google-cloud-storage==1.1.1
google-cloud-translate==0.24.0
google-cloud-vision==0.24.0
google-gax==0.15.13
google-resumable-media==0.0.2
googleapis-common-protos==1.5.2
greenlet==0.4.12
grpc-google-cloud-datastore-v1==0.14.0
grpc-google-iam-v1==0.11.1
grpcio==1.3.5
gunicorn==19.7.1
html5lib==1.0b8
httplib2==0.10.3
idna==2.5
iso8601==0.1.11
Markdown==2.6.6
maxminddb==1.3.0
mysqlclient==1.3.7
numpy==1.13.0
numpydoc==0.6.0
oauth2client==3.0.0
oauthlib==2.0.2
ply==3.8
prompt-toolkit==1.0.14
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-error-reporting-v1beta1==0.15.3
proto-google-cloud-logging-v2==0.91.3
proto-google-cloud-pubsub-v1==0.15.4
proto-google-cloud-spanner-admin-database-v1==0.15.3
proto-google-cloud-spanner-admin-instance-v1==0.15.3
proto-google-cloud-spanner-v1==0.15.3
proto-google-cloud-speech-v1beta1==0.15.3
proto-google-cloud-vision-v1==0.90.3
protobuf==3.3.0
pyasn1==0.2.3
pyasn1-modules==0.0.9
pycrypto==2.6.1
Pygments==2.2.0
pylibmc==1.5.2
python-binary-memcached==0.26.0
python3-openid==3.1.0
pytz==2017.2
requests==2.17.3
requests-oauthlib==0.8.0
rsa==3.4.2
six==1.10.0
tweepy==3.5.0
ua-parser==0.7.3
uritemplate==3.0.0
uritemplate.py==3.0.2
urllib3==1.21.1
user-agents==1.1.0
wcwidth==0.1.7

1. Stacktrace if available
   No error occur.
2. Steps to reproduce
   I don't know whether this is reproducable.
3. Code example

from google.cloud import datastore
c = datastore.Client()
c.get(c.key('kind', 'name')) # freeze here in GAE flex.

[dhermes]

@getumen At the beginning of your post, you mention using the non-public make_datastore_api instead of just using a Client, but below it seems like you are using a Client (I just want to be clear which is done).

• What is c.project?
• Do you mean it freezes or the response is empty ("there is no response" is a bit unclear)?

[getumen]

My understanding is that Datatore Api use make_datastore_api by default.
In my setting, Datastore Api freezes when I try to get an entity in GAE flex.
However by using env_variables: GOOGLE_CLOUD_DISABLE_GRPC: True in app.yaml to use HTTPDatastoreAPI, I can get an entity.

What is c.project?

print(c.project) returns my project id in both the local and the GAE flex environment.

Do you mean it freezes or the response is empty ("there is no response" is a bit unclear)?

I mean it freezes. After c.get(key) or list(query_result_iterator), I get Http 502 because of GAE response time limitation.

[dhermes]

Thanks for the info. @jonparrott I'm not quite sure how to repro in GAE flex, can we chat over Hangouts?

[getumen]

I find how to reproduce this issue now.
Datastore API freezes when I use gevent with gunicorn.

In the local setting, my program work with the following gunicorn.conf.py:

workers = 4
threads = 4
forwarded_allow_ips = '*'
secure_scheme_headers = {'X-APPENGINE-HTTPS': 'on'}

However, my program does not work with the following gunicorn.conf.py

workers = 4
threads = 4
worker_class = 'gevent'
forwarded_allow_ips = '*'
secure_scheme_headers = {'X-APPENGINE-HTTPS': 'on'}

My gunicorn and gevent version are 19.7.1 and 1.2.2, respectively.

Now I can use grpc for Datastore API if I do not use gevent.
I thought I can use gevent because the documentation mention it (https://cloud.google.com/appengine/docs/flexible/python/runtime#recommended_gunicorn_configuration).
Can I use gevent?
To use it, What should I do?

[getumen]

3450

My issue is deplicated.

[dhermes]

Thanks for the update @getumen, unfortunately there is nothing we can do at this time, though you can follow along on https://github.com/grpc/grpc/issues/4629 / voice your concerns there (sorry I can't offer better).

For the time being you can use HTTP instead of gRPC by setting the GOOGLE_CLOUD_DISABLE_GRPC environment variable (to anything at all, just make sure it's set).

[zero-master]

@getumen Have you found any workaround? Thanks.

[zero-master]

@dhermes Do you know why it doesn't affect BigQuery?
We also use gunicorn + bigquery + gevent in production but never faced this issue.

But when we drop in Datastore, it just blocks forever and workers timeout.

[22540] [CRITICAL] WORKER TIMEOUT (pid:22551)

Tested on app engine flexible with

google-cloud-datastore==1.4.0
google-cloud-bigquery==0.28.0
grpcio==1.7.0
gunicorn==19.7.1

[getumen]

@zero-master My understanding is that gRPC doesn't work with gevent correctly.
To avoid it, there are two choices:

1. Set GOOGLE_CLOUD_DISABLE_GRPC (environment variable) true to use Http.
   More precisely, add config in app.yaml as follows:

env_variables:
  GOOGLE_CLOUD_DISABLE_GRPC: True

1. Not use gevent

Hoping that the information will be of some help to you.

[zero-master]

@getumen Thanks again :)

I was able to reproduce this behavior by running gunicorn locally with gunicorn -c gunicorn.conf.py -b :5000 wsgi:app

gunicorn.cong.py contains

worker_class = 'gevent'

Switching to sync workers just doubled my instance count. I'll be trying my app with gRPC disabled.

[dhermes]

@zero-master BigQuery uses an HTTP transport, so there is no gRPC issue.

Most APIs have both HTTP and gRPC support (e.g. Datastore) but some have HTTP only (e.g. BigQuery) and some have gRPC only (e.g. BigTable, Spanner, Firestore). For those that have both @getumen suggestion will work great to avoid gRPC.

[dhermes]

From what I understand, the gunicorn issue was a problem in gprcio==1.6.0 involving multiprocessing (from the Python std. library). As I understand it, grpcio==1.7.0 has fixed this issue.

[zero-master]

@dhermes Thanks you for the details.
I still managed to reproduce this one with grpcio==1.7.0

You just have to get an object from datastore and run gunicorn with "gevent" worker class to successfully reproduce this issue.

[dhermes]

That's great data (as far as making grpcio more reliable)! Thanks for providing it (and sorry that you have a stalled application).

I have never used gunicorn, do you mind providing a code snippet I could run to reproduce / modify / make simpler and report to the grpcio issue tracker?

[zero-master]

@dhermes Here you go
https://github.com/zero-master/grcp-issue

On, step 4 you'll see that it blocks forever but if you use sync worker class by commenting the last line, it will work as expected.

http://docs.gunicorn.org/en/stable/design.html

[sengoontoh]

Has anyone tried this solution? It worked for me

https://github.com/grpc/grpc/issues/4629

from gevent import monkey
monkey.patch_all()

import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

[tszumowski]

Has anyone tried this solution? It worked for me

grpc/grpc#4629

from gevent import monkey
monkey.patch_all()

import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

This worked perfectly for me to get gevent and either datastore OR spanner working. Thanks!