Google-cloud-python: PubSub: import pubsub fails with ImportError: No module named grpc._cython.cygrpc

[commilfo]

Hi,

Our project is based on Python2.7, Google Apps Engine, and we use Pubsub.
In one of our modules we call pubsub to publish events.

The project is legacy and adding new libraries required upgrades to existing libs.
Now we have latest google cloud components, latest google-cloud-core, google-cloud-pubsub and other libs, I'll list them below.
We've made a fresh install of everything - python2.7, google cloud and all the libs.
Everything is identical and fails identical on two machines, Ubuntu 16.04 LTS and Windows 10:

Traceback (most recent call last):
File "C:\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "C:\dev\tracker-api\server\main.py", line 10, in
import routes as routes
File "C:\dev\tracker-api\server\routes.py", line 3, in
from controller import (
File "C:\dev\tracker-api\server\application\controller\entity.py", line 9, in
from controller.users import register_user, delete_user
File "C:\dev\tracker-api\server\application\controller\users.py", line 11, in
from controller.team_notification import enqueue_team_notification_task, dequeue_team_notification_task
File "C:\dev\tracker-api\server\application\controller\team_notification.py", line 17, in
from services.publisher import publish_team_notification
File "C:\dev\tracker-api\server\application\services\publisher.py", line 10, in
from google.cloud import pubsub
File "C:\dev\tracker-api\server\packages\google\cloud\pubsub.py", line 17, in
from google.cloud.pubsub_v1 import PublisherClient
File "C:\dev\tracker-api\server\packages\google\cloud\pubsub_v1__init__.py", line 18, in
from google.cloud.pubsub_v1 import publisher
File "C:\dev\tracker-api\server\packages\google\cloud\pubsub_v1\publisher__init__.py", line 17, in
from google.cloud.pubsub_v1.publisher.client import Client
File "C:\dev\tracker-api\server\packages\google\cloud\pubsub_v1\publisher\client.py", line 21, in
import grpc
File "C:\dev\tracker-api\server\packages\grpc__init__.py", line 22, in
from grpc._cython import cygrpc as _cygrpc
File "C:\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\python\runtime\sandbox.py", line 1095, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named grpc._cython.cygrpc

Of course there's module grpc, there's a folder _cython in it, though there's no cygrpc.py file (but cygrpc.pyd in Windows and cygrpc.so in Ubuntu).

Please advice how to proceed.

grpc-google-iam-v1==0.11.4
grpcio==1.13.0

Google Cloud SDK version 209.0.0.
Our requirements.txt:

nose==1.3.7
ferrisnose==2.3.1
six~=1.10.0
google-api-python-client==1.7.4
googlemaps==3.0.2
responses==0.5.1
python-dateutil
arrow
oauth2client==4.1.2
braintree
google-cloud-core==0.28.0
geopy==1.11.0
polyline==1.3.2
validators
lxml==3.8.0
xmltodict
requests==2.19.1
google-cloud-pubsub==0.35.4
sendgrid==5.4.1
keen==0.5.1
oauthlib==2.1.0
twilio==6.15.1
plivo==4.1.1
phonenumbers==8.9.10
quotequail==0.2.3
addict==2.1.3
sphere-engine==0.14

[tseaver]

GAE support can be problematic (see #1893), particularly for gRPC_based libraries: GAE allows loading only whitelisted C extensions, which doesn't include grpc. google-cloud-pubsub 0.28.0 dropped support for the HTTP (non-gRPC) transport.

For now, the recommended way to talk to Google APIs from App Engine standard remains the Google API Client Library.

[commilfo]

Do you mean Google App Engine and modern PubSub are incompatible and will not work together?

[tseaver]

@commilfo I mean that it is highly likely that current versions of the google-cloud-pubsub library maintained here do not work on GAE as of this writing. The last version with (optional) HTTP-only support was google-cloud-pubsub 0.27.0, with the GOOGLE_CLOUD_DISABLE_GRPC environment variable set to a truthy value.

Please see the GAE Python Release Notes: the release which added grpcio support specifically disclaims using it in conjunction with the google-cloud-* libraries.

We are tracking the overall GAE compatibility issue at #1893.

[commilfo]

Ok, thank you.