Google-cloud-python: PubSub: 0.38.0 produces import error fo 'IAMPolicyStub'

[goern]

Please run down the following list and make sure you've tried the usual "quick fixes":

#### Environment details

1. Cloud PubSub, Hangouts Chat
2. OS

NAME=Fedora
VERSION="29 (Workstation Edition)"
ID=fedora
VERSION_ID=29
PLATFORM_ID="platform:f29"
PRETTY_NAME="Fedora 29 (Workstation Edition)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:29"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=29
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=29
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

1. Python version and virtual environment information: Python 3.6.6
2. google-cloud- version: pip show google-<service> or pip freeze

aiofiles==0.4.0
aniso8601==3.0.2
apispec==1.0.0b5
apistar==0.6.0
asgiref==2.3.2
async-timeout==3.0.1
cachetools==3.0.0
certifi==2018.10.15
chardet==3.0.4
Click==7.0
daiquiri==1.5.0
docopt==0.6.2
google-api-core==1.6.0a1
google-api-python-client==1.7.4
google-auth==1.6.1
google-auth-httplib2==0.0.3
google-cloud-pubsub==0.38.0
googleapis-common-protos==1.6.0b6
graphene==2.1.3
graphql-core==2.1
graphql-relay==0.4.5
graphql-server-core==1.1.1
grpc-google-iam-v1==0.11.4
grpcio==1.16.1
h11==0.8.1
httplib2==0.12.0
httptools==0.0.11
idna==2.7
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
marshmallow==3.0.0b20
oauth2client==4.1.3
parse==1.9.0
prometheus-client==0.4.2
promise==2.2.1
protobuf==3.6.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
python-multipart==0.0.5
pytz==2018.7
PyYAML==4.2b4
requests==2.20.1
requests-toolbelt==0.8.0
responder==1.1.1
rfc3986==1.1.0
rfc5424-logging-handler==1.3.0
rsa==4.0
Rx==1.6.1
sentry-sdk==0.5.4
six==1.11.0
starlette==0.7.4
thoth-common==0.4.0
tzlocal==2.0.0b1
uritemplate==3.0.0
urllib3==1.24.1
uvicorn==0.3.20
uvloop==0.12.0rc1
websockets==7.0
whitenoise==4.1.1

Steps to reproduce

1. git clone github.com:goern/sesheta-google-chatbot.git .
2. pipenv install
3. pipenv shell ./app.py

Code example

# example

Stack trace

./app.py                                                                                                                                                               
2018-11-15 13:57:06,018 [19189] INFO     root: Logging to a Sentry instance is turned off
2018-11-15 13:57:06,018 [19189] INFO     root: Logging to rsyslog endpoint is turned off
2018-11-15 13:57:06,024 [19189] CRITICAL root: Traceback (most recent call last):
  File "./app.py", line 50, in <module>
    subscriber = pubsub.SubscriberClient()
  File "/home/goern/.local/share/virtualenvs/google-chatbot-MYBzcWvJ/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/client.py", line 76, in __init__
    self._api = subscriber_client.SubscriberClient(**kwargs)
  File "/home/goern/.local/share/virtualenvs/google-chatbot-MYBzcWvJ/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 188, in __init__
    credentials=credentials,
  File "/home/goern/.local/share/virtualenvs/google-chatbot-MYBzcWvJ/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/transports/subscriber_grpc_transport.py", line 72, in __init__
    'iam_policy_stub': iam_policy_pb2.IAMPolicyStub(channel),
AttributeError: module 'google.iam.v1.iam_policy_pb2' has no attribute 'IAMPolicyStub'

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[tseaver]

Trying to reproduce by installing google-cloud-pubsub==0.38.0 in a fresh virtual environment:

$ python3.6 -m venv /tmp/gcp/6531
$ /tmp/gcp/6531/bin/pip install --upgrade setuptools pip wheel
...
Successfully installed cachetools-3.0.0 certifi-2018.10.15 chardet-3.0.4 google-api-core-1.5.2 google-auth-1.6.1 google-cloud-pubsub-0.38.0 googleapis-common-protos-1.5.5 grpc-google-iam-v1-0.11.4 grpcio-1.16.1 idna-2.7 protobuf-3.6.1 pyasn1-0.4.4 pyasn1-modules-0.2.2 pytz-2018.7 requests-2.20.1 rsa-4.0 six-1.11.0 urllib3-1.24.1
$ /tmp/gcp/6531/bin/pip freeze
-f file:///...
cachetools==3.0.0
certifi==2018.10.15
chardet==3.0.4
google-api-core==1.5.2
google-auth==1.6.1
google-cloud-pubsub==0.38.0
googleapis-common-protos==1.5.5
grpc-google-iam-v1==0.11.4
grpcio==1.16.1
idna==2.7
protobuf==3.6.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pytz==2018.7
requests==2.20.1
rsa==4.0
six==1.11.0
urllib3==1.24.1
$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"
$ /tmp/gcp/6531/bin/python

Python 3.6.7 (default, Oct 22 2018, 11:30:50) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import pubsub
>>> client = pubsub.SubscriberClient()
>>> import google.iam.v1.iam_policy_pb2
>>> google.iam.v1.iam_policy_pb2.IAMPolicyStub
<class 'google.iam.v1.iam_policy_pb2.IAMPolicyStub'>

Can you check the actually-installed version of grpc-google-iam-v1, and that the imported file is the one you expect? E.g. via:

>>> import pkg_resources
>>> pkg_resources.get_distribution('grpc-google-iam-v1').version
'0.11.4'
>>> import google.iam.v1.iam_policy_pb2
>>> google.iam.v1.iam_policy_pb2.__file__
'/tmp/gcp/6531/lib/python3.6/site-packages/google/iam/v1/iam_policy_pb2.py'

[tseaver]

@goern Please feel free to reopen the issue if you can provide the information needed.

[julian-r]

I have the issue with the python37 environment in google cloud app. The installed version is the correct one according to pkg_resources, but google.iam.v1.iam_policy_pb2.IAMPolicyStub is not there.

[julian-r]

import google.iam.v1.iam_policy_pb2
import pkg_resources
print(google.iam.v1.iam_policy_pb2.__file__)
print(pkg_resources.get_distribution('grpc-google-iam-v1'))

produces:

2018-12-13 15:34:08 default[--]  /env/lib/python3.7/site-packages/google/iam/v1/iam_policy_pb2.py
2018-12-13 15:34:08 default[--]  grpc-google-iam-v1 0.11.4

[tseaver]

@julian-r How are you installing the packages? I just tried installing only grcp-google-iam-v1==0.11.4:

$ python3.7 -m venv /tmp/gcp/6531
$ /tmp/gcp/6531/bin/pip install --upgrade setuptools pip wheel
...
Successfully installed pip-18.1 setuptools-40.6.3 wheel-0.32.3
$ /tmp/gcp/6531/bin/pip install grpc-google-iam-v1==0.11.4
...
Successfully installed googleapis-common-protos-1.5.5 grpc-google-iam-v1-0.11.4 grpcio-1.17.0 protobuf-3.6.1 six-1.12.0
$ /tmp/gcp/6531/bin/pip freeze
-f file:///.../.pip/wheels
googleapis-common-protos==1.5.5
grpc-google-iam-v1==0.11.4
grpcio==1.17.0
protobuf==3.6.1
six==1.12.0
$ /tmp/gcp/6531/bin/python 
Python 3.7.1 (default, Oct 22 2018, 11:31:38) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.iam.v1 import iam_policy_pb2
>>> iam_policy_pb2.IAMPolicyStub
<class 'google.iam.v1.iam_policy_pb2.IAMPolicyStub'>

[julian-r]

Fun fact: it works on my desktop environment, but in app engine it can't find the class. Locally I am using pipenv. I generate a requirements.txt with pipenv freeze --requirements and app engine installs them.

Now I also tried to check if the imports of iam_policy_pb2.py:300 are working

  # THESE ELEMENTS WILL BE DEPRECATED.
  # Please use the generated *_pb2_grpc.py files instead.
  import grpc
  from grpc.beta import implementations as beta_implementations
  from grpc.beta import interfaces as beta_interfaces
  from grpc.framework.common import cardinality
  from grpc.framework.interfaces.face import utilities as face_utilities

which seems to be fine.

[julian-r]

My versions in the requirements.txt:

google-api-core[grpc]==1.6.0
google-auth==1.6.1
google-cloud-core==0.29.0
google-cloud-pubsub==0.39.0
google-cloud-storage==1.13.1
google-resumable-media==0.3.1
googleapis-common-protos[grpc]==1.5.5
grpc-google-iam-v1==0.11.4
grpcio==1.17.0
...
protobuf==3.6.1

do exactly match yours.

[g3rd]

@julian-r have you fixed the issue? I am having the same problem.

My versions are slightly newer than yours and @tseaver.
(I've also tried downgrading to match)

Pipfile: Pipfile.txt

$ pip freeze
appdirs==1.4.3
appnope==0.1.0
astroid==2.1.0
attrs==18.2.0
backcall==0.1.0
black==18.9b0
cachetools==3.0.0
certifi==2018.11.29
chardet==3.0.4
Click==7.0
decorator==4.3.0
google-api-core==1.7.0
google-auth==1.6.2
google-cloud-pubsub==0.39.1
googleapis-common-protos==1.6.0b6
grpc-google-iam-v1==0.11.4
grpcio==1.17.1
idna==2.8
ipython==7.2.0
ipython-genutils==0.2.0
isort==4.3.4
jedi==0.13.2
lazy-object-proxy==1.3.1
mccabe==0.6.1
neo4j-driver==1.7.1
neobolt==1.7.3
neotime==1.7.4
parso==0.3.1
pexpect==4.6.0
pickleshare==0.7.5
prompt-toolkit==2.0.7
protobuf==3.6.1
ptyprocess==0.6.0
pyasn1==0.4.5
pyasn1-modules==0.2.3
Pygments==2.3.1
pylint==2.2.2
python-decouple==3.1
pytz==2018.7
requests==2.21.0
rsa==4.0
six==1.12.0
toml==0.10.0
traitlets==4.3.2
urllib3==1.24.1
wcwidth==0.1.7
wrapt==1.10.11

And my code (followed from https://github.com/googleapis/google-cloud-python/tree/master/pubsub#subscribing)

project_id = config("PROJECT_ID")
topic_name = config("TOPIC_NAME")

publisher = pubsub_v1.PublisherClient()
topic_name = "projects/{project_id}/topics/{topic_name}"
publisher.create_topic(topic_name)

Stacktrace when running:

~/Projects/platform/collectors/platform-collector/keystoneworks/app.py in <module>
     16
     17
---> 18 publisher = pubsub_v1.PublisherClient()
     19 topic_name = "projects/{project_id}/topics/{topic_name}"
     20 publisher.create_topic(topic_name)

~/.local/share/virtualenvs/platform-collector-Dej9gSN_/lib/python3.7/site-packages/google/cloud/pubsub_v1/publisher/client.py in __init__(self, batch_settings, **kwargs)
     87         # Add the metrics headers, and instantiate the underlying GAPIC
     88         # client.
---> 89         self.api = publisher_client.PublisherClient(**kwargs)
     90         self.batch_settings = types.BatchSettings(*batch_settings)
     91

~/.local/share/virtualenvs/platform-collector-Dej9gSN_/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py in __init__(self, transport, channel, credentials, client_config, client_info)
    169         else:
    170             self.transport = publisher_grpc_transport.PublisherGrpcTransport(
--> 171                 address=self.SERVICE_ADDRESS, channel=channel, credentials=credentials
    172             )
    173

~/.local/share/virtualenvs/platform-collector-Dej9gSN_/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/transports/publisher_grpc_transport.py in__init__(self, channel, credentials, address)
     69         # channel and provide a basic method for each RPC.
     70         self._stubs = {
---> 71             "iam_policy_stub": iam_policy_pb2.IAMPolicyStub(channel),
     72             "publisher_stub": pubsub_pb2_grpc.PublisherStub(channel),
     73         }

AttributeError: module 'google.iam.v1.iam_policy_pb2' has no attribute 'IAMPolicyStub'

Stracktrace when importing in the shell:

In [2]: from google.iam.v1 import iam_policy_pb2

In [3]: iam_policy_pb2.IAMPolicyStub
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-08df3e5719b4> in <module>
----> 1 iam_policy_pb2.IAMPolicyStub

AttributeError: module 'google.iam.v1.iam_policy_pb2' has no attribute 'IAMPolicyStub'

Any ideas?

Thanks!

[g3rd]

I got it to work. What I had to do.

Removed black (so I could remove allow_prereleases = true) Then I destroyed the venv and recreated. (Doining an uninstall of black and pipenv clean, didn't work)

Thoughts on why:
Allowing prereleases installed googleapis-common-protos==1.6.0b6.

[psincraian]

I had the same problem. @ambv please release a stable version of black :pray: