Google-cloud-python: KMS: import error when protobuf is < 3.6.x

[wyardley]

Environment details

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types
2. Debian buster/sid (Docker)
3. Python version and virtual environment information: Python 2.7.15
4. google-cloud- version: google-cloud-kms==0.2.1

Steps to reproduce

1. with protobuf < 3.6.0, import the module

Code example

from google.cloud import kms_v1

Stack trace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/google/cloud/kms_v1/__init__.py", line 19, in <module>
    from google.cloud.kms_v1 import types
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/google/cloud/kms_v1/types.py", line 22, in <module>
    from google.api import http_pb2
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/google/api/http_pb2.py", line 22, in <module>
    serialized_pb=_b('\n\x15google/api/http.proto\x12\ngoogle.api\"T\n\x04Http\x12#\n\x05rules\x18\x01 \x03(\x0b\x32\x14.google.api.HttpRule\x12\'\n\x1f\x66ully_decode_reserved_expansion\x18\x02 \x01(\x08\"\x81\x02\n\x08HttpRule\x12\x10\n\x08selector\x18\x01 \x01(\t\x12\r\n\x03get\x18\x02 \x01(\tH\x00\x12\r\n\x03put\x18\x03 \x01(\tH\x00\x12\x0e\n\x04post\x18\x04 \x01(\tH\x00\x12\x10\n\x06\x64\x65lete\x18\x05 \x01(\tH\x00\x12\x0f\n\x05patch\x18\x06 \x01(\tH\x00\x12/\n\x06\x63ustom\x18\x08 \x01(\x0b\x32\x1d.google.api.CustomHttpPatternH\x00\x12\x0c\n\x04\x62ody\x18\x07 \x01(\t\x12\x15\n\rresponse_body\x18\x0c \x01(\t\x12\x31\n\x13\x61\x64\x64itional_bindings\x18\x0b \x03(\x0b\x32\x14.google.api.HttpRuleB\t\n\x07pattern\"/\n\x11\x43ustomHttpPattern\x12\x0c\n\x04kind\x18\x01 \x01(\t\x12\x0c\n\x04path\x18\x02 \x01(\tBj\n\x0e\x63om.google.apiB\tHttpProtoP\x01ZAgoogle.golang.org/genproto/googleapis/api/annotations;annotations\xf8\x01\x01\xa2\x02\x04GAPIb\x06proto3')
TypeError: __new__() got an unexpected keyword argument 'serialized_options'

Confirmed that updating to 3.6.x fixes this. Originally, 3.5.1 is installed. Either the dependencies should be such that >= 3.6.x is forced, or the lib should be called in a backwards-compatible way.

[tseaver]

Working to reproduce in a clean 2.7 virtual environment:

$ python2.7 -m virtualenv /tmp/gcp/7040
New python executable in /tmp/gcp/7040/bin/python2.7
Also creating executable in /tmp/gcp/7040/bin/python
Installing setuptools, pip, wheel...done.
$ /tmp/gcp/7040/bin/pip install --upgrade setuptools pip 
Looking in links: file:///home/tseaver/.pip/wheels
Requirement already up-to-date: setuptools in /tmp/gcp/7040/lib/python2.7/site-packages (40.6.3)
Requirement already up-to-date: pip in /tmp/gcp/7040/lib/python2.7/site-packages (18.1)
$ /tmp/gcp/7040/bin/pip install protobuf==3.5.1
...
Successfully installed protobuf-3.5.1 six-1.12.0
$ /tmp/gcp/7040/bin/pip install google-cloud-kms
...
Successfully installed cachetools-3.0.0 certifi-2018.11.29 chardet-3.0.4 enum34-1.1.6 futures-3.2.0 google-api-core-1.7.0 google-auth-1.6.2 google-cloud-kms-0.2.1 googleapis-common-protos-1.5.5 grpc-google-iam-v1-0.11.4 grpcio-1.17.1 idna-2.8 pyasn1-0.4.5 pyasn1-modules-0.2.3 pytz-2018.7 requests-2.21.0 rsa-4.0 urllib3-1.24.1
$ /tmp/gcp/7040/bin/python -c "import google.cloud.kms; print(google.cloud.kms.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/gcp/7040/lib/python2.7/site-packages/google/cloud/kms.py", line 19, in <module>
    from google.cloud.kms_v1 import KeyManagementServiceClient
  File "/tmp/gcp/7040/lib/python2.7/site-packages/google/cloud/kms_v1/__init__.py", line 19, in <module>
    from google.cloud.kms_v1 import types
  File "/tmp/gcp/7040/lib/python2.7/site-packages/google/cloud/kms_v1/types.py", line 22, in <module>
    from google.api import http_pb2
  File "/tmp/gcp/7040/lib/python2.7/site-packages/google/api/http_pb2.py", line 22, in <module>
    serialized_pb=_b('\n\x15google/api/http.proto\x12\ngoogle.api\"T\n\x04Http\x12#\n\x05rules\x18\x01 \x03(\x0b\x32\x14.google.api.HttpRule\x12\'\n\x1f\x66ully_decode_reserved_expansion\x18\x02 \x01(\x08\"\x81\x02\n\x08HttpRule\x12\x10\n\x08selector\x18\x01 \x01(\t\x12\r\n\x03get\x18\x02 \x01(\tH\x00\x12\r\n\x03put\x18\x03 \x01(\tH\x00\x12\x0e\n\x04post\x18\x04 \x01(\tH\x00\x12\x10\n\x06\x64\x65lete\x18\x05 \x01(\tH\x00\x12\x0f\n\x05patch\x18\x06 \x01(\tH\x00\x12/\n\x06\x63ustom\x18\x08 \x01(\x0b\x32\x1d.google.api.CustomHttpPatternH\x00\x12\x0c\n\x04\x62ody\x18\x07 \x01(\t\x12\x15\n\rresponse_body\x18\x0c \x01(\t\x12\x31\n\x13\x61\x64\x64itional_bindings\x18\x0b \x03(\x0b\x32\x14.google.api.HttpRuleB\t\n\x07pattern\"/\n\x11\x43ustomHttpPattern\x12\x0c\n\x04kind\x18\x01 \x01(\t\x12\x0c\n\x04path\x18\x02 \x01(\tBj\n\x0e\x63om.google.apiB\tHttpProtoP\x01ZAgoogle.golang.org/genproto/googleapis/api/annotations;annotations\xf8\x01\x01\xa2\x02\x04GAPIb\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'serialized_options'
$ /tmp/gcp/7040/bin/pip install --upgrade protobuf==3.6.0
...
Successfully installed protobuf-3.6.0
$ /tmp/gcp/7040/bin/python -c "import google.cloud.kms; print(google.cloud.kms.__file__)"
/tmp/gcp/7040/lib/python2.7/site-packages/google/cloud/kms.pyc

[tseaver]

Looking into it, I believe that googleapis_common_protos-1.5.5 is the culprit: it is the package providing the google/api/http_pb2.py file, but declares only a minimal protobuf >= 3.0.0 dependency.

[crwilcox]

@lukesneeringer released a new googleapis_common_protos==1.5.6 that restricts the versions allowed.

[lukesneeringer]

Just released a googleapis-common-protos 1.5.6 that updates the protobuf dependency to >= 3.6.0.

[crwilcox]

I checked and api_core will pick this up. So this is likely resolved. @wyardley can you confirm?

[lukesneeringer]

It will not retroactively resolve for people who already have 1.5.5 installed (since api-core probably requires >= 1.5.0, and changing _that_ would require a new version of api-core) but it should prevent new folks from getting caught.

[wyardley]

Haven’t confirmed but sounds like this should fix it!

[tseaver]

Trying to reproduce in a fresh Python 2.7 virtual environment:

$ python2.7 -m virtualenv /tmp/gcp/7040
New python executable in /tmp/gcp/7040/bin/python2.7
Also creating executable in /tmp/gcp/7040/bin/python
Installing setuptools, pip, wheel...done.
$ /tmp/gcp/7040/bin/pip install protobuf==3.0.0
...
Successfully installed protobuf-3.0.0 six-1.12.0
$ /tmp/gcp/7040/bin/pip install google-cloud-kms
...
  Found existing installation: protobuf 3.0.0
    Uninstalling protobuf-3.0.0:
      Successfully uninstalled protobuf-3.0.0
Successfully installed cachetools-3.1.0 certifi-2019.3.9 chardet-3.0.4 enum34-1.1.6 futures-3.2.0 google-api-core-1.11.0 google-auth-1.6.3 google-cloud-kms-1.0.0 googleapis-common-protos-1.6.0 grpc-google-iam-v1-0.11.4 grpcio-1.20.1 idna-2.8 protobuf-3.7.1 pyasn1-0.4.5 pyasn1-modules-0.2.5 pytz-2019.1 requests-2.22.0 rsa-4.0 urllib3-1.25.3
$ /tmp/gcp/7040/bin/python -c "from google.cloud import kms_v1; print(kms_v1.__file__)"
/tmp/gcp/7040/lib/python2.7/site-packages/google/cloud/kms_v1/__init__.pyc