Google-cloud-python: Datastore: No module named 'grpc' when importing datastore if installed after bigquery

[thpica]

Summary

When installing google-cloud-datastore after google-cloud-bigquery, datastore is missing grpc.
This is most likely due to the fact that google-cloud-bigquery installs google-cloud-core without the grpc extra.
So when google-cloud-datastore (which requires google-cloud-core[grpc]) is installed afterwards, the google-cloud-core dependency is already satisfied and pip does not see the missing extra.
This only happens when the packages are installed in the same pip install invocation (including when using requirements.txt).

Installing them in the reverse order fixes the issue but this is counter-intuitive.
While this seems related to https://github.com/pypa/pip/issues/988 I think this could be avoided by always requiring grpc in google-cloud-core.

Steps to reproduce:

$ virtualenv test
# ...
$ source test/bin/activate
$ pip install google-cloud-bigquery google-cloud-datastore
# ...
$ python -c "from google.cloud import datastore"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/cloud/datastore/__init__.py", line 61, in <module>
    from google.cloud.datastore.batch import Batch
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/cloud/datastore_v1/__init__.py", line 18, in <module>
    from google.cloud.datastore_v1.gapic import datastore_client
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 18, in <module>
    import google.api_core.gapic_v1.client_info
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/api_core/gapic_v1/__init__.py", line 16, in <module>
    from google.api_core.gapic_v1 import config
  File "/home/user/Desktop/test/lib/python3.6/site-packages/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
ModuleNotFoundError: No module named 'grpc'

Additionnal information

$ python --version
Python 3.6.3

pip --version
pip 18.1 from /home/user/Desktop/test/lib/python3.6/site-packages/pip (python 3.6)

$ pip freeze
cachetools==2.1.0
certifi==2018.10.15
chardet==3.0.4
google-api-core==1.5.0
google-auth==1.5.1
google-cloud-bigquery==1.6.0
google-cloud-core==0.28.1
google-cloud-datastore==1.7.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
idna==2.7
protobuf==3.6.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pytz==2018.5
requests==2.19.1
rsa==4.0
six==1.11.0
urllib3==1.23

[tseaver]

Duplicate of #5776. As noted there:

this is caused by pypa/pip#4957.

I closed that issue "wontfix":

because duplicating the grpc pins everywhere creates roach motels. The workaround is to include grpcio explicitly in the application's requirements.txt

[conlan]

praise the sun! was debugging this issue for over a day (using datastore + bigquery too) and

grpcio explicitly in the application's requirements.txt

solved it for me.