Google-cloud-python: ModuleNotFoundError: No module named 'google.api_core'

[hugovk]

macOS Sierra
Python 3.6.3

[hugo:/tmp] % pip3 install --upgrade google-cloud-bigquery
Requirement already up-to-date: google-cloud-bigquery in /usr/local/lib/python3.6/site-packages
Requirement already up-to-date: google-resumable-media>=0.2.1 in /usr/local/lib/python3.6/site-packages (from google-cloud-bigquery)
Requirement already up-to-date: google-auth>=1.0.0 in /usr/local/lib/python3.6/site-packages (from google-cloud-bigquery)
Requirement already up-to-date: google-api-core<0.2.0dev,>=0.1.1 in /usr/local/lib/python3.6/site-packages (from google-cloud-bigquery)
Requirement already up-to-date: requests>=2.18.0 in /usr/local/lib/python3.6/site-packages (from google-cloud-bigquery)
Requirement already up-to-date: google-cloud-core<0.29dev,>=0.28.0 in /usr/local/lib/python3.6/site-packages (from google-cloud-bigquery)
Requirement already up-to-date: six in /usr/local/lib/python3.6/site-packages (from google-resumable-media>=0.2.1->google-cloud-bigquery)
Requirement already up-to-date: cachetools>=2.0.0 in /usr/local/lib/python3.6/site-packages/cachetools-2.0.1-py3.6.egg (from google-auth>=1.0.0->google-cloud-bigquery)
Requirement already up-to-date: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.6/site-packages/pyasn1_modules-0.1.5-py3.6.egg (from google-auth>=1.0.0->google-cloud-bigquery)
Requirement already up-to-date: rsa>=3.1.4 in /usr/local/lib/python3.6/site-packages/rsa-3.4.2-py3.6.egg (from google-auth>=1.0.0->google-cloud-bigquery)
Requirement already up-to-date: pyasn1>=0.1.7 in /usr/local/lib/python3.6/site-packages/pyasn1-0.3.7-py3.6.egg (from google-auth>=1.0.0->google-cloud-bigquery)
Requirement already up-to-date: googleapis-common-protos<2.0dev,>=1.5.3 in /usr/local/lib/python3.6/site-packages/googleapis_common_protos-1.5.3-py3.6.egg (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
Requirement already up-to-date: setuptools>=34.0.0 in /usr/local/lib/python3.6/site-packages (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
Requirement already up-to-date: protobuf>=3.0.0 in /usr/local/lib/python3.6/site-packages/protobuf-3.4.0-py3.6.egg (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
Requirement already up-to-date: idna<2.7,>=2.5 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->google-cloud-bigquery)
Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->google-cloud-bigquery)
Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->google-cloud-bigquery)
Requirement already up-to-date: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests>=2.18.0->google-cloud-bigquery)

[hugo:/tmp] 2s % pip3 freeze | grep google
google-api-core==0.1.1
google-auth==1.2.1
google-cloud-bigquery==0.28.0
google-cloud-core==0.28.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

Steps to reproduce:

Python 3.6.3 (default, Oct  4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from google.cloud import bigquery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/bigquery/__init__.py", line 34, in <module>
    from google.cloud.bigquery._helpers import DEFAULT_RETRY
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/bigquery/_helpers.py", line 23, in <module>
    from google.api_core import retry
ModuleNotFoundError: No module named 'google.api_core'
>>>

[hugovk]

The good news is it works under a virtualenv with the same Google libraries, so it's my local environment.

[hugo:/tmp] % python3 -m venv venv

[hugo:/tmp] 1m4s % source venv/bin/activate

[hugo:/tmp] [venv] % venv/bin/python --version
Python 3.6.3

⌂65% [hugo:/tmp] [venv] 5s % venv/bin/pip --version
pip 9.0.1 from /private/tmp/venv/lib/python3.6/site-packages (python 3.6)

[hugo:/tmp] [venv] 8s % venv/bin/pip install --upgrade google-cloud-bigquery
Collecting google-cloud-bigquery
  Using cached google_cloud_bigquery-0.28.0-py2.py3-none-any.whl
Collecting requests>=2.18.0 (from google-cloud-bigquery)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud-bigquery)
  Using cached google_cloud_core-0.28.0-py2.py3-none-any.whl
Collecting google-api-core<0.2.0dev,>=0.1.1 (from google-cloud-bigquery)
  Using cached google_api_core-0.1.1-py2.py3-none-any.whl
Collecting google-auth>=1.0.0 (from google-cloud-bigquery)
  Using cached google_auth-1.2.1-py2.py3-none-any.whl
Collecting google-resumable-media>=0.2.1 (from google-cloud-bigquery)
  Using cached google_resumable_media-0.3.1-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests>=2.18.0->google-cloud-bigquery)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.18.0->google-cloud-bigquery)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.18.0->google-cloud-bigquery)
  Using cached certifi-2017.11.5-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests>=2.18.0->google-cloud-bigquery)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting protobuf>=3.0.0 (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
  Using cached protobuf-3.4.0-py2.py3-none-any.whl
Collecting setuptools>=34.0.0 (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
  Using cached setuptools-36.7.2-py2.py3-none-any.whl
Collecting six>=1.10.0 (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.3 (from google-api-core<0.2.0dev,>=0.1.1->google-cloud-bigquery)
  Using cached googleapis-common-protos-1.5.3.tar.gz
Collecting cachetools>=2.0.0 (from google-auth>=1.0.0->google-cloud-bigquery)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from google-auth>=1.0.0->google-cloud-bigquery)
  Using cached pyasn1_modules-0.1.5-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from google-auth>=1.0.0->google-cloud-bigquery)
  Using cached pyasn1-0.3.7-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from google-auth>=1.0.0->google-cloud-bigquery)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Installing collected packages: urllib3, chardet, certifi, idna, requests, setuptools, six, protobuf, cachetools, pyasn1, pyasn1-modules, rsa, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-resumable-media, google-cloud-bigquery
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
  Running setup.py install for googleapis-common-protos ... done
Successfully installed cachetools-2.0.1 certifi-2017.11.5 chardet-3.0.4 google-api-core-0.1.1 google-auth-1.2.1 google-cloud-bigquery-0.28.0 google-cloud-core-0.28.0 google-resumable-media-0.3.1 googleapis-common-protos-1.5.3 idna-2.6 protobuf-3.4.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 requests-2.18.4 rsa-3.4.2 setuptools-36.7.2 six-1.11.0 urllib3-1.22

[hugo:/tmp] [venv] 5m24s % venv/bin/python
Python 3.6.3 (default, Oct  4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import bigquery
>>> exit()

[hugo:/tmp] [venv] 26s 1 % venv/bin/pip freeze | grep google
google-api-core==0.1.1
google-auth==1.2.1
google-cloud-bigquery==0.28.0
google-cloud-core==0.28.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

[dhermes]

@hugovk It seems you have a packaging issue. There are a few typical explanations, so let's try those first:

• Your python binary is not the one associated with pip. Can you use /usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/python3 -m pip rather than pip3?
• You have a stray or missing .pth file in your site-packages
• You have installed a namespace package with setup.py install rather than pip (which is problematic). This will leave the file easy-install.pth in your site-packages

[dhermes]

The good news is it works under a virtualenv

That is good news!

If you don't mind, I'd like to close this issue (for tracking reasons). Our packaging setup is "working as intended", so there is nothing to change on our end (as far as I know).

However, the pain of dealing with namespace packages is a bit our responsibility, so I'm happy to keep discussing here for as long as needed.

[hugovk]

Got it working, through the principle of turning it off and on again.

• Did brew cleanup to remove old Pythons 3.4 and 3.5, no help.
• Did brew uninstall python3 and brew install python3 to de/re-install Python 3.6, no help.
• Did brew uninstall python3, deleted /usr/local/lib/python3.4, /usr/local/lib/python3.5 and /usr/local/lib/python3.6, then brew install python3, bingo!

Cheers!

[dhermes]

Thanks for the follow up!

[j3pic]

Adding the following line to my project's requirements.txt allowed google-cloud-bigquery to load:

google-api-core==1.14.2

If a PyPI package depends on another one, that dependency should be listed in the first package's requirements.txt so that the dependency gets installed. It should not be the responsibility of downstream apps to figure out what your library's dependencies are.

[j3pic]

Actually, all the above does is allow the pip3 install command to succeed. The resulting installation suffers from https://github.com/googleapis/google-cloud-python/issues/8268. I posted the solution in the comments of that bug (google-cloud-core==1.0.3, not google-api-core).

The documentation at https://googleapis.dev/python/bigquery/latest/index.html claims that all you have to do to get this library installed is tell pip to install it. Clearly, this is not actually the case.