Google-cloud-python: pyinstaller executable can't find package info

[mtwu-aviatrix]

The pyinstaller complains the error with the following line:

from gcloud.pubsub.client import Client

The module is installed correctly in /usr/local/lib/python2.7/dist-packages. I run setup.py (build/ install) in addition pip install --upgrade cloud too.

Traceback (most recent call last):
  File "<string>", line 267, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 271, in load_module
    exec(bytecode, module.__dict__)
  File "/home/mtwu/gce/cloudn/cloudx-local/gce/gce_pubsub.py", line 6, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 271, in load_module
    exec(bytecode, module.__dict__)
  File "/home/mtwu/gce/cloudn/install/build/cloudx_cli/out00-PYZ.pyz/gcloud", line 19, in <module>
  File "/home/mtwu/gce/cloudn/install/build/cloudx_cli/out00-PYZ.pyz/pkg_resources", line 551, in get_distribution
The following import will cause the following pyinstaller error.
from gcloud.pubsub.client import Client
  File "/home/mtwu/gce/cloudn/install/build/cloudx_cli/out00-PYZ.pyz/pkg_resources", line 431, in get_provider
  File "/home/mtwu/gce/cloudn/install/build/cloudx_cli/out00-PYZ.pyz/pkg_resources", line 952, in require
  File "/home/mtwu/gce/cloudn/install/build/cloudx_cli/out00-PYZ.pyz/pkg_resources", line 839, in resolve
pkg_resources.DistributionNotFound: The 'gcloud' distribution was not found and is required by the application

mtwu@ubuntu:/usr/local/lib/python2.7/dist-packages$ ls -l gcloud 
total 360
drwxr-sr-x 2 root staff  4096 Oct 16 09:13 bigquery
drwxr-sr-x 3 root staff  4096 Oct 16 09:13 bigtable
-rw-r--r-- 1 root staff  7564 Oct 16 09:13 client.py
-rw-r--r-- 1 root staff  8038 Oct 16 09:13 client.pyc
-rw-r--r-- 1 root staff 12949 Oct 16 09:13 connection.py
-rw-r--r-- 1 root staff 12304 Oct 16 09:13 connection.pyc
-rw-r--r-- 1 root staff 14075 Oct 16 09:13 credentials.py
-rw-r--r-- 1 root staff 13661 Oct 16 09:13 credentials.pyc
drwxr-sr-x 3 root staff  4096 Oct 16 09:13 datastore
-rw-r--r-- 1 root staff  4036 Oct 16 09:13 demo.py
-rw-r--r-- 1 root staff  4010 Oct 16 09:13 demo.pyc
drwxr-sr-x 2 root staff  4096 Oct 16 09:13 dns
-rw-r--r-- 1 root staff  1418 Oct 16 09:13 environment_vars.py
-rw-r--r-- 1 root staff   647 Oct 16 09:13 environment_vars.pyc
-rw-r--r-- 1 root staff  5819 Oct 16 09:13 exceptions.py
-rw-r--r-- 1 root staff  9026 Oct 16 09:13 exceptions.pyc
-rw-r--r-- 1 root staff  8048 Oct 16 09:13 _helpers.py
-rw-r--r-- 1 root staff 10032 Oct 16 09:13 _helpers.pyc
-rw-r--r-- 1 root staff   736 Oct 16 09:13 __init__.py
-rw-r--r-- 1 root staff   333 Oct 16 09:13 __init__.pyc
-rw-r--r-- 1 root staff  4957 Oct 16 09:13 iterator.py
-rw-r--r-- 1 root staff  5210 Oct 16 09:13 iterator.pyc
drwxr-sr-x 2 root staff  4096 Oct 16 09:13 pubsub
drwxr-sr-x 2 root staff  4096 Oct 16 09:13 resource_manager
drwxr-sr-x 2 root staff  4096 Oct 16 09:13 search
drwxr-sr-x 3 root staff  4096 Oct 16 09:13 storage
-rw-r--r-- 1 root staff  7021 Oct 16 09:13 test_client.py
-rw-r--r-- 1 root staff  9101 Oct 16 09:13 test_client.pyc
-rw-r--r-- 1 root staff 13090 Oct 16 09:13 test_connection.py
-rw-r--r-- 1 root staff 15451 Oct 16 09:13 test_connection.pyc
-rw-r--r-- 1 root staff 22622 Oct 16 09:13 test_credentials.py
-rw-r--r-- 1 root staff 28116 Oct 16 09:13 test_credentials.pyc
-rw-r--r-- 1 root staff  2844 Oct 16 09:13 test_exceptions.py
-rw-r--r-- 1 root staff  3664 Oct 16 09:13 test_exceptions.pyc
-rw-r--r-- 1 root staff 12177 Oct 16 09:13 test__helpers.py
-rw-r--r-- 1 root staff 19874 Oct 16 09:13 test__helpers.pyc
-rw-r--r-- 1 root staff  1143 Oct 16 09:13 _testing.py
-rw-r--r-- 1 root staff  1212 Oct 16 09:13 _testing.pyc
-rw-r--r-- 1 root staff  6781 Oct 16 09:13 test_iterator.py
-rw-r--r-- 1 root staff  8076 Oct 16 09:13 test_iterator.pyc

[dhermes]

I'm not familiar with PyInstaller (I presume it is http://www.pyinstaller.org/).

Can you elaborate and maybe explain how you expect this to work?

The problem you describe seems to be a bug in part of PyInstaller. It is a chicken-and-egg problem. The get_distribution() check occurs only after the library is installed. It is used to set the __version__.

[mtwu-aviatrix]

We have installed quite a few new modules from github lately. This is the first time we bump into the executable build issue. We do expect the module can work with pyinstaller after it is built and installed. I can run the client in the python native mode. We have reinstalled the module and pyinstaller few times without any luck.

[mtwu-aviatrix]

This issue should be related to the version compatibility between pyinstaller and setuptools. Please advise the correct versions used in your environment which the executable can be built from pyinstaller.

[dhermes]

We don't have any version requirements on setuptools. The latest version should work just fine.

[mtwu-aviatrix]

Your code has to consider the run-time environment when it is bundled to an executable. In this case, pkg_resources can't work in the bundled executable though it works in native python mode. The following workaround diffs make the executable work.

diff --git a/gcloud/__init__.py b/gcloud/__init__.py
index 8c1c7b4..e70010a 100644
--- a/gcloud/__init__.py
+++ b/gcloud/__init__.py
@@ -16,4 +16,5 @@

 from pkg_resources import get_distribution

-__version__ = get_distribution('gcloud').version
+#__version__ = get_distribution('gcloud').version
+__version__ = '0.8.0'
diff --git a/gcloud/connection.py b/gcloud/connection.py
index b732113..7650625 100644
--- a/gcloud/connection.py
+++ b/gcloud/connection.py
@@ -63,7 +63,8 @@ class Connection(object):
     :param http: An optional HTTP object to make requests.
     """

-    USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
+#    USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
+    USER_AGENT = "gcloud-python/{0}".format('0.8.0')
     """The user agent for gcloud-python requests."""

     SCOPE = None

[dhermes]

If we were to hardcode it we'd replace get_distribution('gcloud').version with __version__ in gcloud/__init__.py.

However, we intentionally haven't hardcoded it so we can rely on setuptools (a great part of the Python ecosystem) to get the version from the package info.

It seems strange to me that a bundler would ignore setuptools, a core part of Python packaging. When was the bundler last updated? setuptools has been the de facto way to distribute packages for years now (@tseaver probably has a better sense of this than I do).

[mtwu-aviatrix]

I am not sure if the setuptools can help much in this case. An executable has no visibility of the package info. At the setup time, it is a different story. I have used pyinstaller 2.x and 3.0 and they all ended up the same run-time error.

[dhermes]

Yes, I'm saying a binary installer worth it's salt should work seemlessly with setuptools, even while running the packaged executable.

We may just do a try ... except. It seems pkg_resources imports just fine in your packaged environment but none of the package info. ISTM that pkg_resources just searches sys.path / PYTHONPATH until it finds the distribution

>>> import pprint
>>> from pkg_resources import get_distribution
>>> distro = get_distribution('gcloud')
>>> pprint.pprint(distro.__dict__)
{'_key': 'gcloud',
 '_provider': <pkg_resources.PathMetadata instance at 0x7f8961021200>,
 '_version': '0.7.1',
 'location': '/usr/local/lib/python2.7/dist-packages',
 'platform': None,
 'precedence': -1,
 'project_name': 'gcloud',
 'py_version': None}
>>> pprint.pprint(distro._provider.__dict__)
{'egg_info': '/usr/local/lib/python2.7/dist-packages/gcloud-0.7.1.dist-info',
 'module_path': '/usr/local/lib/python2.7/dist-packages'}

[mtwu-aviatrix]

Have you tried to bundle gcloud to an executable file and run the file?

In the native python interpreter, it always works. Once it is in an executable, can the external path search work?

[dhermes]

No I haven't tried it. I've not had a use for one of these installers and am trying to understand how they work, who uses them and where they use them.

[mtwu-aviatrix]

We build an executable file to support our solution.

[dhermes]

@mtwu-aviatrix I am trying to reproduce this error with PyInstaller and am not having luck.

I made a bogus file (bogus.py):

from gcloud import datastore
print(datastore)

and then ran

$ pyinstaller bogus.py

without any issues. What am I missing?

[dhermes]

@mtwu-aviatrix Closing this out since I can't reproduce. Feel free to reopen if it's still an issue.

[hu7241]

@dhermes Did you run the packaged executable? pyinstaller bogus.py went without problem. But at run time it spits error:

File "/tmp/pip-build-JhwLSY/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "gcloud/__init__.py", line 19, in <module>
  File "pkg_resources/__init__.py", line 559, in get_distribution
  File "pkg_resources/__init__.py", line 433, in get_provider
  File "pkg_resources/__init__.py", line 970, in require
  File "pkg_resources/__init__.py", line 856, in resolve
pkg_resources.DistributionNotFound: The 'gcloud' distribution was not found and is required by the application

I too have this problem in my project.

[dhermes]

@hu7241

1. Can you give me explicit instructions on what to run so I can reproduce?
2. We no longer distribute the gcloud package, it is no google.cloud. Please update

[hu7241]

In the root folder of bogus.py two directories (build, dist) are created after executing

$ pyinstaller bogus.py

execute the binary by

$ cd /dist/bogus && ./bogus

and the said error should pop up.

I updated to google-cloud as instructed, the problem persists:

  Traceback (most recent call last):
  File "bogus.py", line 1, in <module>
  File "/tmp/pip-build-JhwLSY/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "google/cloud/datastore/__init__.py", line 55, in <module>
  File "/tmp/pip-build-JhwLSY/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "google/cloud/datastore/connection.py", line 23, in <module>
  File "/tmp/pip-build-JhwLSY/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "google/cloud/connection.py", line 31, in <module>
  File "pkg_resources/__init__.py", line 559, in get_distribution
  File "pkg_resources/__init__.py", line 433, in get_provider
  File "pkg_resources/__init__.py", line 970, in require
  File "pkg_resources/__init__.py", line 856, in resolve
pkg_resources.DistributionNotFound: The 'google-cloud-core' distribution was not found and is required by the application
Failed to execute script bogus

As @mtwu-aviatrix commented on Oct 29, 2015, the native interpreter has no problem with this import. The problem arises when it has been bundled by Pyinstaller (ver. 3.2).

[jessesuen]

You will need to create a pyinstaller hook for packages that depend on setuptools' get_distribution() to obtain metadata information about the package.

Create a hook file in some hooks directory. The file name should be hook-<modulename>.py for pyinstaller to find it properly (i.e. hook-gcloud.py).

The content will be:

from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata('google-cloud-core')

The hooks directory should be set in the Analysis phase of your pyinstaller .spec file:

a = Analysis(['bogus.py'],
             pathex=[],
             binaries=[],
             hiddenimports=[],
             hookspath="/path/to/your/hooks/dir",
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)

I haven't tested the above, but this is what worked for me for another python package which is doing something similar (APScheduler).

If and when you get this working, consider contributing the hook back to pyinstaller project.

[dhermes]

Thanks @jessesuen!

[HoangGiang88]

Hi I have create hook file but can't work

Traceback (most recent call last):
  File "syn_start.py", line 1, in <module>
  File "c:\users\titan\appdata\local\temp\pip-build-qocd5q\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "syn_firebase\firebase.py", line 9, in <module>
  File "c:\users\titan\appdata\local\temp\pip-build-qocd5q\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "site-packages\pyrebase\__init__.py", line 1, in <module>
  File "c:\users\titan\appdata\local\temp\pip-build-qocd5q\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "site-packages\pyrebase\pyrebase.py", line 18, in <module>
  File "c:\users\titan\appdata\local\temp\pip-build-qocd5q\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
  File "site-packages\gcloud\__init__.py", line 19, in <module>
  File "site-packages\pkg_resources\__init__.py", line 557, in get_distribution
  File "site-packages\pkg_resources\__init__.py", line 431, in get_provider
  File "site-packages\pkg_resources\__init__.py", line 973, in require
  File "site-packages\pkg_resources\__init__.py", line 859, in resolve
pkg_resources.DistributionNotFound: The 'gcloud' distribution was not found and is required by the application
Failed to execute script syn_start

[Shoshin23]

Hi,
I'm still facing this issue while trying to create an executable out of my current python programme that uses cloud vision but am unable to do so as i get this very same error. Any workarounds?

[srkn]

@Shoshin23 I solved this problem by editing pyinstaller original hook-google.cloud.py file. I simply added this line, you can change it according your needs
datas += copy_metadata('google-api-core')

[Vonisoa]

@Shoshin23 , I create the file hook-google.cloud.py and add this :
datas += copy_metadata('google-api-core')
but I get the following error
PyInstaller.compat.FileNotFoundError: Hook directory "C:\hook-google.cloud.py" not found.
Can someone help with this?

[srkn]

@Vonisoa I believe you mentioned me, you need to change original hook file, no need to create a new hook file. You can find original hook file in site-packages/PyInstaller/hooks or dist-packages/PyInstaller/hooks folder according to your OS.

[Vonisoa]

@srkn , thanks for your answer, it solve my problem. Now I have another trouble when running the .exe

Exception in 'grpc._cython.cygrpc.ssl_roots_override_callback' ignored E0731 16:52:01.684000000 13212 src/core/lib/security/security_connector/security _connector.cc:1173] assertion failed: pem_root_certs != nullptr
I don't know what can produce this error!!!

[srkn]

@Vonisoa You need to set "GRPC_DEFAULT_SSL_ROOTS_FILE_PATH" environ variable in your code. Check this out https://github.com/grpc/grpc/blob/master/doc/environment_variables.md . I cant find right now but @dhermes coded a gist that helps you to find ssl root file in your OS.

[bloxofcodes]

Hi doing datas += copy_metadata('google-api-core') doesnt solve my error about gcloud...what particular steps you do to make it work?

[Vonisoa]

@bloxofcodes Maybe you should look at C:\Python27\Lib\site-packages\PyInstaller\hooks or somewhere else according to your OS. And find if there is a hook for hook-.py, so in your casehook-google.cloud.py. If it exists but pyinstaller can't find it, you should import it in your code like this:
import google.cloud,
but if it doesn't exists, create this hook in the hook directory of Pyinstaller and add this inside:
from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata('google-cloud-core')

[Vonisoa]

@srkn , thanks for your advice, I will try this and give feedback!

[Rahuld753]

You will need to create a pyinstaller hook for packages that depend on setuptools' get_distribution() to obtain metadata information about the package.

Create a hook file in some hooks directory. The file name should be hook-<modulename>.py for pyinstaller to find it properly (i.e. hook-gcloud.py).

The content will be:

from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata('gcloud')

The hooks directory should be set in the Analysis phase of your pyinstaller .spec file:

a = Analysis(['bogus.py'],
             pathex=[],
             binaries=[],
             hiddenimports=[],
             hookspath="/path/to/your/hooks/dir",
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)

I haven't tested the above, but this is what worked for me for another python package which is doing something similar (APScheduler).

If and when you get this working, consider contributing the hook back to pyinstaller project.

[murtuzamdahod]

@Shoshin23 I solved this problem by editing pyinstaller original hook-google.cloud.py file. I simply added this line, you can change it according your needs
datas += copy_metadata('google-api-core')

Thanks alot this worked for me :)

[murtuzamdahod]

@srkn , thanks for your answer, it solve my problem. Now I have another trouble when running the .exe

Exception in 'grpc._cython.cygrpc.ssl_roots_override_callback' ignored E0731 16:52:01.684000000 13212 src/core/lib/security/security_connector/security _connector.cc:1173] assertion failed: pem_root_certs != nullptr
I don't know what can produce this error!!!

first of all find the root.pem file from gcrp package then copy the file to your cwd and build using --add-data "root.pem:/grpc/_cython/_credentials/"