Pyinstaller: Calling `pkg_resources.get_distribution` at runtime never works

[marioortizmanero]

Description of the issue

I have a package which uses pkg_resources.get_distribution for dependency injection reasons, to check if a dependency installed. This example always works when using python test.py:

from pkg_resources import *

# Importing doesn't throw errors
import numpy
import pandas
import requests

# Trying to check if they are installed at runtime doesn't work
for mod in ('numpy', 'pandas', 'requests'):
    try:
        print(get_distribution(mod))
    except DistributionNotFound as e:
        print(mod, "not found:", e)

but when using Pyinstaller, DistributionNotFound is always raised. Here's a run with docker:

❯ sudo docker run -v $PWD:/test python:3.6 /bin/sh -c 'pip install numpy pandas requests pyins
taller && python test/test.py && pyinstaller test/test.py --hidden-import pkg_resources.py2_warn &&
./dist/test/test'
Collecting numpy
  Downloading numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl (20.2 MB)
Collecting pandas
  Downloading pandas-1.0.3-cp36-cp36m-manylinux1_x86_64.whl (10.0 MB)
Collecting requests
  Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting pyinstaller
  Downloading PyInstaller-3.6.tar.gz (3.5 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting python-dateutil>=2.6.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2017.2
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Collecting altgraph
  Downloading altgraph-0.17-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from pyinstalle
r) (46.1.3)
Collecting six>=1.5
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: pyinstaller
  Building wheel for pyinstaller (PEP 517): started
  Building wheel for pyinstaller (PEP 517): finished with status 'done'
  Created wheel for pyinstaller: filename=PyInstaller-3.6-py3-none-any.whl size=2926545 sha256=a9619
70520e28422d1dfe16cf59012d87b0b7bf24dcd92b55f19eb1c814e0a7c
  Stored in directory: /root/.cache/pip/wheels/2a/8b/ca/d508e27e78c9756556b9a5090889efda5e632d8c92d9
af8820
Successfully built pyinstaller
Installing collected packages: numpy, six, python-dateutil, pytz, pandas, chardet, certifi, idna, ur
llib3, requests, altgraph, pyinstaller
Successfully installed altgraph-0.17 certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 numpy-1.18.2 pandas-1
.0.3 pyinstaller-3.6 python-dateutil-2.8.1 pytz-2019.3 requests-2.23.0 six-1.14.0 urllib3-1.25.8
numpy 1.18.2
pandas 1.0.3
requests 2.23.0
38 INFO: PyInstaller: 3.6
38 INFO: Python: 3.6.10
39 INFO: Platform: Linux-5.6.2-arch1-2-x86_64-with-debian-10.3
39 INFO: wrote /test.spec
40 INFO: UPX is not available.
41 INFO: Extending PYTHONPATH with paths
['/test', '/']
41 INFO: checking Analysis
42 INFO: Building Analysis because Analysis-00.toc is non existent
42 INFO: Initializing module dependency graph...
43 INFO: Caching module graph hooks...
46 INFO: Analyzing base_library.zip ...
2428 INFO: Caching module dependency graph...
2493 INFO: running Analysis Analysis-00.toc
2504 INFO: Analyzing /test/test.py
2619 INFO: Processing pre-find module path hook   distutils
2620 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.6'
3710 INFO: Processing pre-safe import module hook   setuptools.extern.six.moves
4008 INFO: Processing pre-find module path hook   site
4009 INFO: site: retargeting to fake-dir '/usr/local/lib/python3.6/site-packages/PyInstaller/fake-mo
dules'
7300 INFO: Processing pre-safe import module hook   six.moves
9510 INFO: Processing pre-safe import module hook   urllib3.packages.six.moves
10364 INFO: Analyzing hidden import 'pkg_resources.py2_warn'
10364 INFO: Processing module hooks...
10364 INFO: Loading module hook "hook-certifi.py"...
10365 INFO: Loading module hook "hook-pydoc.py"...
10365 INFO: Loading module hook "hook-numpy.core.py"...
10434 INFO: Loading module hook "hook-numpy.py"...
10435 INFO: Loading module hook "hook-sysconfig.py"...
10441 INFO: Loading module hook "hook-sqlite3.py"...
10481 INFO: Loading module hook "hook-xml.py"...
10542 INFO: Loading module hook "hook-pkg_resources.py"...
10858 INFO: Processing pre-safe import module hook   win32com
10861 INFO: Excluding import '__main__'
10862 INFO:   Removing import of __main__ from module pkg_resources
10862 INFO: Loading module hook "hook-lib2to3.py"...
10863 INFO: Loading module hook "hook-encodings.py"...
10911 INFO: Loading module hook "hook-distutils.py"...
10912 INFO: Loading module hook "hook-pandas.py"...
11455 INFO: Loading module hook "hook-pytz.py"...
11470 INFO: Loading module hook "hook-setuptools.py"...
11914 INFO: Looking for ctypes DLLs
11979 WARNING: library user32 required via ctypes not found
12028 WARNING: library msvcrt required via ctypes not found
12039 INFO: Analyzing run-time hooks ...
12049 INFO: Including run-time hook 'pyi_rth_certifi.py'
12050 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
12053 INFO: Including run-time hook 'pyi_rth_pkgres.py'
12068 INFO: Looking for dynamic libraries
13164 INFO: Looking for eggs
13164 INFO: Using Python library /usr/local/lib/libpython3.6m.so.1.0
13181 INFO: Warnings written to /build/test/warn-test.txt
13263 INFO: Graph cross-reference written to /build/test/xref-test.html
13298 INFO: checking PYZ
13298 INFO: Building PYZ because PYZ-00.toc is non existent
13298 INFO: Building PYZ (ZlibArchive) /build/test/PYZ-00.pyz
14297 INFO: Building PYZ (ZlibArchive) /build/test/PYZ-00.pyz completed successfully.
14313 INFO: checking PKG
14313 INFO: Building PKG because PKG-00.toc is non existent
14313 INFO: Building PKG (CArchive) PKG-00.pkg
14331 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
14332 INFO: Bootloader /usr/local/lib/python3.6/site-packages/PyInstaller/bootloader/Linux-64bit/run
14332 INFO: checking EXE
14332 INFO: Building EXE because EXE-00.toc is non existent
14333 INFO: Building EXE from EXE-00.toc
14333 INFO: Appending archive to ELF section in EXE /build/test/test
14344 INFO: Building EXE from EXE-00.toc completed successfully.
14346 INFO: checking COLLECT
14346 INFO: Building COLLECT because COLLECT-00.toc is non existent
14347 INFO: Building COLLECT COLLECT-00.toc
14563 INFO: Building COLLECT COLLECT-00.toc completed successfully.
numpy not found: The 'numpy' distribution was not found and is required by the application
pandas not found: The 'pandas' distribution was not found and is required by the application
requests not found: The 'requests' distribution was not found and is required by the application

This is what happens when using python as usual:

numpy 1.18.2
pandas 1.0.3
requests 2.23.0

And this was PyInstaller's version:

numpy not found: The 'numpy' distribution was not found and is required by the application
pandas not found: The 'pandas' distribution was not found and is required by the application
requests not found: The 'requests' distribution was not found and is required by the application

Context information (for bug reports)

• Output of pyinstaller --version: 3.6
• Version of Python: 3.6
• Platform: Debian Buster, default settings from docker container

• Did you also try this on another platform? Does it work there? Same on Arch Linux

• try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

Doesn't work either. Tried:

❯ sudo docker run -v $PWD:/test python:3.6 /bin/sh -c 'pip install numpy pandas requests && python test/test.py && pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip && pyinstaller test/test.py --hidden-import pkg_resources.py2_warn && ./dist/test/test && pyinstaller --version'

• follow all the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

• [X] start with celan installation
• [X] use the latest development version
• [X] Run your frozen program from a command window (shell) — instead of double-clicking on it
• [X] Package your program in --onedir mode
• [X] Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• [X] Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

Now, the thing is that I suspect this is actually expected behavior. What would be the best way to overcome this? Does PyInstaller offer any fixes to know if a module is installed at runtime?

Thanks in advance

[Legorooj]

@marioortizmanero thanks for a detailed report. It's a nice change.

This is a sort of expected behaviour. Because of how PyInstaller sets up the environment, it can mess up standard import tracing.

What would be the best way to overcome this? Does PyInstaller offer any fixes to know if a module is installed at runtime?

Kind of. If you want to make sure the module is imported when building, add it to hiddenimports. (--hidden-import=numpy). PyInstaller will then collect numpy as if you'd imported it in your script.

But no you can't check at runtime as far as I know, without using plain old try-except statements.

[marioortizmanero]

Okay, I'll have to override get_distribution before building or find a better solution than this function.

Thank you! Have a nice day

[bjones1]

By default, PyInstaller doesn't include metadata. You need a hook to include this -- see copy_metadata under https://pyinstaller.readthedocs.io/en/stable/hooks.html#useful-items-in-pyinstaller-utils-hooks.