Pyinstaller: Building Fails

[phirst]

Description of the issue

Building fails.

Context information (for bug reports)

• Output of pyinstaller --version: 4.0.dev0
• Version of Python: vanilla 3.7.6
• Platform: Catalina 10.15.6
• Did you also try this on another platform? Does it work there?

Works on Windows 10
Worked prior to upgrade to 10.15.6

• [X] start with clean installation
• [X] use the latest development version
• [ ] Run your frozen program from a command window (shell) — instead of double-clicking on it
• [X] Package your program in --onedir mode
• [X] Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• [ ] Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

Doesn't build to get this far...

Stacktrace / full error message

Traceback (most recent call last):
  File "/Users/my.user.account/.pyenv/versions/buildpy/bin/PyInstaller", line 8, in <module>
    sys.exit(run())
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "myscript.spec", line 84, in <module>
    win_private_assemblies=False,
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 244, in __init__
    self.__postinit__()
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
    self.assemble()
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 422, in assemble
    self.graph.process_post_graph_hooks()
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/depend/analysis.py", line 311, in process_post_graph_hooks
    module_hook.post_graph()
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/depend/imphook.py", line 417, in post_graph
    self._load_hook_module()
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/depend/imphook.py", line 384, in _load_hook_module
    self.hook_module_name, self.hook_filename)
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/compat.py", line 797, in importlib_load_source
    return mod_loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 907, in load_module
  File "<frozen importlib._bootstrap_external>", line 732, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/hooks/hook-sysconfig.py", line 32, in <module>
    datas = [(_CONFIG_H, relpath_to_config_or_make(_CONFIG_H))]
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/utils/hooks/__init__.py", line 807, in relpath_to_config_or_make
    prefix = _find_prefix(filename)
  File "/Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/utils/hooks/__init__.py", line 796, in _find_prefix
    return possible_prefixes[0]
IndexError: list index out of range

[bwoodsend]

@phirst We need some more background information than this.

• What command did you use to build?
• Do you get this issue if you build any Python script or just a particular one?

[phirst]

Command used to Build: PyInstaller -F --clean myscript.spec
Just this App or Others: All that I've tried (3 or 4)

Here's the .spec file:

import gooey
import os

if os.name == 'nt':
    iconFile = os.path.join('assets', 'program_icon.ico')
    oneFileName = 'myscript.exe'

else:
    iconFile = os.path.join('assets', 'program_icon.icns')
    oneFileName = 'myscript.app'

gooey_root = os.path.dirname(gooey.__file__)
gooey_languages = Tree(os.path.join(gooey_root, 'languages'), prefix = 'gooey/languages')
gooey_images = Tree(os.path.join(gooey_root, 'images'), prefix = 'gooey/images')
assetFiles = [
  ('./assets/*.png','assets/'),
  ('./assets/*.ico','assets/'),
  ('./assets/*.cfg','assets/'),
  ('./assets/*.hqx','assets/'),
  ('./assets/*.icns','assets/')
]

hiddenimports=['pkg_resources.py2_warn']

a = Analysis(['myscript.py'],
             hiddenimports=hiddenimports,
             hookspath=None,
             runtime_hooks=None,
             excludes=['matplotlib','pipreqs','pipreq','pylint','pipenv'],
             datas = assetFiles,
             win_no_prefer_redirects=False,
             win_private_assemblies=False
             )

pyz = PYZ(a.pure,cipher=None)

options = [('u', None, 'OPTION')]

exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          options,
          gooey_languages, # Add them in to collected files
          gooey_images, # Same here.
          name=oneFileName,
          debug=False,
          strip=None,
          upx=True,
          console=True,
          icon=iconFile)

I've also rolled back to PyInstaller 3.6 without any difference.

[bwoodsend]

Hmm, something has as moved. Can you run the following and paste the output back here?

from PyInstaller.utils.hooks import base_prefix
print(base_prefix)
import sys
print(sys.prefix)
from PyInstaller import __file__ as pyi_root
print(pyi_root)

[phirst]

Python 3.7.6 (default, Jul 21 2020, 15:29:33) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyInstaller.utils.hooks import base_prefix
>>> print(base_prefix)
/Users/my.user.name/.pyenv/versions/3.7.6/Python.framework/Versions/3.7
>>> import sys
>>> print(sys.prefix)
/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy
>>> from PyInstaller import __file__ as pyi_root
>>> print(pyi_root)
/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/__init__.py

[bwoodsend]

Could you inject a print statement into your source code.
At the top of the _find_prefix() function in /Users/my.user.account/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/utils/hooks/__init__.py put print("filename =", filename). Then re-run the build.

[phirst]

filename = /Users/my.user.name/.pyenv/versions/3.7.6/include/python3.7m/pyconfig.h
Traceback (most recent call last):
  File "/Users/my.user.name/.pyenv/versions/buildpy/bin/PyInstaller", line 8, in <module>
    sys.exit(run())
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "myscript.spec", line 32, in <module>
    win_private_assemblies=False
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 244, in __init__
    self.__postinit__()
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
    self.assemble()
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 422, in assemble
    self.graph.process_post_graph_hooks()
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/depend/analysis.py", line 311, in process_post_graph_hooks
    module_hook.post_graph()
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/depend/imphook.py", line 417, in post_graph
    self._load_hook_module()
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/depend/imphook.py", line 384, in _load_hook_module
    self.hook_module_name, self.hook_filename)
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/compat.py", line 797, in importlib_load_source
    return mod_loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 907, in load_module
  File "<frozen importlib._bootstrap_external>", line 732, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/hooks/hook-sysconfig.py", line 32, in <module>
    datas = [(_CONFIG_H, relpath_to_config_or_make(_CONFIG_H))]
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/utils/hooks/__init__.py", line 808, in relpath_to_config_or_make
    prefix = _find_prefix(filename)
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/lib/python3.7/site-packages/PyInstaller/utils/hooks/__init__.py", line 797, in _find_prefix
    return possible_prefixes[0]
IndexError: list index out of range

[phirst]

The plot thickens...

I switched to a different virtual environment and tried rebuilding. I also added the print statement. The different environment (created pre-OS upgrade) worked just fine. Here's the output of the print statement:

87339 INFO: Loading module hook 'hook-numpy.py' from '/Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy/lib/python3.7/site-packages/PyInstaller/hooks'...
87340 INFO: Loading module hook 'hook-distutils.py' from '/Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy/lib/python3.7/site-packages/PyInstaller/hooks'...
filename = /Users/my.user.name/.pyenv/versions/3.7.6/include/python3.7m/pyconfig.h
filename = /Users/my.user.name/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/Makefile
87342 INFO: Loading module hook 'hook-pygments.py' from '/Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy/lib/python3.7/site-packages/PyInstaller/hooks'...

[bwoodsend]

Ok well I'll start with the environment that doesn't work. When PyInstaller tries to work out where to put pyconfig.h it wants to find its location

filename = /Users/my.user.name/.pyenv/versions/3.7.6/include/python3.7m/pyconfig.h

relative to either of

base_prefix = /Users/my.user.name/.pyenv/versions/3.7.6/Python.framework/Versions/3.7
sys.prefix = /Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy

But neither of them are strict base paths, so the list of possible candidates possible_prefixes is empty - giving us the IndexError.

[bwoodsend]

For the pre OS upgrade env could you rerun those print tests from above?

[phirst]

Python 3.7.6 (default, Jul 21 2020, 15:29:33) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyInstaller.utils.hooks import base_prefix
>>> print(base_prefix)
/Users/my.user.name/.pyenv/versions/3.7.6/Python.framework/Versions/3.7
>>> import sys
>>> print(sys.prefix)
/Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy
>>> from PyInstaller import __file__ as pyi_root
>>> print(pyi_root)
/Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy/lib/python3.7/site-packages/PyInstaller/__init__.py

[bwoodsend]

Hmm that appears to contradict what originally thought a couple of messages up. Could you also run in the env that works:

from PyInstaller.utils.hooks import _find_prefix
print(_find_prefix("/Users/my.user.name/.pyenv/versions/3.7.6/include/python3.7m/pyconfig.h"))

[phirst]

>>> from PyInstaller.utils.hooks import _find_prefix
>>> print(_find_prefix("/Users/my.user.name/.pyenv/versions/3.7.6/include/python3.7m/pyconfig.h"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy/lib/python3.7/site-packages/PyInstaller/utils/hooks/__init__.py", line 830, in _find_prefix
    return possible_prefixes[0]
IndexError: list index out of range

[bwoodsend]

Grr that makes no sense. This one and this one now contradict each other. I don't understand why that function would work during a PyInstaller build but not in the console.

[bwoodsend]

I notice in #1545 that the pyconfig.h might not actually be needed in the 1st place. In which case we might as well just scrap it rather than try to fix this. I'll look into it.

[bwoodsend]

Another question: Where does sys.executable point to in your environments?

[phirst]

Working : /Users/my.user.name/.pyenv/versions/3.7.6/envs/mainpy/bin/python3
Non-Working : /Users/my.user.name/.pyenv/versions/3.7.6/envs/buildpy/bin/python3

[bwoodsend]

What about sys.real_prefix and sys.base_prefix? Chances are only one of them exists.

[phirst]

sys.real_prefix
Working: AttributeError: module 'sys' has no attribute 'real_prefix'
Not-Working: AttributeError: module 'sys' has no attribute 'real_prefix'

sys.base_prefix
Working: /Users/my.user.name/.pyenv/versions/3.7.6/Python.framework/Versions/3.7
Not-Working: /Users/my.user.name/.pyenv/versions/3.7.6/Python.framework/Versions/3.7

[bwoodsend]

@phirst We have an open PR which may fix this. Could you give it a go?

[kontur]

Having a similar issue but the bugfix does not seem to resolve it.

With the bugfix, my trace is (removed some of the hook warnings):

38 INFO: PyInstaller: 4.0.dev0
38 INFO: Python: 3.8.3
42 INFO: Platform: macOS-10.15.6-x86_64-i386-64bit
42 INFO: wrote /Users/johannes/Packages/typeface-strings/typestrings.spec
45 INFO: UPX is not available.
45 INFO: Removing temporary files and cleaning cache in /Users/johannes/Library/Application Support/pyinstaller
46 INFO: Extending PYTHONPATH with paths
['/Users/johannes/Packages/typeface-strings/src',
 '/Users/johannes/Packages/typeface-strings']
49 INFO: checking Analysis
49 INFO: Building Analysis because Analysis-00.toc is non existent
49 INFO: Initializing module dependency graph...
50 INFO: Caching module graph hooks...
54 WARNING: Several hooks defined for module 'llvmlite'.Please take care they do not conflict.
[...]
60 WARNING: Several hooks defined for module 'win32com'.Please take care they do not conflict.
61 WARNING: Runtime hooks for certifi have already been defined. Skipping the runtime hooks for certifi that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
61 WARNING: Runtime hooks for enchant have already been defined. Skipping the runtime hooks for enchant that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
61 WARNING: Runtime hooks for osgeo have already been defined. Skipping the runtime hooks for osgeo that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
61 WARNING: Runtime hooks for traitlets have already been defined. Skipping the runtime hooks for traitlets that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
61 WARNING: Runtime hooks for twisted.internet.reactor have already been defined. Skipping the runtime hooks for twisted.internet.reactor that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
61 WARNING: Runtime hooks for usb have already been defined. Skipping the runtime hooks for usb that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
62 WARNING: Runtime hooks for nltk have already been defined. Skipping the runtime hooks for nltk that are defined in /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks.
62 INFO: Analyzing base_library.zip ...
1806 INFO: Processing pre-find module path hook distutils from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1807 INFO: distutils: retargeting to non-venv dir '/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/lib/python3.8'
3494 INFO: Caching module dependency graph...
3578 INFO: running Analysis Analysis-00.toc
3584 INFO: Analyzing /Users/johannes/Packages/typeface-strings/src/typestrings.py
/Users/johannes/Packages/typeface-strings/src/typestrings.py:122: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if generatedText is not "":
3923 INFO: Processing pre-safe import module hook six.moves from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
4019 INFO: Processing pre-safe import module hook win32com from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
4019 INFO: Processing pre-safe import module hook win32com from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-win32com.py'.
5713 INFO: Processing module hooks...
5713 INFO: Loading module hook 'hook-appdirs.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
5714 INFO: Excluding import 'win32com'
5715 INFO:   Removing import of win32com from module appdirs
5715 INFO: Loading module hook 'hook-regex.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
5715 INFO: Loading module hook 'hook-pkg_resources.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6001 WARNING: Hidden import "pkg_resources.py2_warn" not found!
6002 INFO: Excluding import '__main__'
6003 INFO:   Removing import of __main__ from module pkg_resources
6003 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6003 INFO: Loading module hook 'hook-pytz.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6073 INFO: Loading module hook 'hook-lib2to3.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6105 INFO: Loading module hook 'hook-encodings.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6153 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
Traceback (most recent call last):
  File "/Users/johannes/.pyenv/versions/typestrings/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 716, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 663, in build
    exec(code, spec_namespace)
  File "/Users/johannes/Packages/typeface-strings/typestrings.spec", line 6, in <module>
    a = Analysis(['src/typestrings.py'],
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
    self.assemble()
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/building/build_main.py", line 418, in assemble
    self.graph.process_post_graph_hooks()
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/depend/analysis.py", line 365, in process_post_graph_hooks
    module_hook.post_graph()
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/depend/imphook.py", line 440, in post_graph
    self._load_hook_module()
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/depend/imphook.py", line 406, in _load_hook_module
    self._hook_module = importlib_load_source(
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/compat.py", line 588, in importlib_load_source
    return mod_loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 462, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 962, in load_module
  File "<frozen importlib._bootstrap_external>", line 787, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/hook-sysconfig.py", line 28, in <module>
    datas = [(_CONFIG_H, relpath_to_config_or_make(_CONFIG_H))]
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/utils/hooks/__init__.py", line 841, in relpath_to_config_or_make
    prefix = _find_prefix(filename)
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/utils/hooks/__init__.py", line 830, in _find_prefix
    return possible_prefixes[0]
IndexError: list index out of range

I'm running pyinstaller -F --log-level INFO --clean src/typestrings.py but simply pyinstaller src/typestrings.py ends in the same error.

I tested this with Python 3.7.6 and 3.8.3 installed via pyenv with the --enable-framework flag.

Funnily enough I built the same project/script a couple of days ago on my other Mac, also after installing pyenv with the --enable-framework flag like I just did on this machine where the error occurs.

[kontur]

Aaaactually. I just went to the other machine to compile and checked if something is still different between the two and it turns out on the other machine it works to be running pyinstaller from a pyenv version (3.8.3) but what does not work on this machine is running it from a pyenv virtualenv using python 3.8.3 — however, also on this machine where I originally got the error above when I do not use pyenv virtualenv but just a pyenv python 3.8.3 it does compile without error.

Does this help in any way? I presume you'd have to be quite familiar with the inner workings of pyenv and pyenv-virtualenv but this is the only discernible difference I could find between the two setups.

[bwoodsend]

OK, thanks @kontur. I think this confirms that this is a separate issue to the one the PR addresses.

pyenv-virtualenv

That's a thing too? How many venv equivalents does Python need? Grrr.

@kontur Could you clarify for me - do the virtual environments share the pyconfig.h file for a Python installation or do they each get their own copy?

[kontur]

That's a thing too? How many venv equivalents does Python need? Grrr.

Ikr :D pyenv & pyenv-virtualenv is really quite neat, from a user point of view. Easy python version switching and venvs associated with those.

As to your question I cannot say. pyenv generally works by shimming the binaries to redirect calls, so my guess would be that the pyenv-virtualenvs use the python version of the pyenv they were created with and only isolate the "site-packages" to that pyenv-virtualvenv.

[bwoodsend]

and only isolate the "site-packages" to that pyenv-virtualvenv.

That's what I used to think venv does, But it turned out to mess with quite a bit more.

Could you run the following inside a virtual environmemt?

from distutils.sysconfig import get_python_inc
print(get_python_inc())
import sys
print(sys.prefix)
print(sys.base_prefix)
print(sys.exec_prefix)

[kontur]

From the "failing" environment:

/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/include/python3.8
/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings

Vs. just python 3.8.3 where it works:

/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/include/python3.8
/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
/Users/johannes/.pyenv/versions/3.8.3

[bwoodsend]

Ok so it looks like base_prefix is what PyInstaller should be using.

Can I also have sys.real_prefix (may be an attribute error) and PyInstaller.compat.base_prefix? I suspect PyInstaller is picking up the wrong one. It's got a rather messy nest of logic to try and choose,

[bwoodsend]

Ohh, I see there has been some changes in that area since that would mask this issue. 4d66c7d10a44ad5fb9cdf2dfeebad56e8c49de5d would cause the exception to go away but would mislead you into thinking it is fixed because it would put the file in the wrong place.

[bwoodsend]

I've made some small changes. Could you test them?

pip install https://github.com/bwoodsend/pyinstaller/archive/fix-sysconfig.zip

You should be able to test it just by running:

import os
from distutils.sysconfig import get_python_inc
from PyInstaller.utils.hooks import _find_prefix
print(_find_prefix(os.path.join(get_python_inc(), "pyconfig.h")))
print(_find_prefix(os.path.join(get_python_inc(True), "pyconfig.h")))

I've chucked in some extra logging calls so could you copy/paste everything it says...

[kontur]

With the fix sysconfig install it works 👏

37 INFO: PyInstaller: 4.1.dev0
37 INFO: Python: 3.8.3
42 INFO: Platform: macOS-10.15.6-x86_64-i386-64bit
43 INFO: wrote /Users/johannes/Packages/typeface-strings/typestrings.spec
46 INFO: UPX is not available.
46 INFO: Removing temporary files and cleaning cache in /Users/johannes/Library/Application Support/pyinstaller
47 INFO: Extending PYTHONPATH with paths
['/Users/johannes/Packages/typeface-strings/src',
 '/Users/johannes/Packages/typeface-strings']
51 INFO: checking Analysis
51 INFO: Building Analysis because Analysis-00.toc is non existent
51 INFO: Initializing module dependency graph...
52 INFO: Caching module graph hooks...
57 INFO: Analyzing base_library.zip ...
1749 INFO: Processing pre-find module path hook distutils from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1750 INFO: distutils: retargeting to non-venv dir '/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/lib/python3.8'
3423 INFO: Caching module dependency graph...
3509 INFO: running Analysis Analysis-00.toc
3515 INFO: Analyzing /Users/johannes/Packages/typeface-strings/src/typestrings.py
3851 INFO: Processing pre-safe import module hook six.moves from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
3945 INFO: Processing pre-safe import module hook win32com from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
5574 INFO: Processing module hooks...
5574 INFO: Loading module hook 'hook-appdirs.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
5575 INFO: Excluding import 'win32com'
5576 INFO:   Removing import of win32com from module appdirs
5576 INFO: Loading module hook 'hook-regex.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
5576 INFO: Loading module hook 'hook-pkg_resources.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
5856 WARNING: Hidden import "pkg_resources.py2_warn" not found!
5856 WARNING: Hidden import "pkg_resources.markers" not found!
5857 INFO: Excluding import '__main__'
5857 INFO:   Removing import of __main__ from module pkg_resources
5858 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
5858 INFO: Loading module hook 'hook-pytz.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
5926 INFO: Loading module hook 'hook-lib2to3.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
5960 INFO: Loading module hook 'hook-encodings.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6003 INFO: Loading module hook 'hook-distutils.util.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6005 INFO: Excluding import 'lib2to3.refactor'
6006 INFO:   Removing import of lib2to3.refactor from module distutils.util
6006 INFO: Loading module hook 'hook-pickle.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6007 INFO: Excluding import 'argparse'
6007 INFO:   Removing import of argparse from module pickle
6007 INFO: Loading module hook 'hook-heapq.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6008 INFO: Excluding import 'doctest'
6009 INFO:   Removing import of doctest from module heapq
6009 INFO: Loading module hook 'hook-difflib.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6009 INFO: Excluding import 'doctest'
6010 INFO:   Removing import of doctest from module difflib
6019 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6020 INFO: Excluding import 'test'
6020 INFO:   Removing import of test from module multiprocessing.util
6021 INFO:   Removing import of test.support from module multiprocessing.util
6021 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6021 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/include/python3.8/pyconfig.h
6021 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
6021 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6021 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6021 INFO: No prefix found. Assuming running from build directory.
6021 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/lib/python3.8/config-3.8-darwin/Makefile
6021 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
6021 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6021 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6055 INFO: Loading module hook 'hook-xml.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6095 INFO: Loading module hook 'hook-distutils.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
6096 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/include/python3.8/pyconfig.h
6096 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
6096 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6096 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6096 INFO: No prefix found. Assuming running from build directory.
6096 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/lib/python3.8/config-3.8-darwin/Makefile
6096 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
6096 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6096 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
6119 INFO: Looking for ctypes DLLs
6152 WARNING: library msvcrt required via ctypes not found
6163 INFO: Analyzing run-time hooks ...
6169 INFO: Including run-time hook '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
6172 INFO: Including run-time hook '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py'
6180 INFO: Looking for dynamic libraries
6419 INFO: Looking for eggs
6419 INFO: Using Python library /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/Python
6423 INFO: Warnings written to /Users/johannes/Packages/typeface-strings/build/typestrings/warn-typestrings.txt
6460 INFO: Graph cross-reference written to /Users/johannes/Packages/typeface-strings/build/typestrings/xref-typestrings.html
6500 INFO: checking PYZ
6500 INFO: Building PYZ because PYZ-00.toc is non existent
6500 INFO: Building PYZ (ZlibArchive) /Users/johannes/Packages/typeface-strings/build/typestrings/PYZ-00.pyz
7035 INFO: Building PYZ (ZlibArchive) /Users/johannes/Packages/typeface-strings/build/typestrings/PYZ-00.pyz completed successfully.
7048 INFO: checking PKG
7048 INFO: Building PKG because PKG-00.toc is non existent
7048 INFO: Building PKG (CArchive) PKG-00.pkg
9683 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
9698 INFO: Bootloader /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/bootloader/Darwin-64bit/run
9698 INFO: checking EXE
9698 INFO: Building EXE because EXE-00.toc is non existent
9698 INFO: Building EXE from EXE-00.toc
9699 INFO: Appending archive to EXE /Users/johannes/Packages/typeface-strings/dist/typestrings
9709 INFO: Fixing EXE for code signing /Users/johannes/Packages/typeface-strings/dist/typestrings
9711 INFO: Building EXE from EXE-00.toc completed successfully.

Running the test as python test.py I get:

30 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/include/python3.8/pyconfig.h
30 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
30 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
30 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
30 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/include/python3.8/pyconfig.h
30 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
30 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
30 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
30 INFO: No prefix found. Assuming running from build directory.
/Users/johannes/.pyenv/versions/3.8.3/include/python3.8

Thanks so much for your effort! Let me know if I can test anything else 👍

[bwoodsend]

Ooh no. What I said about above changes that would mask this error has happened. This isn't actually fixed. It's now defaulting to something inappropriate so that your pyconfig.h file is presumably in the root of your app instead of wherever it's supposed to be.

What I though would happen inside PyInstaller was this output:

30 INFO: Searching for a prefix for /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/include/python3.8/pyconfig.h
30 INFO: Reject prefix /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings
30 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8
30 INFO: Accept prefix /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8

But it turns out pyenv-virtualenv does copy the header files so actually we see the second behaviour which it handles wrong.

But I do have a better idea. Hang on...

[kontur]

I can compile binaries outside the pyenv-virtualenv, so no rush on this from my side. Appreciate your tenaciousness 👍

[bwoodsend]

OK. I've prodded it somewhere else. Let's see what happens.

Could you: Try the pip install https://github.com/bwoodsend/pyinstaller/archive/fix-sysconfig.zip again. Then build your app with the --clean option and share the build log. Or you can just run the PyInstaller/hooks/hook-sysconfig.py script inside the PyInstaller installation (wherever pyenv-virtualenv put it) and see what that says (hopefully the same thing).

There's no need to run those print statements from above again - I'm not touching the _find_prefix() function this time.

[kontur]

Hey,

it does build the binary, but executing it fails:

101 INFO: PyInstaller: 4.1.dev0
101 INFO: Python: 3.8.3
122 INFO: Platform: macOS-10.15.6-x86_64-i386-64bit
122 INFO: wrote /Users/johannes/Packages/typeface-strings/typestrings.spec
126 INFO: UPX is not available.
126 INFO: Removing temporary files and cleaning cache in /Users/johannes/Library/Application Support/pyinstaller
133 INFO: Extending PYTHONPATH with paths
['/Users/johannes/Packages/typeface-strings/src',
 '/Users/johannes/Packages/typeface-strings']
138 INFO: checking Analysis
138 INFO: Building Analysis because Analysis-00.toc is non existent
138 INFO: Initializing module dependency graph...
139 INFO: Caching module graph hooks...
145 INFO: Analyzing base_library.zip ...
1946 INFO: Processing pre-find module path hook distutils from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1946 INFO: distutils: retargeting to non-venv dir '/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/lib/python3.8'
4350 INFO: Caching module dependency graph...
4431 INFO: running Analysis Analysis-00.toc
4443 INFO: Analyzing /Users/johannes/Packages/typeface-strings/src/typestrings.py
4902 INFO: Processing pre-safe import module hook six.moves from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
5016 INFO: Processing pre-safe import module hook win32com from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
7071 INFO: Processing module hooks...
7071 INFO: Loading module hook 'hook-appdirs.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
7105 INFO: Excluding import 'win32com'
7105 INFO:   Removing import of win32com from module appdirs
7105 INFO: Loading module hook 'hook-regex.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
7106 INFO: Loading module hook 'hook-pkg_resources.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7617 WARNING: Hidden import "pkg_resources.py2_warn" not found!
7617 WARNING: Hidden import "pkg_resources.markers" not found!
7618 INFO: Excluding import '__main__'
7619 INFO:   Removing import of __main__ from module pkg_resources
7619 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7619 INFO: Loading module hook 'hook-pytz.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7695 INFO: Loading module hook 'hook-lib2to3.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7773 INFO: Loading module hook 'hook-encodings.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7822 INFO: Loading module hook 'hook-distutils.util.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7824 INFO: Excluding import 'lib2to3.refactor'
7825 INFO:   Removing import of lib2to3.refactor from module distutils.util
7825 INFO: Loading module hook 'hook-pickle.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7826 INFO: Excluding import 'argparse'
7827 INFO:   Removing import of argparse from module pickle
7827 INFO: Loading module hook 'hook-heapq.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7828 INFO: Excluding import 'doctest'
7828 INFO:   Removing import of doctest from module heapq
7828 INFO: Loading module hook 'hook-difflib.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7829 INFO: Excluding import 'doctest'
7830 INFO:   Removing import of doctest from module difflib
7830 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7830 INFO: Excluding import 'test'
7831 INFO:   Removing import of test from module multiprocessing.util
7831 INFO:   Removing import of test.support from module multiprocessing.util
7831 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7834 INFO: Add 'pyconfig.h' platform_specific=0 from '/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/include/python3.8' at 'include/python3.8'.
7835 INFO: Skip 'Makefile' platform_specific=0 from '/Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/include/python3.8' at 'include/python3.8'.
7835 INFO: Add 'pyconfig.h' platform_specific=1 from '/Users/johannes/.pyenv/versions/3.8.3/include/python3.8' at 'include/python3.8'.
7835 INFO: Skip 'Makefile' platform_specific=1 from '/Users/johannes/.pyenv/versions/3.8.3/include/python3.8' at 'include/python3.8'.
7835 INFO: Loading module hook 'hook-xml.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7903 INFO: Loading module hook 'hook-distutils.py' from '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks'...
7926 INFO: Looking for ctypes DLLs
7954 WARNING: library msvcrt required via ctypes not found
7965 INFO: Analyzing run-time hooks ...
7969 INFO: Including run-time hook '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
7971 INFO: Including run-time hook '/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py'
7979 INFO: Looking for dynamic libraries
8153 INFO: Looking for eggs
8153 INFO: Using Python library /Users/johannes/.pyenv/versions/3.8.3/Python.framework/Versions/3.8/Python
8156 INFO: Warnings written to /Users/johannes/Packages/typeface-strings/build/typestrings/warn-typestrings.txt
8192 INFO: Graph cross-reference written to /Users/johannes/Packages/typeface-strings/build/typestrings/xref-typestrings.html
8222 INFO: checking PYZ
8222 INFO: Building PYZ because PYZ-00.toc is non existent
8222 INFO: Building PYZ (ZlibArchive) /Users/johannes/Packages/typeface-strings/build/typestrings/PYZ-00.pyz
8751 INFO: Building PYZ (ZlibArchive) /Users/johannes/Packages/typeface-strings/build/typestrings/PYZ-00.pyz completed successfully.
8763 INFO: checking PKG
8763 INFO: Building PKG because PKG-00.toc is non existent
8763 INFO: Building PKG (CArchive) PKG-00.pkg
11323 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
11337 INFO: Bootloader /Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/bootloader/Darwin-64bit/run
11337 INFO: checking EXE
11337 INFO: Building EXE because EXE-00.toc is non existent
11337 INFO: Building EXE from EXE-00.toc
11337 INFO: Appending archive to EXE /Users/johannes/Packages/typeface-strings/dist/typestrings
11392 INFO: Fixing EXE for code signing /Users/johannes/Packages/typeface-strings/dist/typestrings
11403 INFO: Building EXE from EXE-00.toc completed successfully.

&

(typestrings) Johanness-iMac:typeface-strings johannes$ ./dist/typestrings 
Traceback (most recent call last):
  File "PyInstaller/hooks/rthooks/pyi_rth_pkgres.py", line 13, in <module>
  File "/Users/johannes/.pyenv/versions/3.8.3/envs/typestrings/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "pkg_resources/__init__.py", line 959, in <module>
  File "pkg_resources/__init__.py", line 963, in Environment
  File "pkg_resources/__init__.py", line 190, in get_supported_platform
  File "pkg_resources/__init__.py", line 395, in get_build_platform
  File "sysconfig.py", line 679, in get_platform
  File "sysconfig.py", line 551, in get_config_vars
  File "sysconfig.py", line 422, in _init_posix
ModuleNotFoundError: No module named '_sysconfigdata__darwin_darwin'
[31528] Failed to execute script pyi_rth_pkgres

[bwoodsend]

Well the build log is doing what I wanted it to do. Does --hiddenimport=_sysconfigdata__darwin_darwin fix the runtime issue? I can't understand why that's emerging now...

[yehuda-elementryx]

@bwoodsend - I wanted to let you know that I had a very similar issue and installing the fix you deployed on this branch: pip install https://github.com/bwoodsend/pyinstaller/archive/fix-sysconfig.zip worked for me once I added a hidden import for: _sysconfigdata_m_darwin_darwin. Hoping this sheds some light.

[bwoodsend]

Thanks @yehuda-elementryx. I guess the next thing to test now that PyInstaller seemingly builds ok is that sysconfig is able to find that header at runtime. Could you build and run the following? I haven't fixed the hidden import so continue to add --hiddenimport=_sysconfigdata_m_darwin_darwin explicitly.

import os
import sysconfig

print("Expecting config.h at:", sysconfig.get_config_h_filename())

if os.path.exists(sysconfig.get_config_h_filename()):
    print("Hah Hah Seh Lah! Found config.h!")
    print(sysconfig.get_config_vars()["py_version"])
else:
    print("config.h not found :(")

[yehuda-elementryx]

@bwoodsend - I added the above script at the start of the app that I'm freezing. I then ran pyinstaller to generate a single file executable. Running the single file executable renders:

Expecting config.h at: /var/folders/zz/rsdnjbjd32d7mhfb2tw1y37m0000gn/T/_MEIZILCoW/include/python3.7m/pyconfig.h
Hah Hah Seh Lah! Found config.h!

Is that what you'd had in mind?

[bwoodsend]

Yep. That's what I wanted to hear. I'll turn this into a pull request then...

[bwoodsend]

I've made some more changes. It looks like these files aren't actually needed in the first place so the fix is just to remove the breaking code. But I'd like a final confirmation that it still works before it gets merged.

Anyone on here willing to test for me?

You'd pip install https://github.com/bwoodsend/pyinstaller/archive/fix-sysconfig.zip again and rebuild your app with the --clean option and check it still works.

[kontur]

Thank you for all the investigating. For me a new build works 👍