Pyinstaller: unicode decode error in collect_submodules / exec_command

[handzsujt]

I am using the current develop state: 3.3.dev0+b78bfe5 of pyinstaller with Anconda and Python 3.5.2. on windows7 64bit

The following short script leads to crash during freezing:

import matplotlib
print('test')

with the following output / traceback:

(C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3) C:\Users\handzsuj\Projekte
\semg_dview\diagnostic_view>pyinstaller test_pyinstaller.py -y --clean > output.
txt
250 INFO: PyInstaller: 3.3.dev0+b78bfe5
250 INFO: Python: 3.5.2
250 INFO: Platform: Windows-7-6.1.7601-SP1
252 INFO: wrote C:\Users\handzsuj\Projekte\semg_dview\diagnostic_view\test_pyins
taller.spec
255 INFO: UPX is not available.
257 INFO: Removing temporary files and cleaning cache in C:\Users\handzsuj\AppDa
ta\Roaming\pyinstaller
269 INFO: Extending PYTHONPATH with paths
['C:\\Users\\handzsuj\\Projekte\\semg_dview\\diagnostic_view',
 'C:\\Users\\handzsuj\\Projekte\\semg_dview\\diagnostic_view']
270 INFO: checking Analysis
270 INFO: Building Analysis because out00-Analysis.toc is non existent
270 INFO: Initializing module dependency graph...
276 INFO: Initializing module graph hooks...
280 INFO: Analyzing base_library.zip ...
10293 INFO: running Analysis out00-Analysis.toc
11497 INFO: Caching module hooks...
11514 INFO: Analyzing C:\Users\handzsuj\Projekte\semg_dview\diagnostic_view\test
_pyinstaller.py
12001 INFO: Processing pre-find module path hook   distutils
23050 INFO: Processing pre-find module path hook   site
23051 INFO: site: retargeting to fake-dir 'C:\\Users\\handzsuj\\AppData\\Local\\
Continuum\\Anaconda3\\lib\\site-packages\\PyInstaller\\fake-modules'
23152 INFO: Processing pre-safe import module hook   win32com
29176 INFO: Processing pre-safe import module hook   six.moves
79279 INFO: Loading module hooks...
79279 INFO: Loading module hook "hook-encodings.py"...
79776 INFO: Loading module hook "hook-IPython.py"...
79812 INFO: Excluding import 'PyQt5'
79824 WARNING:   Removing import IPython.external.qt_loaders from module PyQt5
79830 INFO: Excluding import 'gtk'
79840 WARNING:   Removing import IPython.lib.inputhook from module gtk
79847 WARNING:   Removing import IPython.lib.inputhookgtk from module gtk
79849 INFO: Excluding import 'PySide'
79861 WARNING:   Removing import IPython.external.qt_loaders from module PySide
79868 INFO: Excluding import 'PyQt4'
79880 WARNING:   Removing import IPython.external.qt_loaders from module PyQt4.Q
tGui
79881 WARNING:   Removing import IPython.external.qt_loaders from module PyQt4.Q
tSvg
79881 WARNING:   Removing import IPython.external.qt_loaders from module PyQt4.Q
tCore
79881 WARNING:   Removing import IPython.external.qt_loaders from module PyQt4
79887 INFO: Excluding import 'tkinter'
79898 WARNING:   Removing import IPython.lib.inputhook from module tkinter
79905 WARNING:   Removing import IPython.lib.clipboard from module tkinter
79907 INFO: Excluding import 'matplotlib'
79913 WARNING:   Removing import IPython.core.pylabtools from module matplotlib.
figure
79913 WARNING:   Removing import IPython.core.pylabtools from module matplotlib.
_pylab_helpers
79914 WARNING:   Removing import IPython.core.pylabtools from module matplotlib.
pyplot
79914 WARNING:   Removing import IPython.core.pylabtools from module matplotlib.
pylab
79914 WARNING:   Removing import IPython.core.pylabtools from module matplotlib
79927 INFO: Loading module hook "hook-distutils.py"...
79930 INFO: Loading module hook "hook-PIL.py"...
79941 INFO: Import to be excluded not found: 'FixTk'
79942 INFO: Excluding import 'PySide'
79948 WARNING:   Removing import PIL.ImageQt from module PySide
79949 INFO: Excluding import 'PyQt5'
79956 WARNING:   Removing import PIL.ImageQt from module PyQt5
79957 INFO: Excluding import 'tkinter'
79964 INFO: Excluding import 'PyQt4'
79972 WARNING:   Removing import PIL.ImageQt from module PyQt4.QtGui
79973 WARNING:   Removing import PIL.ImageQt from module PyQt4.QtCore
79974 INFO: Loading module hook "hook-setuptools.py"...
79978 INFO: Loading module hook "hook-jinja2.py"...
80038 INFO: Loading module hook "hook-win32com.py"...
80402 INFO: Loading module hook "hook-shelve.py"...
80463 INFO: Loading module hook "hook-pkg_resources.py"...
82529 INFO: Loading module hook "hook-pycparser.py"...
82807 INFO: Loading module hook "hook-numpy.core.py"...
82970 INFO: MKL libraries found when importing numpy. Adding MKL to binaries
82979 INFO: Loading module hook "hook-PyQt4.QtGui.py"...
83908 INFO: Loading module hook "hook-cryptography.py"...
90378 INFO: Loading module hook "hook-PIL.Image.py"...
91485 INFO: Loading module hook "hook-PyQt4.py"...
91490 INFO: Loading module hook "hook-sysconfig.py"...
91493 INFO: Loading module hook "hook-matplotlib.backends.py"...
92972 INFO:   Matplotlib backend "GTK": ignored
    Gtk* backend requires pygtk to be installed.
94045 INFO:   Matplotlib backend "GTKAgg": ignored
    Gtk* backend requires pygtk to be installed.
94792 INFO:   Matplotlib backend "GTKCairo": ignored
    No module named 'gtk'
95994 INFO:   Matplotlib backend "MacOSX": ignored
    cannot import name '_macosx'
97165 INFO:   Matplotlib backend "Qt4Agg": added
98325 INFO:   Matplotlib backend "Qt5Agg": added
99520 INFO:   Matplotlib backend "TkAgg": added
100657 INFO:   Matplotlib backend "WX": ignored
    Matplotlib backend_wx and backend_wxagg require wxPython >=2.8.12
101708 INFO:   Matplotlib backend "WXAgg": ignored
    Matplotlib backend_wx and backend_wxagg require wxPython >=2.8.12
102552 INFO:   Matplotlib backend "GTK3Cairo": ignored
    Gtk3 backend requires pygobject to be installed.
103673 INFO:   Matplotlib backend "GTK3Agg": ignored
    Gtk3 backend requires pygobject to be installed.
106161 INFO:   Matplotlib backend "WebAgg": added
109030 INFO:   Matplotlib backend "nbAgg": added
110121 INFO:   Matplotlib backend "agg": added
110869 INFO:   Matplotlib backend "cairo": ignored
    Cairo backend requires that cairocffi or pycairo is installed.
111702 INFO:   Matplotlib backend "emf": ignored
    No module named 'matplotlib.backends.backend_emf'
112510 INFO:   Matplotlib backend "gdk": ignored
    No module named 'gobject'
113574 INFO:   Matplotlib backend "pdf": added
115052 INFO:   Matplotlib backend "pgf": added
116203 INFO:   Matplotlib backend "ps": added
117321 INFO:   Matplotlib backend "svg": added
118535 INFO:   Matplotlib backend "template": added
119661 INFO: Loading module hook "hook-lib2to3.py"...
119675 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
119679 INFO: Loading module hook "hook-zmq.py"...
Traceback (most recent call last):
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\Scripts\pyinstaller-
script.py", line 9, in <module>
    load_entry_point('PyInstaller==3.3.dev0+b78bfe5', 'console_scripts', 'pyinst
aller')()
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\__main__.py", line 90, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\__main__.py", line 46, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\build_main.py", line 788, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'
))
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\build_main.py", line 734, in build
    exec(text, spec_namespace)
  File "<string>", line 16, in <module>
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\build_main.py", line 212, in __init__
    self.__postinit__()
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\datastruct.py", line 161, in __postinit__
    self.assemble()
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\build_main.py", line 470, in assemble
    module_hook.post_graph()
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\imphook.py", line 409, in post_graph
    self._load_hook_module()
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\building\imphook.py", line 376, in _load_hook_module
    self.hook_module_name, self.hook_filename)
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\compat.py", line 744, in importlib_load_source
    return mod_loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 388, in _check_name_wrappe
r
  File "<frozen importlib._bootstrap_external>", line 809, in load_module
  File "<frozen importlib._bootstrap_external>", line 668, in load_module
  File "<frozen importlib._bootstrap>", line 268, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 693, in _load
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\hooks\hook-zmq.py", line 18, in <module>
    hiddenimports.extend(collect_submodules('zmq.backend'))
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\utils\hooks\__init__.py", line 619, in collect_submodules
    repr(pkg_dir), package))
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\utils\hooks\__init__.py", line 90, in exec_statement
    return __exec_python_cmd(cmd)
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\utils\hooks\__init__.py", line 77, in __exec_python_cmd
    txt = exec_python(*cmd, env=pp_env)
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\compat.py", line 549, in exec_python
    return exec_command(*cmdargs, **kwargs)
  File "C:\Users\handzsuj\AppData\Local\Continuum\Anaconda3\lib\site-packages\Py
Installer\compat.py", line 356, in exec_command
    out = out.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 178: invali
d start byte

[handzsujt]

I've added simply a print statement before the line causing the error in compat.py. This shows the following byte-string:
b'_cffi_ext.c\r\nC:\\Users\\handzsuj\\AppData\\Local\\Continuum\\Anaconda3\\lib\\site-packages\\zmq\\backend\\cffi\\__pycache__\\_cffi_ext.c(209): fatal error C1083: Datei (Include) kann nicht ge\x94ffnet werden: "sys/un.h": No such file or directory\r\nzmq.backend.cffi\r\nzmq.backend.cython\r\nzmq.backend.cython._device\r\nzmq.backend.cython._poll\r\nzmq.backend.cython._version\r\nzmq.backend.cython.constants\r\nzmq.backend.cython.context\r\nzmq.backend.cython.error\r\nzmq.backend.cython.message\r\nzmq.backend.cython.socket\r\nzmq.backend.cython.utils\r\nzmq.backend.select\r\n'

At position 178 there is the 0x94 that is coming from the german "geöffnet"

If I replace
out = out.decode(encoding)
with
out = out.decode(encoding, errors='ignore')
then freezing is succeeding and the generated dist is working :-)

If I do it with my real application I run into some "self-made" import errors but it seems to work.

Is this a solution?

[hyy1111]

This is works for me. Thank you!!!!!!

[rpolitex]

Also obtain this problem.

It happens because:

1. \Lib\site-packages\zmq\backend\cffi\__pycache__\_cffi_ext.c file in ZMQ module declares depending from <sys/un.h>, that is not available for Windows.
2. PyInstaller try to analyze this <sys/un.h>, but cannot find it and receives an error message from the Operating System.
3. This error message is a center of our problem, because it is given in the system language (may be Russian, German, Mandarin, etc.) and its coding depends from the command-line default code page.

So, my way to solve this problem is very simple. And it doesn't hurting the original PyInstaller script.

Just change your default codepage to UTF-8 with chcp 65001 command.

Hope it helps.

[rokm]

This issue was caused by an attempted compilation of C extension in zmq.backend.cffi during module collection. In a frozen application, this backend is non-functional (and cython backend seems to be the first choice anyway), so PR pyinstaller/pyinstaller-hooks-contrib#59 modified the zmq hook to avoid collecting it in the first place. The change is available in the latest release of pyinstaller-hooks-contrib (v.2020.10).

This should also address this issue, so I'm closing it now. If you still happen to be experiencing this problem, please open a new issue with updated details.