Pyinstaller: distutils not included with latest virtualenv (16.4.0)

Created on 12 Feb 2019  ·  41Comments  ·  Source: pyinstaller/pyinstaller

This was already reported in #4031. The issue was closed without a fix so I'm creating this one.

With virtualenv 16.4.0, pyinstaller reports :

3583 INFO: distutils: retargeting to non-venv dir '/usr/lib64/python3.6/distutils/__init__.py'

and then during "Loading module hook" sequence, the hook-distutils.py is missing and distutils modules are not included into the final executable binary.

When executing the binary the error is:

ModuleNotFoundError: No module named 'distutils'
[10373] Failed to execute script <name here>

With virtualenv 16.1.0, pyinstaller reports :

3157 INFO: Processing pre-find module path hook   distutils
5053 INFO: Loading module hook "hook-distutils.py"...

and distutils modules are included into the final executable binary.

hooks good first issue pull-request wanted

Most helpful comment

Downgrading my system virtualenv to 16.1 did not seem to help.

However, based on @tappi287 's great analysis, I came up with the following work-around. Close to the start of your spec file, add the following:

# work-around for https://github.com/pyinstaller/pyinstaller/issues/4064
import distutils
if distutils.distutils_path.endswith('__init__.py'):
    distutils.distutils_path = os.path.dirname(distutils.distutils_path)

This will temporarily override the distutils_path to what pyinstaller expects, resulting in working builds.

All 41 comments

I also have a pbl with disutils on 16.3. Pyinstaller does not says anything:

5055 INFO: Processing pre-find module path hook   distutils
5055 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.6/distutils/__init__.py'

But later on

16793 WARNING: Hidden import "distutils.command.build_ext" not found!
...
17570 DEBUG: Collecting submodules for distutils
17676 DEBUG: collect_submodules - Found submodules: {'distutils'}

But at execution I still get:

ModuleNotFoundError: No module named 'distutils'

Worked fine on 16.1

Running pyi-archive_viewer on the executable does not list distutils package.
On an executable that worked, pyi-archive_viewer shows the following packages:

 'distutils': (1, 1487595, 267),
 'distutils._msvccompiler': (0, 1487862, 7159),
 'distutils.archive_util': (0, 1495021, 3352),
 'distutils.ccompiler': (0, 1498373, 12557),
 'distutils.cmd': (0, 1510930, 6328),
 'distutils.command': (1, 1517258, 326),
 'distutils.command.bdist': (0, 1517584, 2030),
 'distutils.command.build_ext': (0, 1519614, 8083),
 'distutils.command.build_scripts': (0, 1527697, 2375),
 'distutils.command.install': (0, 1530072, 6322),
 'distutils.command.install_lib': (0, 1536394, 2602),
 'distutils.command.install_scripts': (0, 1538996, 1128),
 'distutils.command.sdist': (0, 1540124, 6183),
 'distutils.config': (0, 1546307, 1895),
 'distutils.core': (0, 1548202, 3246),
 'distutils.debug': (0, 1551448, 137),
 'distutils.dep_util': (0, 1551585, 1430),
 'distutils.dir_util': (0, 1553015, 3048),
 'distutils.dist': (0, 1556063, 13966),
 'distutils.errors': (0, 1570029, 1966),
 'distutils.extension': (0, 1571995, 3282),
 'distutils.fancy_getopt': (0, 1575277, 5261),
 'distutils.file_util': (0, 1580538, 2989),
 'distutils.filelist': (0, 1583527, 4493),
 'distutils.log': (0, 1588020, 1080),
 'distutils.msvc9compiler': (0, 1589100, 8918),
 'distutils.spawn': (0, 1598018, 2749),
 'distutils.sysconfig': (0, 1600767, 6128),
 'distutils.text_file': (0, 1606895, 3793),
 'distutils.util': (0, 1610688, 7966),
 'distutils.version': (0, 1618654, 3110),
 'distutils.versionpredicate': (0, 1621764, 2346),

Solution for the moment: force the system's virtualenv to be 16.1 (freeze the virtualenv package version inside the virtualenvironment is not enough)

Just to pinpoint the problem a bit further: <virtual_env>/Lib/distutils/__init__.py attribute distutils_path returns a path to a file <system_interpreter>/Lib/distutils/__init__.py from virtualenv 16.4.0 on, but previously returned a path to the directory <system_interpreter>/Lib/distutils/.

PyInstaller hook hook-distutils uses os.path.dirname to expand the search directorys which will result in <system_interpreter>/Lib/distutils/ with virtualenv 16.4.0 which would result with <system_interpreter>/Lib/ with previous version of virtualenv.

Downgrading my system virtualenv to 16.1 did not seem to help.

However, based on @tappi287 's great analysis, I came up with the following work-around. Close to the start of your spec file, add the following:

# work-around for https://github.com/pyinstaller/pyinstaller/issues/4064
import distutils
if distutils.distutils_path.endswith('__init__.py'):
    distutils.distutils_path = os.path.dirname(distutils.distutils_path)

This will temporarily override the distutils_path to what pyinstaller expects, resulting in working builds.

I also had this problem with virtualenv. So i switch to use python's built in venv.

Does anyone know something? I am working with jenkins. I build a virtualenv with this packages:

  • altgraph==0.16.1
  • cx-Freeze==5.1.1
  • cycler==0.10.0
  • distutils-pytest==0.1
  • future==0.17.1
  • kiwisolver==1.0.1
  • macholib==1.11
  • matplotlib==3.0.2
  • numpy==1.15.0
  • pandas==0.23.4
  • pefile==2018.8.8
  • PyInstaller==3.4
  • pyparsing==2.3.1
  • python-dateutil==2.7.5
  • pytz==2018.9
  • pywin32-ctypes==0.2.0
  • six==1.12.0
  • tornado==6.0.1
  • virtualenv==16.2.0

But my problem is that in jenkins fails becaouse my app said: "ModuleNotFoundError: No module named 'distutils' after buids the my app. I exetue this app (is .exe to Windows) but the console said that.

If i build the app wit pyinstaller in a console with virtualenv and the same packacts as befere, the build app NOT FAILD. Help me please!

Same probleam here.

Same probleam here.

Hi @Bernardoow . My problem appear when I am executing pyinstaller to generate my app in jenkins job into a virtualenv. I don´t know because it happens.
If I execute the pyinstaller in my local machine into a virtualenv the promblem not appear and my app is built successfully. I appreciate any help

Did you tried installing virtualenv 16.1 on your system (pip install —user virtualenv==16.1)?

Before I sent the message I was testing on the linux and pyinstaler not running after compiled.

Now I'm testing on the windows and pyinstaler is working normally.

I'm Using virtualenv 16.1.0 in the boths os and the same requirements.txt. Python 3.6.8

i'm using this code suggested by @cpbotha : import distutils if distutils.distutils_path.endswith('__init__.py'): distutils.distutils_path = os.path.dirname(distutils.distutils_path)

How about a small variation on @cpbotha's suggestion and putting the workaround directly in the hook?

def pre_find_module_path(api):
    # Absolute path of the system-wide "distutils" package when run from within
    # a venv or None otherwise.
    distutils_dir = getattr(distutils, 'distutils_path', None)
    if distutils_dir is not None:
        # workaround for https://github.com/pyinstaller/pyinstaller/issues/4064
        if distutils_dir.endswith('__init__.py'):
            distutils_dir = os.path.dirname(distutils_dir)

        # Find this package in its parent directory.
        api.search_dirs = [os.path.dirname(distutils_dir)]

        logger.info('distutils: retargeting to non-venv dir %r' % distutils_dir)

NB: reverting system virtualenv AND re-creating the venv also worked for me.

virtualenv 16.1 worked for me, thanks!

Thanks to all, I am going to try revet my virtualevn 16.1, when I finish I explain here the results. Maybe be util to all. But I think that the problem is being create virtualenv into Jenkins 😄

Ran into this error, and can confirm the change to virtualenv 16.1.0 was required, but it had to be the virtualenv module that was used by pipenv in my case, which is in line with @gsemet 's comment earlier.

Hi all!
It's impossible generate into a virtualen one app with pyinstaller in a job jenkins with this packages:
https://github.com/pyinstaller/pyinstaller/issues/4064#issuecomment-478358441
I changes to virtualenv==16.1.0 but the problem persists.

"ModuleNotFoundError: No module named 'distutils', I feel frustrated :man_facepalming:
If someone could give me some idea I would be enormously grateful. Thanks to all!

Hi all!
It's impossible generate into a virtualen one app with pyinstaller in a job jenkins with this packages:
#4064 (comment)
I changes to virtualenv==16.1.0 but the problem persists.

"ModuleNotFoundError: No module named 'distutils', I feel frustrated 🤦‍♂️
If someone could give me some idea I would be enormously grateful. Thanks to all!

you probably need to change the virtualenv on the system level of your jenkins build environment, not the virtualenv version of whatever package you're developing

I'm still getting this issue with virtualenv 16.4.3. Down grading to 16.2.0 has worked in the interim.

Hi guys! it was impossible to solve my problem with
_"ModuleNotFoundError: No module named 'distutils'_. (I was trying to build a .exe through pyinsteller). I found a new option to build my python app: Azure DevOps Pipelines. It is similar to Jenkins (I was building my app in a Jenkins server in my local machine but it was impossible for that problem). The .exe works perfect!

Thanks to all for try to help me with problem! 👍

@maxslimmer 's solution (thanks!) worked for me (on a cpy373 based venv), as in (it needs to be in a pre_find_module_path subdir within the hooks dir):

(cpy373_1) oberstet@intel-nuci7:~/scm/crossbario/crossbarfx$ cat pyinstaller/pre_find_module_path/hook-distutils.py 
import distutils
import os

from PyInstaller.utils.hooks import logger

# https://github.com/pyinstaller/pyinstaller/issues/4064
# https://pythonhosted.org/PyInstaller/hooks.html#the-pre-find-module-path-pfmp-api-method

def pre_find_module_path(api):
    # Absolute path of the system-wide "distutils" package when run from within
    # a venv or None otherwise.
    distutils_dir = getattr(distutils, 'distutils_path', None)
    if distutils_dir is not None:
        # workaround for https://github.com/pyinstaller/pyinstaller/issues/4064
        if distutils_dir.endswith('__init__.py'):
            distutils_dir = os.path.dirname(distutils_dir)

        # Find this package in its parent directory.
        api.search_dirs = [os.path.dirname(distutils_dir)]

        logger.info('>>>>>>> CUSTOM >>>>>>>>> distutils: retargeting to non-venv dir %r' % distutils_dir)

create exe without virtualenv , worked for me.

Same exception while running pipenv + python 3.7. Couldn't quite understand how to solve that yet...

This is the pyinstaller output if anybody happen to understand how to solve that:

INFO:project_packer.packer:Removing 0 old egg files.
INFO:project_packer.packer:creating egg file for ../project-framework with: /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/python3.7 setup.py bdist_egg --dist-dir /Users/username/Clients/company/code/project/project-packer
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

INFO:project_packer.packer:egg file created successfully for project-framework
INFO:project_packer.packer:creating egg file for ../pyucwa with: /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/python3.7 setup.py bdist_egg --dist-dir /Users/username/Clients/company/code/project/project-packer
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

INFO:project_packer.packer:egg file created successfully for pyucwa
Running pyinstaller command: /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/pyinstaller --clean --name project --additional-hooks-dir=/Users/username/Clients/company/code/project/project-packer/project_packer/pyinstaller_hooks --distpath ./project --onefile --paths pyucwa-1.2.1-py3.7.egg --paths project_framework-1.1.1-py3.7.egg /Users/username/Clients/company/code/project/project-packer/project_packer/project.py
53 INFO: PyInstaller: 3.4
53 INFO: Python: 3.7.2
60 INFO: Platform: Darwin-18.5.0-x86_64-i386-64bit
60 INFO: wrote /Users/username/Clients/company/code/project/project-packer/project.spec
64 INFO: UPX is not available.
65 INFO: Removing temporary files and cleaning cache in /Users/username/Library/Application Support/pyinstaller
84 INFO: Extending PYTHONPATH with paths
['/Users/username/Clients/company/code/project/project-packer',
 '/Users/username/Clients/company/code/project/project-packer/pyucwa-1.2.1-py3.7.egg',
 '/Users/username/Clients/company/code/project/project-packer/project_framework-1.1.1-py3.7.egg',
 '/Users/username/Clients/company/code/project/project-packer']
84 INFO: checking Analysis
84 INFO: Building Analysis because Analysis-00.toc is non existent
84 INFO: Initializing module dependency graph...
87 INFO: Initializing module graph hooks...
88 INFO: Analyzing base_library.zip ...
3123 INFO: running Analysis Analysis-00.toc
3131 INFO: Caching module hooks...
3134 INFO: Analyzing /Users/username/Clients/company/code/project/project-packer/project_packer/project.py
3568 INFO: Processing pre-find module path hook   distutils
3568 INFO: distutils: retargeting to non-venv dir '/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/__init__.py'
4011 INFO: Processing pre-safe import module hook   urllib3.packages.six.moves
5045 INFO: Processing pre-safe import module hook   six.moves
8187 INFO: Loading module hooks...
8187 INFO: Loading module hook "hook-pkg_resources.py"...
8703 INFO: Processing pre-safe import module hook   win32com
8706 INFO: Loading module hook "hook-babel.py"...
8766 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
8766 INFO: Loading module hook "hook-pytz.py"...
8797 INFO: Loading module hook "hook-cryptography.py"...
9170 INFO: Loading module hook "hook-encodings.py"...
9291 INFO: Loading module hook "hook-certifi.py"...
9293 INFO: Loading module hook "hook-sysconfig.py"...
9304 INFO: Loading module hook "hook-xml.py"...
9305 INFO: Loading module hook "hook-pydoc.py"...
9305 INFO: Loading module hook "hook-websocket.py"...
9377 INFO: checking Tree
9377 INFO: Building Tree because Tree-00.toc is non existent
9377 INFO: Building Tree Tree-00.toc
9383 INFO: checking Tree
9384 INFO: Building Tree because Tree-01.toc is non existent
9384 INFO: Building Tree Tree-01.toc
9384 INFO: Looking for ctypes DLLs
9402 WARNING: library user32 required via ctypes not found
9413 INFO: Analyzing run-time hooks ...
9425 INFO: Including run-time hook 'pyi_rth_pkgres.py'
9426 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
9453 INFO: Looking for dynamic libraries
9567 INFO: Looking for eggs
9567 INFO: Using Python library /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/../.Python
9575 INFO: Warnings written to /Users/username/Clients/company/code/project/project-packer/build/project/warn-project.txt
9661 INFO: Graph cross-reference written to /Users/username/Clients/company/code/project/project-packer/build/project/xref-project.html
9757 INFO: checking PYZ
9758 INFO: Building PYZ because PYZ-00.toc is non existent
9758 INFO: Building PYZ (ZlibArchive) /Users/username/Clients/company/code/project/project-packer/build/project/PYZ-00.pyz
10958 INFO: Building PYZ (ZlibArchive) /Users/username/Clients/company/code/project/project-packer/build/project/PYZ-00.pyz completed successfully.
10990 INFO: checking PKG
10990 INFO: Building PKG because PKG-00.toc is non existent
10990 INFO: Building PKG (CArchive) PKG-00.pkg
19258 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
19308 INFO: Bootloader /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/lib/python3.7/site-packages/PyInstaller/bootloader/Darwin-64bit/run
19308 INFO: checking EXE
19308 INFO: Building EXE because EXE-00.toc is non existent
19308 INFO: Building EXE from EXE-00.toc
19308 INFO: Appending archive to EXE ./project/project
19334 INFO: Fixing EXE for code signing ./project/project
19338 INFO: Building EXE from EXE-00.toc completed successfully.

@oberstet You rock!!!!!! Your solution made it! Would it be assimilated into pyinstaller officially?

Downgrading my system virtualenv to 16.1 did not seem to help.

However, based on @tappi287 's great analysis, I came up with the following work-around. Close to the start of your spec file, add the following:

# work-around for https://github.com/pyinstaller/pyinstaller/issues/4064
import distutils
if distutils.distutils_path.endswith('__init__.py'):
    distutils.distutils_path = os.path.dirname(distutils.distutils_path)

This will temporarily override the distutils_path to what pyinstaller expects, resulting in working builds.

Thank you! This solution solve my problem in pipenv+python3.7+pyinstaller3.4

@truuuuegate can you tell me where excatly did you made the change? I have my app.spec and I add these lines. However, nothing change.
I am using the same solution (pipenv+P3+Pyinst)
Should I add these lines in my app.py?

@jmbruges
I add that code at the beginning of the *.spec file.
Then the pyinstaller process in pipenv could find the distutils module path that in native enviroment.

@jmbruges
I add that code at the beginning of the *.spec file.
Then the pyinstaller process in pipenv could find the distutils module path that in native enviroment.

Thanks, I tried that and did not work. I will see if cx_freeze can resolve it but so far I have other issues trying to install that package.

@maxslimmer 's solution (thanks!) worked for me (on a cpy373 based venv), as in (it needs to be in a pre_find_module_path subdir within the hooks dir):

(cpy373_1) oberstet@intel-nuci7:~/scm/crossbario/crossbarfx$ cat pyinstaller/pre_find_module_path/hook-distutils.py 
import distutils
import os

from PyInstaller.utils.hooks import logger

# https://github.com/pyinstaller/pyinstaller/issues/4064
# https://pythonhosted.org/PyInstaller/hooks.html#the-pre-find-module-path-pfmp-api-method

def pre_find_module_path(api):
    # Absolute path of the system-wide "distutils" package when run from within
    # a venv or None otherwise.
    distutils_dir = getattr(distutils, 'distutils_path', None)
    if distutils_dir is not None:
        # workaround for https://github.com/pyinstaller/pyinstaller/issues/4064
        if distutils_dir.endswith('__init__.py'):
            distutils_dir = os.path.dirname(distutils_dir)

        # Find this package in its parent directory.
        api.search_dirs = [os.path.dirname(distutils_dir)]

        logger.info('>>>>>>> CUSTOM >>>>>>>>> distutils: retargeting to non-venv dir %r' % distutils_dir)

Thanks for your tip, I searched pyinstaller docs and finally understand how it works

Since there is a solution pointed out here, anybody going to create a pull-request? See our Development Guide on how to create pull-requests for PyInstaller.

I ran into this, but I'm not sure continuing to rely on the internal distutils_path is a good idea... maybe the hook should be doing its own system distutils detection instead? Something like:

def pre_find_module_path(api):
    # opcode is not a virtualenv module, so we can use it to find the stdlib
    # Technique taken from virtualenv's "distutils" package detection at
    # https://github.com/pypa/virtualenv/blob/0ab032ce1a/virtualenv_embedded/distutils-init.py#L5
    import opcode

    system_module_path = os.path.normpath(os.path.dirname(opcode.__file__))
    loaded_module_path = os.path.normpath(os.path.dirname(distutils.__file__))
    if system_module_path != loaded_module_path:
        # Find this package in its parent directory.
        api.search_dirs = [system_module_path]
        logger.info('distutils: retargeting to non-venv dir %r' % system_module_path)

@tangm I'm still waiting for someone to provide a pull-request. Then we can discuss teh code there.

Also I'd appreciate a summary of the caus for this issue, so I can understand and comment on the solution.

@htgoebel I've created https://github.com/pyinstaller/pyinstaller/pull/4372 , with a description of the problem.

Thanks @tangm for your pull request. I was able to implement your code myself and fix the distutils issue for me!

spec

also work by doing below:
in the .spec file, at the line “hiddenimports=[]”, change to "hiddenimports=['distutils']"

This bug appears to be closed, but the problem still exists in PyInstaller 3.5.

I can reliably reproduce this with PyInstaller 3.5 in virtualenv 16.7.7 as documented here

Is there an accepted solution that remedies this?

This bug appears to be closed, but the problem still exists in PyInstaller 3.5.

I can reliably reproduce this with PyInstaller 3.5 in virtualenv 16.7.7 as documented here

Is there an accepted solution that remedies this?

the fix probably sitting in git tree waiting for release, have you tried the head version?

This bug appears to be closed, but the problem still exists in PyInstaller 3.5.
I can reliably reproduce this with PyInstaller 3.5 in virtualenv 16.7.7 as documented here
Is there an accepted solution that remedies this?

the fix probably sitting in git tree waiting for release, have you tried the head version?

@AndCycle
I'm a novice when it comes to Git, I'm not sure how I would do that.

Can you point me to the relevant documentation/instructions?

@txoof
maybe just take the shortcut,
the pull req https://github.com/pyinstaller/pyinstaller/pull/4372 is just a patched hook file,
replace the one should do the job.

This bug appears to be closed, but the problem still exists in PyInstaller 3.5.
I can reliably reproduce this with PyInstaller 3.5 in virtualenv 16.7.7 as documented here
Is there an accepted solution that remedies this?

the fix probably sitting in git tree waiting for release, have you tried the head version?

@AndCycle
I'm a novice when it comes to Git, I'm not sure how I would do that.

Can you point me to the relevant documentation/instructions?

https://pyinstaller.readthedocs.io/en/stable/installation.html

This is working for me:
pip install https://github.com/pyinstaller/pyinstaller/tarball/develop

just wanted to note: the workarounds above don't work on pyinstaller 3.6 anymore.

pyinstaller 3.6 has changed its behavior, adding an internal check:

        # Safety check, see above
        global HOOKS_MODULE_NAMES
        assert self.hook_module_name not in HOOKS_MODULE_NAMES
        HOOKS_MODULE_NAMES.add(self.hook_module_name)

to the new file cpy381_1/lib/python3.8/site-packages/PyInstaller/depend/imphook.py.


however: with pyinstaller 3.6, above workaround isn't needed anymore (tested). so simply removing it will make everything happy again=)

Was this page helpful?
0 / 5 - 0 ratings