Requests: Issue with pyinstaller

[gRanger777]

Can't seem to wrap my head around this.

having a problem compiling a windows exe in python 2.7.9 that uses the requests library and can't find anything on google about the specific error. My script runs fine from the interpreter but when i use pyinstaller, i get :ImportError: No module named 'requests.packages.chardet.sys'

I can also compile windows executables that don't use requests just fine.

###Sample Script
----------------Begin
#!/usr/bin/python

import requests

r = requests.get('https://google.com')

print(r.text)
----------------End

###command run to compile into windows exe
---------------Begin
pyinstaller --onefile test.py
OR
pyinstaller test.py
---------------End

###Output
---------------Begin
C:\Python27>pyinstaller test.py
76 INFO: wrote C:\Python27\test.spec
102 INFO: Testing for ability to set icons, version resources...
125 INFO: ... resource update available
129 INFO: UPX is not available.
164 INFO: Processing hook hook-os
409 INFO: Processing hook hook-time
417 INFO: Processing hook hook-cPickle
536 INFO: Processing hook hook-_sre
773 INFO: Processing hook hook-cStringIO
944 INFO: Processing hook hook-encodings
965 INFO: Processing hook hook-codecs
1687 INFO: Extending PYTHONPATH with C:\Python27
1687 INFO: checking Analysis
1687 INFO: Building Analysis because out00-Analysis.toc non existent
1688 INFO: running Analysis out00-Analysis.toc
1690 INFO: Adding Microsoft.VC90.CRT to dependent assemblies of final executable

1781 INFO: Searching for assembly x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21
022.8_none ...
1782 WARNING: Assembly not found
1782 ERROR: Assembly x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_none no
t found
1954 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\python.exe
2039 INFO: Searching for assembly x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21
022.8_none ...
2040 WARNING: Assembly not found
2042 ERROR: Assembly x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_none no
t found
2263 WARNING: lib not found: MSVCR90.dll dependency of C:\Windows\system32\pytho
n27.dll
2266 INFO: Analyzing C:\Python27\lib\site-packages\pyinstaller-2.1.1.dev0-py2.7.
egg\PyInstaller\loader\_pyi_bootstrap.py
2284 INFO: Processing hook hook-os
2309 INFO: Processing hook hook-site
2339 INFO: Processing hook hook-encodings
2582 INFO: Processing hook hook-time
2590 INFO: Processing hook hook-cPickle
2715 INFO: Processing hook hook-_sre
2975 INFO: Processing hook hook-cStringIO
3164 INFO: Processing hook hook-codecs
3907 INFO: Processing hook hook-pydoc
4185 INFO: Processing hook hook-email
4309 INFO: Processing hook hook-httplib
4368 INFO: Processing hook hook-email.message
4517 INFO: Analyzing C:\Python27\lib\site-packages\pyinstaller-2.1.1.dev0-py2.7.
egg\PyInstaller\loader\pyi_importers.py
4690 INFO: Analyzing C:\Python27\lib\site-packages\pyinstaller-2.1.1.dev0-py2.7.
egg\PyInstaller\loader\pyi_archive.py
4865 INFO: Analyzing C:\Python27\lib\site-packages\pyinstaller-2.1.1.dev0-py2.7.
egg\PyInstaller\loader\pyi_carchive.py
5040 INFO: Analyzing C:\Python27\lib\site-packages\pyinstaller-2.1.1.dev0-py2.7.
egg\PyInstaller\loader\pyi_os_path.py
5069 INFO: Analyzing test.py
6014 INFO: Processing hook hook-requests
7263 INFO: Processing hook hook-xml
7445 INFO: Processing hook hook-xml.sax
7516 INFO: Processing hook hook-pyexpat
7646 INFO: Hidden import 'codecs' has been found otherwise
7648 INFO: Hidden import 'encodings' has been found otherwise
7648 INFO: Looking for run-time hooks
7830 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\lib\site-pack
ages\win32\win32pipe.pyd
7987 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\select.p
yd
8144 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\unicoded
ata.pyd
8319 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\lib\site-pack
ages\win32\win32wnet.pyd
8501 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\_hashlib
.pyd
8671 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\bz2.pyd
8859 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\_ssl.pyd

9052 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\_ctypes.
pyd
9223 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\pyexpat.
pyd
9460 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\lib\site-pack
ages\win32\win32api.pyd
9632 WARNING: lib not found: MSVCR90.dll dependency of C:\Python27\DLLs\_socket.
pyd
9828 WARNING: lib not found: MSVCR90.dll dependency of C:\Windows\system32\pywin
types27.dll
9848 INFO: Using Python library C:\Windows\system32\python27.dll
10016 INFO: Warnings written to C:\Python27\build\test\warntest.txt
10023 INFO: checking PYZ
10023 INFO: Rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
10024 INFO: Building PYZ (ZlibArchive) out00-PYZ.toc
12259 INFO: checking PKG
12261 INFO: Rebuilding out00-PKG.toc because out00-PKG.pkg is missing
12261 INFO: Building PKG (CArchive) out00-PKG.pkg
12286 INFO: checking EXE
12287 INFO: Rebuilding out00-EXE.toc because test.exe missing
12289 INFO: Building EXE from out00-EXE.toc
12292 INFO: Appending archive to EXE C:\Python27\build\test\test.exe
12296 INFO: checking COLLECT
12296 INFO: Building COLLECT out00-COLLECT.toc
---------------End

###What happens when running the executable
---------------Begin

C:\Users\gRanger\Desktop\dist\test>test.exe
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "c:\python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstall
er\loader\pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\gRanger\Desktop\build\test\out00-PYZ.pyz\requests", line 58, in
 <module>
  File "c:\python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstall
er\loader\pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\gRanger\Desktop\build\test\out00-PYZ.pyz\requests.utils", line
26, in <module>
  File "c:\python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstall
er\loader\pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\gRanger\Desktop\build\test\out00-PYZ.pyz\requests.compat", line
 7, in <module>
  File "c:\python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstall
er\loader\pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\gRanger\Desktop\build\test\out00-PYZ.pyz\requests.packages.char
det", line 19, in <module>
  File "C:\Users\gRanger\Desktop\build\test\out00-PYZ.pyz\requests.packages", li
ne 83, in load_module
ImportError: No module named 'requests.packages.chardet.sys'
---------------End

[sigmavirus24]

So it looks like pyinstaller isn't detecting an import of the standard library's sys module even though it finds httplib, os and others. You should see if anyone on StackOverflow can help you with this (or if this is a bug in pyinstaller). It's certainly not a bug in requests though

[sigmavirus24]

So this is sort of a bug with that import machinery and sort of a bug with python 2 and sort of a bug with chardet. On the bright side, there's a fix for it that can be performed in chardet and then vendored into requests, etc. On the not-so-bright side, this _can_ affect urllib3 as well.

I was trying to debug https://github.com/jakubroztocil/httpie/issues/315 with pdb to figure out why I was seeing a different error and ran into an issue with this import logic trying to import requests.packages.urllib3.pdb because on Py2, import pdb is treated as a implicit relative import first and then a non-relative import second. (Woo, thanks Python 2.) The temporary work-around was to add from __future__ import absolute_import to the top of the file I was trying to debug in. This, of course, could be applied to urllib3 and chardet both. I think the better option is to attempt to fix the import machinery stolen wholesale from pip.

@dstufft definitely understands this code better than I do, but as I understand it now: we stop trying to import it at L83 if the __import__(name) (which in these cases are chardet.sys and urllib3.pdb) fails. Instead, I think we need to figure out how to try one last case to actually mimic the regular import machinery. I can imagine more complex import failures, like seeing something like chardet.os.path fail, so something like

import_name = name
while import_name:
    (_, import_name) = import_name.split('.', 1)
    try:
        __import__(import_name)
        module = sys.modules(import_name)
    except ImportError:
        pass

if not module:  # or if module is None:
    raise ImportError(...)

sys.modules[name] = module
return module

Does that make sense?

[gRanger777]

Hey, so I finally registered on stackoverflow and posted my question (and this will probably provide more detail for you to debug the issue) but I think something did break in requests. Someone told me to force version 2.5.1 and it worked flawlessly! So it appears whatever issue is plauging pyinstaller/requests has been present since version 2.5.2

http://stackoverflow.com/questions/28775276/pyinstaller-error-importerror-no-module-named-requests-packages-chardet-sys

I'm not blaming requests fully here either =) I realize that it could be an issue on pyinstaller's part still, but I just wanted to report and let you know that I was able to uninstall requests with pip and force version 2.5.1 and now i'm not having any compiling errors or execution errors.

[Lukasa]

@gRanger915 We agree, there's a candidate fix in #2466.

[aanand]

I'm afraid I'm still seeing this with requests 2.6.0 and pyinstaller at both version 2.1 and the current master (67610f). I've created a minimal failing case at aanand/requests-pyinstaller-bug:

$ make
script/build-linux
>> Building Docker image
>> Running bin/test
Importing requests
Imported requests 2.6.0
>> Building dist/test
>> Running dist/test
Importing requests
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PyInstaller/loader/pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "/code/build/test/out00-PYZ.pyz/requests", line 58, in <module>

  File "/usr/local/lib/python2.7/dist-packages/PyInstaller/loader/pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "/code/build/test/out00-PYZ.pyz/requests.utils", line 26, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PyInstaller/loader/pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "/code/build/test/out00-PYZ.pyz/requests.compat", line 7, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PyInstaller/loader/pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "/code/build/test/out00-PYZ.pyz/requests.packages.chardet", line 19, in <module>
  File "/code/build/test/out00-PYZ.pyz/requests.packages", line 95, in load_module
ImportError: No module named 'requests.packages.chardet.sys'
make: *** [default] Error 255

[benmao]

+1

[gwhenne]

+1

[Lukasa]

@sigmavirus24 Seems that we're still having some trouble here with our new vendoring logic. Care to take a look?

[sigmavirus24]

Yep it's on my list.

[sigmavirus24]

@aanand if you can remove as many variables from your reproduction case as possible that'd be appreciated (e.g., just PyInstaller and Requests, no docker or anything else)

[sigmavirus24]

Alternatively, if all the people taking time to reply "+1" can instead provide their reproduction steps, that'd be far more useful.

[sigmavirus24]

Sorry, meant to reopen this after @aanand commented but never did.

So here's the minimal number of steps to reproduce this:

1. Create a virtualenv
2. pip install pyinstaller requests
3. Create a file (e.g., foo.py) that imports requests (and optionally uses it)
4. Run pyinstaller -F <filename> (e.g., pyinstaller -F foo.py)
5. Run the created single-file executable (e.g., ./dist/foo)

Here's the slightly more entertaining thing. In a plain 2.7 environment, if I do this:

 ❯❯❯ pip install requests==2.5.0
Collecting requests==2.5.0
  Downloading requests-2.5.0-py2.py3-none-any.whl (464kB)
    100% |################################| 466kB 561kB/s
Installing collected packages: requests

Successfully installed requests-2.5.0
❯❯❯ python
Python 2.7.9 (default, Dec 13 2014, 22:20:22)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.5.0'
>>> import sys
>>> modules = [m for m in sys.modules.keys() if m.startswith('requests.')]
>>> modules
['requests.packages.urllib3.util.select', 'requests.packages.urllib3.codecs', 'requests.packages.urllib3.socket', 'requests.packages.urllib3.datetime', 'requests.time', 'requests.io', 'requests.packages.urllib3.util.errno', 'requests.packages.urllib3.packages.six.moves', 'requests.urllib2', 'requests.packages.urllib3.packages.StringIO', 'requests.utils', 'requests.packages.urllib3.fields', 'requests.adapters', 'requests.hooks', 'requests.packages.urllib3.util.logging', 'requests.packages.urllib3.packages.types', 'requests.packages.urllib3.util.socket', 'requests.packages.urllib3', 'requests.packages.urllib3.util.request', 'requests.exceptions', 'requests.packages.urllib3.response', 'requests.urlparse', 'requests.os', 'requests.packages.urllib3.util.collections', 'requests.models', 'requests.socket', 'requests.packages.urllib3.poolmanager', 'requests.packages.urllib3.Queue', 'requests.packages.urllib3.sys', 'requests.logging', 'requests.hashlib', 'requests.threading', 'requests.packages.urllib3.httplib', 'requests.sessions', 'requests.codecs', 'requests.packages.urllib3.packages.ordered_dict', 'requests.packages.urllib3.contrib', 'requests.status_codes', 'requests.auth', 'requests.re', 'requests.datetime', 'requests.packages.urllib3.mimetypes', 'requests.collections', 'requests.api', 'requests.struct', 'requests.packages.chardet', 'requests.urllib', 'requests.packages.urllib3.packages.sys', 'requests.packages.urllib3.exceptions', 'requests.packages.urllib3.util.retry', 'requests.packages.urllib3.connection', 'requests.packages.urllib3.ssl', 'requests.StringIO', 'requests.certs', 'requests.platform', 'requests.packages.urllib3.packages.ssl_match_hostname', 'requests.packages.urllib3.util.url', 'requests.packages.urllib3.util.binascii', 'requests.warnings', 'requests.packages.urllib3.util', 'requests.packages.urllib3.util.base64', 'requests.cookies', 'requests.packages.urllib3._collections', 'requests.packages.urllib3.connectionpool', 'requests.packages.urllib3.threading', 'requests.packages.urllib3.util.connection', 'requests.compat', 'requests.packages.urllib3.collections', 'requests.packages.urllib3.util.hashlib', 'requests.packages.urllib3.util.ssl_', 'requests.packages', 'requests.packages.urllib3.zlib', 'requests.cgi', 'requests.base64', 'requests.packages.urllib3.util.ssl', 'requests.packages.urllib3.filepost', 'requests.cookielib', 'requests.json', 'requests.Cookie', 'requests.packages.urllib3.packages.operator', 'requests.packages.urllib3.packages.ssl_match_hostname.ssl', 'requests.packages.chardet.sys', 'requests.packages.urllib3.uuid', 'requests.packages.urllib3.packages._abcoll', 'requests.packages.urllib3.util.response', 'requests.packages.urllib3.util.timeout', 'requests.structures', 'requests.packages.urllib3.packages.thread', 'requests.packages.urllib3.errno', 'requests.packages.urllib3.urllib', 'requests.packages.urllib3.request', 'requests.sys', 'requests.packages.urllib3.urlparse', 'requests.packages.urllib3.packages', 'requests.packages.urllib3.email', 'requests.packages.urllib3.util.time', 'requests.packages.urllib3.io', 'requests.packages.urllib3.logging', 'requests.packages.urllib3.warnings', 'requests.packages.urllib3.packages.six']
>>> 'requests.packages.chardet.sys' in modules
True
>>> print sys.modules['requests.packages.chardet.sys']
None

So this is, in a large part, a wart in how the import machinery in Python 2.x works. If PyInstaller weren't experimental on Python 3, I'd spend time seeing if I could reproduce it there, but I suspect it wouldn't happen. I have a hunch for how to fix this though (and it doesn't involve needing to toy with the custom import machinery any further).

[sigmavirus24]

Further intrigue, pyinstaller places meta-path modifiers into sys.meta_path so debugging into load_module shows:

[<pyi_importers.BuiltinImporter object at 0x105a32990>, <pyi_importers.FrozenImporter object at 0x105a32a10>, <pyi_importers.CExtensionImporter object at 0x105a32b90>, <requests.packages.VendorAlias object at 0x105b0eb90>]

I think the existence of those Importer objects is preventing the default Python behaviour from taking over. (In other words, Python is attempting to do a module local import before doing an absolute import and those meta_path modifiers are preventing it from working as expected.)

So there are two options here:

1. Update requests, chardet, and urllib3 to add from __future__ import absolute_import to the top of each submodule (annoying but in general just a good idea and will help with Py2/Py3 import behaviour compatibility).
2. Start down the rabbit hole I had started chasing the first time. - Develop an algorithm that continuously tries to import an algorithm. Right now we do the following:
   
   
   
   • Given a name, e.g., requests.packages.chardet.sys, try to import that
   
   
   • If that fails, strip requests.packages. from the name and try to import that
   
   
   • If that fails, give up
   
   

We could modify that to instead then attempt to strip requests.packages.chardet. or requests.packages.urllib3 from the name but then something like requests.packages.urllib3.contrib.pyopenssl.ssl.

I have some more experimentation that I'd like to play with, before acting on either of these, but I strongly prefer option 1.

[sigmavirus24]

So the simplest solution, although not one I like best, is to insert the VendorAlias into the first position of sys.meta_path (e.g., sys.meta_path.insert(0, VendorAlias(...))) and the resulting script behaves the same with (this slightly modified) 2.6.0 as with 2.5.0.

That said, I'm still in favor of 1. although it seems far less necessary.

[sigmavirus24]

So I read a bit of the PEP that explains the part sys.meta_path plays and talked to @dstufft a bit on IRC. Provided the way PyInstaller registers it's import hooks on the meta path you can see that it registers it's "BuiltinImporter" first. Since the first hook to return a module wins, we probably want to register our hook first. This allows Python 2's behaviour of try an implicit relative import then try an absolute import to kind of just work.

As a quick fix, I'm going to change our behaviour from appending to the meta path to inserting at the start of it. We kind of want that behaviour anyway.

[aanand]

Nice. #2533 fixes it for me.

Is there an ETA for this fix landing in a stable version?

[sigmavirus24]

Not really. We'll probably discuss if we want to release 2.6.1 during PyCon or if we want to wait and get a bit more in for a 2.7.0

[aanand]

Just checking in. Any chance of this making it into a release soon? I'm eager to get it downstream into Docker Compose.

[ErikBjare]

Still getting this with Python 3.5, pyinstaller 3.2 and requests 2.7.0. Any update?

Edit: Nevermind, I seem to have had something nasty in my ./build, once removed and rebuilt it seems to work.

[maartenbreddels]

I still have this issue, and my fix was to do sth following in the spec file:

import requests
requests_base_dir = os.path.dirname(requests.__file__)
requests_tree = Tree(requests_base_dir, prefix='requests')
....
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               icon_tree,
               data_tree,
               astropy_tree,requests_tree,
               strip=None,
               upx=True,
               name=vaex.__build_name__)

This was the source is copied, and found, this also works for difficult packages, such as astropy, hope this helps people in the future.