Nuitka: Scons: Argument list too long

[codeyman]

$ python -m nuitka --version
0.6.2
Python: 3.6.7 (default, Feb 26 2019, 11:59:03)
Executable: /home/sridhar/devel/python-3.6.7/malenv3/bin/python
OS: Linux
Arch: x86_64

I'm trying to convert a sklearn/tensorflow based project into a standalone executable. The compilations exits because of the size of the argument list:

$python -m nuitka --follow-imports --standalone --follow-stdlib   -j 6 --show-progress run_server.py
...
Nuitka:INFO:Optimizing module '_codecs_iso2022', 3 more modules to go after that.
Nuitka:INFO:Optimizing module '_codecs_hk', 2 more modules to go after that.
Nuitka:INFO:Optimizing module '_codecs_cn', 1 more modules to go after that.
Nuitka:INFO:Optimizing module '_bz2', 0 more modules to go after that.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py:74' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/ipython_genutils/importstring.py:31' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/ipython_genutils/importstring.py:39' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/jedi/evaluate/compiled/access.py:140' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/joblib/_memmapping_reducer.py:403' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py:965' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py:1000' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/joblib/externals/cloudpickle/cloudpickle.py:1196' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/numpy/core/function_base.py:453' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/numpy/lib/utils.py:366' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/numpy/lib/utils.py:865' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/numpy/lib/utils.py:923' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/pandas/compat/pickle_compat.py:160' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/pygments/formatters/__init__.py:39' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/pygments/lexers/__init__.py:41' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/pygments/styles/__init__.py:64' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/requests/packages.py:7' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/scipy/_lib/_util.py:160' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/scipy/_lib/six.py:74' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/scipy/optimize/optimize.py:3041' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/sklearn/externals/joblib/_memmapping_reducer.py:403' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:944' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:979' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:1179' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/sklearn/externals/six.py:76' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/sklearn/utils/testing.py:663' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/traitlets/utils/importstring.py:34' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/traitlets/utils/importstring.py:42' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/urllib3/packages/six.py:82' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:INFO:Total memory usage before running scons: 4.70 GB (5047599104 bytes):
scons: *** [run_server.dist/run_server] sh: Argument list too long

Is it related to the warnings? How could I handle them? "--include-plugin-*" option is not available in nuitka.

[kayhayen]

This is duplicate of #267 and needs a Scons improvement. On Linux et al the line length limit is way harder to hit, but it's still the same than for MinGW64 on Windows.

To try and fix this, uninstall system scons and debug the matching inline copy of scons (there is one for Python2, one for Python3 of yours, the 3.0.4 one) and find the spot where it is used, then create the @file_that_contains_object_names_but_better_name.txt and use that, and it ought to work.

[tab1tha]

please assign this issue to me @kayhayen

[kayhayen]

Consider yourself assigned @tab1tha I am not sure how to assign to non-members right now.

[kayhayen]

@tab1tha Any progress on this one? Notice that I am soon going to put a change on develop, where practically everything except Python 2.6 (which we don't care much about) uses the 3.0.4 inline copy of scons. So far only Python3.x did that, but Python 2.7 joins the party now. That might affect you, depending on where you test Python version wise.

[tab1tha]

@tab1tha Any progress on this one? Notice that I am soon going to put a change on develop, where practically everything except Python 2.6 (which we don't care much about) uses the 3.0.4 inline copy of scons. So far only Python3.x did that, but Python 2.7 joins the party now. That might affect you, depending on where you test Python version wise.

Nothing helpful yet,I'm afraid. I got a little carried away writing standalone tests.
(Windows 10 OS Python version=3.7.2, Ubuntu OS Python version=3.5.2 : Nuitka version=0.6.2)

[kayhayen]

I have figured out, how to change the Scons linker commandline for at least gcc_mode which is gcc, MinGW64, and clang, to use a temporary file with the argument list.

[kayhayen]

I am re-opening it, because I think that for MSVC the bug should not exist, or definitely need different solution than #267, which I need to investigate another time.

[tab1tha]

Please what is the next step forward here?

[kayhayen]

I pushed pushed the fix to develop, it will be part of the next pre-release package set, and subsequent release.

[kayhayen]

@tab1tha I took on the issue myself, mostly because it is a severe scalability issue to Nuitka, but a problem in Scons code, that I don't think you had a chance to find, nor did I want you to focus on that, as it's no Nuitka knowledge to be gained.

[tab1tha]

Okay. thank you.

[intgogo]

God,same issue here... linux

[codeyman]

I pushed pushed the fix to develop, it will be part of the next pre-release package set, and subsequent release.

I still see the issue on 0.6.3.1

$ python -m nuitka --version
0.6.3.1
Python: 3.6.7 (default, Feb 26 2019, 11:59:03)
Executable: /home/sridhar/devel/python-3.6.7/malenv3/bin/python
OS: Linux
Arch: x86_64

$ python -m nuitka --follow-imports --standalone --follow-stdlib   -j 6 --show-progress run_server.py
...
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/traitlets/utils/importstring.py:42' may require use of '--inc
lude-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/sridhar/devel/python-3.6.7/malenv3/lib/python3.6/site-packages/urllib3/packages/six.py:82' may require use of '--include-plu
gin-directory' or '--include-plugin-files'.
Nuitka:INFO:Total memory usage before running scons: 4.72 GB (5063598080 bytes):
scons: *** [run_server.dist/run_server] sh: Argument list too long

[kayhayen]

The next release meant is 0.6.4, the fix was considered too involved for a hotfix. And that is still not released. Notice the "develop" tag. That means the fix is only there.

[jakams]

Hi,
I seem to have stumbled into this problem as well.
Perhaps any rough estimates on release date of 0.6.4?
Is there a simple way to get 'develop' version 0.6.4 via pip. Else?

[jakams]

found the tar file on the downloads page http://www.nuitka.net/pages/download.html
and installed via pip install Nuitka-0.6.4rc6.tar.gz

also found:
The develop version can be fetched from the Official git repo of Nuitka like this:
pip install -U 'http://nuitka.net/gitweb/?p=Nuitka.git;a=snapshot;h=refs/heads/develop;sf=tgz'
:)

[codeyman]

I tried running Nuitka-0.6.4rc6.tar.gz, but ran into #416. I went to rc3.
It compiles fine, but the generated binary fails to run:

Error processing line 1 of /home/sridhar/devel/py3env/lib/python3.6/site-packages/protobuf-3.6.1-py3.6-nspkg.pth:

  Traceback (most recent call last):
    File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/site.py", line 168, in addpackage
    File "<string>", line 1, in <module>
  KeyError: 'sitedir'

Remainder of file ignored
Traceback (most recent call last):
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/versa-sain.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/__init__.py", line 42, in <module pandas>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/core/api.py", line 26, in <module pandas.core.api>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/core/groupby/__init__.py", line 1, in <module pandas.core.groupby>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/core/groupby/groupby.py", line 37, in <module pandas.core.groupby.groupby>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/core/frame.py", line 87, in <module pandas.core.frame>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/core/generic.py", line 49, in <module pandas.core.generic>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/io/formats/format.py", line 34, in <module pandas.io.formats.format>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/sridhar/devel/ML_WC1/Sain/versa-sain.dist/pandas/io/common.py", line 455, in <module pandas.io.common>
AttributeError: module 'zipfile' has no attribute 'ZipFile'

[kayhayen]

@codeyman create new issues, don't hijack them

[kayhayen]

Released as 0.6.4