Nuitka: When using standalone mode, limit the recursion into imported modules to only those modules that are actually utilized.

[robguinness]

When trying to build a standalone executable, the number of included modules can quickly explode, especially if the application uses large third-party packages. Ideally, Nuitka should only recurse into the modules that are _actually_ used by the application. Currently there is no mechanism to do so, apart from manually specifying --nofollow-import-to flags for all the unwanted modules.

As a simple example for motivation, suppose an application uses the popular machine learning package sklearn (AKA scikit-learn):

from sklearn import datasets
datasets.load_iris()

Even though the above code uses only load_iris(), Nuitka will simply look at the imports for the datasets module, which include the following:

from .base import load_breast_cancer
from .base import load_boston
from .base import load_diabetes
from .base import load_digits
from .base import load_files
from .base import load_iris
from .base import load_linnerud
from .base import load_sample_images
from .base import load_sample_image
from .base import load_wine
from .base import get_data_home
from .base import clear_data_home
from .covtype import fetch_covtype
from .kddcup99 import fetch_kddcup99
from .lfw import fetch_lfw_pairs
from .lfw import fetch_lfw_people
from .twenty_newsgroups import fetch_20newsgroups
from .twenty_newsgroups import fetch_20newsgroups_vectorized
from .mldata import fetch_mldata, mldata_filename
from .openml import fetch_openml
from .samples_generator import make_classification
from .samples_generator import make_multilabel_classification
from .samples_generator import make_hastie_10_2
from .samples_generator import make_regression
from .samples_generator import make_blobs
from .samples_generator import make_moons
from .samples_generator import make_circles
from .samples_generator import make_friedman1
from .samples_generator import make_friedman2
from .samples_generator import make_friedman3
from .samples_generator import make_low_rank_matrix
from .samples_generator import make_sparse_coded_signal
from .samples_generator import make_sparse_uncorrelated
from .samples_generator import make_spd_matrix
from .samples_generator import make_swiss_roll
from .samples_generator import make_s_curve
from .samples_generator import make_sparse_spd_matrix
from .samples_generator import make_gaussian_quantiles
from .samples_generator import make_biclusters
from .samples_generator import make_checkerboard
from .svmlight_format import load_svmlight_file
from .svmlight_format import load_svmlight_files
from .svmlight_format import dump_svmlight_file
from .olivetti_faces import fetch_olivetti_faces
from .species_distributions import fetch_species_distributions
from .california_housing import fetch_california_housing
from .rcv1 import fetch_rcv1

Nuitka will start to recurse into all of these modules, even though base.load_iris is the only one that is actually needed. Furthermore, it will recurse into hundreds of additional other modules after reading the import statements of the above scikit-learn modules.

Referencing a short conversation with @kayhayen on Gitter:

The idea here would be to use the import tracing hidden in hints.py in nuitka git checkout, run your program with it, collect all imports, and then to make a change to Nuitka, where --no-follow is added to the command line with --follow-to=all_the,names,you,saw,being,actually,imported
That way, sort of making it possible to avoid packages that never even do get imported at least.

I will start to look into this technique. If others are interested to contribute, please let me know.

[sannanansari]

It is interesting. I want to contribute. kayhayen had simplified thing. I am confused how to import
trace?

[kayhayen]

Consider this code:
https://github.com/Nuitka/Nuitka/blob/aa474c92dca350068f23244b9b56c830d7da08e0/lib/hints.py#L61

That is the import tracing we are talking about. If you collect the names, you know that these names are the only ones relevant.

In the next step, you could identify modules, that are only imported, but never accessed, by instead of returning a module, returning a non-module object that wraps the module. It forwards attribute lookups to the real module, overloading __getattr__ and records if that happens or not.

For these more unused modules, you could try to compile them only to bytecode.

There is a PyPI package called demandimport which was once extraced from Mercurial, which does the kind of module wrapping needed. They did it to accelerate Python program startup, by avoiding the import of unused modules. We can lend the code that does the wrapping.

Then running your program with that kind of collection, and exiting it, ought to produce a file with the data collected, and we need to turn that to command line options and run Nuitka with it.

[kayhayen]

The next step would be, to monkey patch loaded module functions with a wrapper that registers their usage, and then to know, which functions are used after the initial module load at all. If we ever manage to teach Nuitka (well possible) to compile a module where some functions become C code, and some stay bytecode, we could e.g. not compile the 1000 functions of a module, but only the 2 used ones.

That ought to result is an even more slim, where hybrid modules, bytecode and C code mixed, can be used and base that on the measurement of the example run of a Python program.

[kayhayen]

I am seeing this as a pre-cursor to PGO (profile guided optimization), which will then also record types used in a program, and generate optimized C code with a fallback.

[robguinness]

@sannanansari , great that you can contribute! I'm not sure how quickly I can get to this, as it seems a bit more complicated than I originally thought. But I am still definitely interested in working on it. I will let you know if I make any progress.

[JorjMcKie]

@robguinness - this is a topic that interests me too and I played around a bit.
Here is what I have working already:

• a script that runs a program under control of hints.py and collects its output in an array of called things (saved as a JSON file)
• a nuitka compile script which, reads the JSON file for the program and generates Nuitka options from it, then invokes Nuitka with these options.

This is the output of a small program using some numpy functions. Sorry for the long printout, but note that every --recurse-to= statement has been generated from the JSON file np-test.json belonging to np-test.py ... including invocation of the numpy plugin.

As you can see, the invoking ccommand line can become very small using this technique.

D:\Jorj\Desktop\Develop\Nuitka>python nuitka-incl.py np-test.py
NUITKA is compiling 'np-test.py' with these options:
 --standalone
 --mingw64
 --python-flag=nosite
 --remove-output
 --experimental=use_pefile
 --disable-dll-dependency-cache
 --recurse-none
 --recurse-to=__future__
 --recurse-to=_ctypes
 --recurse-to=abc
 --recurse-to=argparse
 --recurse-to=ast
 --recurse-to=collections
 --recurse-to=collections.abc
 --recurse-to=contextlib
 --recurse-to=copyreg
 --recurse-to=ctypes
 --recurse-to=datetime
 --recurse-to=decimal
 --recurse-to=difflib
 --recurse-to=encodings
 --recurse-to=fnmatch
 --recurse-to=functools
 --recurse-to=gettext
 --recurse-to=glob
 --recurse-to=heapq
 --recurse-to=io
 --recurse-to=locale
 --recurse-to=logging
 --recurse-to=ntpath
 --recurse-to=numbers
 --recurse-to=numpy
 --recurse-to=numpy.__config__
 --recurse-to=numpy._globals
 --recurse-to=numpy._pytesttester
 --recurse-to=numpy.compat
 --recurse-to=numpy.compat._inspect
 --recurse-to=numpy.compat.py3k
 --recurse-to=numpy.core
 --recurse-to=numpy.core._dtype
 --recurse-to=numpy.core._internal
 --recurse-to=numpy.core._multiarray_tests
 --recurse-to=numpy.core._multiarray_umath
 --recurse-to=numpy.core._string_helpers
 --recurse-to=numpy.core._type_aliases
 --recurse-to=numpy.core.arrayprint
 --recurse-to=numpy.core.defchararray
 --recurse-to=numpy.core.einsumfunc
 --recurse-to=numpy.core.fromnumeric
 --recurse-to=numpy.core.function_base
 --recurse-to=numpy.core.getlimits
 --recurse-to=numpy.core.info
 --recurse-to=numpy.core.machar
 --recurse-to=numpy.core.memmap
 --recurse-to=numpy.core.multiarray
 --recurse-to=numpy.core.numeric
 --recurse-to=numpy.core.numerictypes
 --recurse-to=numpy.core.overrides
 --recurse-to=numpy.core.records
 --recurse-to=numpy.core.shape_base
 --recurse-to=numpy.core.umath
 --recurse-to=numpy.fft
 --recurse-to=numpy.fft.fftpack
 --recurse-to=numpy.fft.helper
 --recurse-to=numpy.fft.info
 --recurse-to=numpy.lib
 --recurse-to=numpy.lib._datasource
 --recurse-to=numpy.lib._iotools
 --recurse-to=numpy.lib._version
 --recurse-to=numpy.lib.arraypad
 --recurse-to=numpy.lib.arraysetops
 --recurse-to=numpy.lib.arrayterator
 --recurse-to=numpy.lib.financial
 --recurse-to=numpy.lib.function_base
 --recurse-to=numpy.lib.histograms
 --recurse-to=numpy.lib.index_tricks
 --recurse-to=numpy.lib.info
 --recurse-to=numpy.lib.mixins
 --recurse-to=numpy.lib.nanfunctions
 --recurse-to=numpy.lib.npyio
 --recurse-to=numpy.lib.polynomial
 --recurse-to=numpy.lib.shape_base
 --recurse-to=numpy.lib.stride_tricks
 --recurse-to=numpy.lib.twodim_base
 --recurse-to=numpy.lib.type_check
 --recurse-to=numpy.lib.ufunclike
 --recurse-to=numpy.lib.utils
 --recurse-to=numpy.linalg
 --recurse-to=numpy.linalg.info
 --recurse-to=numpy.linalg.linalg
 --recurse-to=numpy.ma
 --recurse-to=numpy.ma.core
 --recurse-to=numpy.ma.extras
 --recurse-to=numpy.matrixlib
 --recurse-to=numpy.matrixlib.defmatrix
 --recurse-to=numpy.polynomial
 --recurse-to=numpy.polynomial._polybase
 --recurse-to=numpy.polynomial.chebyshev
 --recurse-to=numpy.polynomial.hermite
 --recurse-to=numpy.polynomial.hermite_e
 --recurse-to=numpy.polynomial.laguerre
 --recurse-to=numpy.polynomial.legendre
 --recurse-to=numpy.polynomial.polynomial
 --recurse-to=numpy.random.mtrand
 --recurse-to=numpy.testing
 --recurse-to=numpy.testing._private
 --recurse-to=numpy.testing._private.nosetester
 --recurse-to=numpy.testing._private.utils
 --recurse-to=numpy.version
 --recurse-to=operator
 --recurse-to=os
 --recurse-to=pathlib
 --recurse-to=posixpath
 --recurse-to=pprint
 --recurse-to=re
 --recurse-to=shutil
 --recurse-to=signal
 --recurse-to=stat
 --recurse-to=tempfile
 --recurse-to=textwrap
 --recurse-to=threading
 --recurse-to=traceback
 --recurse-to=types
 --recurse-to=unittest
 --recurse-to=unittest.case
 --recurse-to=unittest.loader
 --recurse-to=unittest.main
 --recurse-to=unittest.result
 --recurse-to=unittest.runner
 --recurse-to=unittest.signals
 --recurse-to=unittest.suite
 --recurse-to=unittest.util
 --recurse-to=urllib.parse
 --recurse-to=warnings
 --recurse-to=weakref
 --enable-plugin=numpy-plugin

Nuitka:WARNING:Didn't recurse to '__future__', apparently not used.
Nuitka:WARNING:Didn't recurse to 'argparse', apparently not used.
Nuitka:WARNING:Didn't recurse to 'datetime', apparently not used.
Nuitka:WARNING:Didn't recurse to 'difflib', apparently not used.
Nuitka:WARNING:Didn't recurse to 'encodings', apparently not used.
Nuitka:WARNING:Didn't recurse to 'fnmatch', apparently not used.
Nuitka:WARNING:Didn't recurse to 'gettext', apparently not used.
Nuitka:WARNING:Didn't recurse to 'heapq', apparently not used.
Nuitka:WARNING:Didn't recurse to 'logging', apparently not used.
Nuitka:WARNING:Didn't recurse to 'numpy.core._multiarray_tests', apparently not used.
Nuitka:WARNING:Didn't recurse to 'posixpath', apparently not used.
Nuitka:WARNING:Didn't recurse to 'signal', apparently not used.
Nuitka:WARNING:Didn't recurse to 'stat', apparently not used.
Nuitka:WARNING:Didn't recurse to 'traceback', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.loader', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.main', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.result', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.runner', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.signals', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.suite', apparently not used.
Nuitka:WARNING:Didn't recurse to 'unittest.util', apparently not used.
Nuitka:INFO: Now copying extra binaries from 'numpy' installation:
Nuitka:INFO: c:\users\jorj\appdata\local\programs\python\python37\lib\site-packages\numpy\.libs\libopenblas.ipbc74c7kurv7cb2pkt5z5fnr3sibv4j.gfortran-win_amd64.dll
Nuitka:INFO: Copied 1 binary.

D:\Jorj\Desktop\Develop\Nuitka>

[JorjMcKie]

Did some further experiments, also including demandimport.
Here are some intermediate findings:

demandimport

Basically creates 2 output categories: "delayed" and "triggered".

• Delayed imports can be viewed as the imports the script is encountering while executing. It is a subset of the imports that Nuitka finds during syntax inspection. demandimport initially prevents their execution and stores them in an array instead.
• Once the program actually needs a delayed import (e.g. via a method call) then the previously delayed import happens before this. This is when a trigger event gets reported.

This logic does not / cannot cover all cases (as is also documented by demandimport):

• "*" and __import__ imports will not be delayed, from a import b will only delay b, but a will be immediately be imported, etc.
• correspondingly, triggers will not be reported in cases where there was no previous delay. For example consider a script that uses PIL like from PIL import Image. Then you will find PIL.Import in the "delay" and "trigger" parts of the report (if Image is actually ever used), but you will not find PIL in either of them.

Additional complications also exist: some harmless looking imports must never be delayed (see their documentation). What they are is package-specific. It seemed to me that there is some kind of match with our list implemented in ImplicitImports.py. So some hope for relief here.

hints.py

Basically reports all import events. So this is a (usually true) superset of the "trigger" events reported by demandimport. For example, it will report PIL and PIL.Import from the above example.

Combining both

I tried combining both of these reports. Basically produced 2 arrays:

• "triggered" = hints.py output
• "never triggered": "delayed" output minus "triggered"

The "triggered" list can be used to generate --recurse-to= parameters, and resp. "never triggered" for --recurse-not-to=.

Issues

Following the idea
"use --recurse-none and then append --recurse-to= for every item in triggered",
worked in the above PIL example.

However:
PIL has a plethora of submodules apart from Image, among them ImageTk, ImageQt etc.
To use any of them, it must be imported like from PIL import ImageQt.
But Nuitka will include all of them, because of my generated (and required!) --recurse-to=PIL! This will obviously still lead to a far-too-large standalone output (including the big Qt modules in this example).
As yet, I have no idea how to get around this (without again pouring a lot of special package knowledge into the process).
Anyone with a better idea?

[JorjMcKie]

Currently making some progress using a promising different approach:

• Create an array of calls to modules / functions the script is actually making by using hints.py. Forget about demandimport.
• Compile the script by passing this array to a user plugin (I am currently calling it hinted-mods.py) which in turn uses the plugin method onModuleEncounter to accept or decline every single module based on whether or not that module appears in this calls array.

This plugin hinted-mods.py is no big deal. The crucial point is the correct content of the calls array created via hints.py.
This is currently causing me headaches, because it must reflect all the forms that an import statement can take on. But I think it is possible.

Another attractive aspect of this approach:
The need for most or many of the standard plugins can be detected automatically. Based on the calls array, plugin hinted-mods.py can dynamically generate --enable-plugin=, --disable-plugin= and other options. So for many situations, a standalone compile statement of the following form will be sufficient:
python -m nuitka --standalone --user-plugin=hinted-mods.py myscript.py

[JorjMcKie]

Just finished a first version the stuff mentioned in my previous post.
It consists of the following components:

• a (slightly) modified version of hints.py
• script get-hints.py which reads the to-be-examined script and and runs it under control of hints.py. It collects the output and creates a JSON file script.json for the Python script.py. The JSON file is a sorted array of all imports made by the script during the test run. Duplicates have been removed.
• script nuitka-hints.py, which invokes the Nuitka compiler, passing any appropriate or site / specific options to it (among them --standalone), and also --user-plugin=hinted-mods.py=script.json. File script.json is the file from the previous test run.
• hinted-mods.py is a user plugin, which in its __init__ reads the JSON file and adds some options in response to the JSON contents, like enabling plugins. Its main logic is contained in its onModuleEncounter method, where it checks for each module handed in by the Nuitka compiler, whether it is part of the JSON file.

@robguinness, @kayhayen, @sannanansari:
I am going to put a ZIP file of these components on the utilities repo. Maybe you are interested in trying something out yourself.
Here is a sample output for a script that uses PIL / Pillow to open an image file:

D:\Jorj\Desktop\Develop\Nuitka>python nuitka-hints.py pil-test.py
NUITKA is compiling 'pil-test.py' with these options:
 --standalone
 --mingw64
 --python-flag=nosite
 --remove-output
 --experimental=use_pefile
 --disable-dll-dependency-cache
 --user-plugin=hinted-mods.py=pil-test.json

Nuitka:INFO: hinted-mods.py is adding the following options:
Nuitka:INFO: --disable-plugin=numpy-plugin
Nuitka:INFO: --recurse-not-to=numpy
Nuitka:INFO: --disable-plugin=tk-plugin
Nuitka:INFO: --recurse-not-to=PIL.ImageTk
Nuitka:INFO: --disable-plugin=qt-plugins
Nuitka:INFO: --recurse-not-to=PIL.ImageQt
Nuitka:INFO:User plugin 'hinted-mods.py' loaded.
Nuitka:INFO: excluding unreferenced tempfile
Nuitka:INFO: excluding unreferenced PIL.PyAccess
Nuitka:INFO: excluding unreferenced PIL.ImageFilter
Nuitka:INFO: excluding unreferenced PIL.ImageQt
Nuitka:INFO: excluding unreferenced PIL.ImageQt
Nuitka:INFO: excluding unreferenced PIL.ImageQt
Nuitka:INFO: excluding unreferenced PIL.ImageQt
Nuitka:INFO: excluding unreferenced PIL.ImageShow
Nuitka:INFO: excluding unreferenced colorsys
Nuitka:INFO: excluding unreferenced colorsys
Nuitka:INFO: excluding unreferenced random
Nuitka:INFO: excluding unreferenced subprocess
Nuitka:INFO: excluding unreferenced tempfile
Nuitka:INFO: excluding unreferenced subprocess
Nuitka:INFO: excluding unreferenced PIL.MpoImagePlugin
Nuitka:INFO: excluding unreferenced copy
Nuitka:INFO: excluding unreferenced subprocess
Nuitka:INFO: excluding unreferenced _cffi_backend
Nuitka:INFO: excluding unreferenced cffi.cparser
Nuitka:INFO: excluding unreferenced cffi.verifier
Nuitka:INFO: excluding unreferenced sysconfig
Nuitka:INFO: excluding unreferenced distutils.dir_util
Nuitka:INFO: excluding unreferenced cffi.recompiler
Nuitka:INFO: excluding unreferenced cffi.recompiler
Nuitka:INFO: excluding unreferenced cffi.recompiler
Nuitka:INFO: excluding unreferenced cffi.recompiler
Nuitka:INFO: excluding unreferenced ctypes.util
Nuitka:WARNING:Unresolved '__import__' call at 'C:\Users\Jorj\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\Image.py:428' may require use of '--include-plugin-directory' or '--include-plugin-files'.

[JorjMcKie]

@robguinness - next thing I will try is your sklearn example ...

[kayhayen]

@JorjMcKie just what i was thinking, plugins have full control, not restrained by command line.

[JorjMcKie]

Uploaded the ZIP file mentioned to the utilities repo.

[JorjMcKie]

The whole approach has still some open points and rough edges:

1. _Implicit imports_ are mostly undetected - even by hints.py. So they are missing in the call protocol. This gap needs to be filled by the user plugin. Found a way how to do that and testing it.
2. Automatic _standard plugin detection_ is a nice thing and should be possible for all of them - so far only numpy and tkinter are working. BTW @kayhayen, following your comment on naming plugins, I think I should rename numpy-plugin to just numpy, agreed?
3. Somewhat unrelated: I need to create a scipy plugin. This package also needs (Windows only?) undetectable extra shared libs much like numpy. I currently think I will integrate it in the numpy plugin - maybe with an option-based invocation. Both packages are often used together anyway, and both are also referencing each other internally.
4. When trying @robguinness sklearn mini-script, it seemed, that this also requires a bunch of extra implicit imports at least.

But, before this sounds to negative, here is what already is working:

• tests are running successfully for tkinter, numpy, Qt, wxPython, PIL, and scipy (except for the extra DLLs on Windows)
• no extra command line parameter is needed in any of these cases (Qt coming soon - under construction): they all work with python nuitka-hints.py yourscript.py.

[imrickywong]

any progress?

[JorjMcKie]

Yes, actually quite a lot. Have a look in this folder.
You will find a fairly full description of the process and its requirements. Here is an overview in a nutshell:

• Trace the imports of a script with get-hints.py. This will create a dict containing the calls and the invoked modules. Output is stored in a JSON file.
• After having reached some confidence level in the previous step, compile the script with appropriate parameters. This is a standalone compile, which makes use of the generated JSON dictionary. The basic logic is: --recurse-none followed by a sequence of --recurse-to=.... The latter being derived from the modules that were actually named during the tracing run.

During compilation, you will see a number of information and warning messages. Currently, these are numerous and we may consider switching them off or redirecting them to some compile log.

The above scripts (import tracing and hinted compile) work and have been tested on Windows and Linux so far. They require no platform-specific adaptions. But please do have a look into the options, which the compile script nuitka-hints.py will pass on to the Nuitka compiler. You may need to adapt some to your environment (compiler choice, etc.). Ideally, your compile command will look like just this:

python nuitka-hints.py yourscript.py

During compile, the required standard plugins (like tk-inter, numpy, qt-plugins, enum, etc.) are automatically identified and the appropriate --enable-plugin=... options are then generated accordingly on the fly.

Obviously your Nuitka version must contain those plugins - nothing new here.

I have developed some new standard plugins, which have not yet found their way into Nuitka. My list of standard plugins looks like so:

ConsiderPyLintAnnotationsPlugin.py
DataFileCollectorPlugin.py
EnumPlugin.py
GeventPlugin.py (*)
ImplicitImports.py
MultiprocessingPlugin.py
NumpyPlugin.py
PmwPlugin.py
PySidePyQtPlugin.py
SklearnPlugin.py (*)
TensorflowPlugin.py (*)
TkinterPlugin.py
TorchPlugin.py (*)

The (*) marks indicate plugins which I have recently developed (gevent and tensorflow still under construction). You will need to wait for the next Nuitka version to get those.

[JorjMcKie]

@RickyWong33
You are welcome to try out the above. If you want to try out some of the plugins marked (*), drop me a note and I will send you a ZIP file.

[kayhayen]

@JorjMcKie as long as you have the commits relatively clean, how about you create your own branch "jorj" directly in the Nuitka repository, and then I can cherry-pick or comment there? Just would need to rebase to develop if not done already. I can sort things out, no problem.

[JorjMcKie]

ok, will do. Branch is already created. Just need to upload it.

[robguinness]

Hi all, any update on this?

[JorjMcKie]

@robguinness - welcome back!

Absolutely yes! There is this repo / folder.
It also has explanations on how all this works. I think it is best to give it a try ...

[JorjMcKie]

To avoid any misconception:
Hinting compilation does a --recurse-none, followed by --recurse-to for all modules for which an import is actually executed in runtime.
In addition, all standard plugins are automatically enabled based on that information.

Hinted compilation cannot / does not detect, if a program imports something and then never uses it ...

[robguinness]

Great to hear...thanks!