Nuitka: Would it be possible to enable cross-compilation ?

[davidovich]

I would like some guidance on what would be necessary to use mingw-64 to build a Windows executable from a linux machine.

We have many go builds that enjoy a simple linux-based ci builds to produce Windows executables. We would probably be interested in the same for nuitka --standalone from a linux machine.

I have come across this: https://www.freelists.org/post/nuitka-dev/Patch-Added-crosscompilation-support-to-nuitka, but it seemingly has not received any feedback. This patch seems to point to a relatively easy thing to support.

Am I correct in my assumptions ? What would be needed for such a use-case ?

[kayhayen]

Nuitka uses the running Python version to determine things at compile time. That does not tie in with cross compilation at all. What would be needed is to run a slave Python instance to make computations, cache its results, and then use that cache for re-compilation, typically never needing it again. This is an effort, I doubt anybody will do. I am not interested for now.

[frmdstryr]

I'm also interested in using this to cross compile python distributions.

[kayhayen]

At one time I was using CPython under Linux using "wine". If the MinGW64 compiler worked with wine, then you would have have to invoke it that way.

I would appreciate if somebody tried installing wine, CPython for Windows into it, MinGW64 (32 bits or is wine 64 bits good already, maybe try both) for Windows into it, then run a compilation with it, and report. Nuitka should see os.name == "nt" and behave the same. Standalone mode, and wheels might both work, or not. Let us know. :-)

[kayhayen]

I was then however using gcc cross compiler, but that won't work anymore, I gave up on that. But what I wrote above, ought to work, maybe with a little love. I would even use it myself.

[xgid]

Hi there, I would also love to see that working! I have no time right now to look into it, but you may be interested in starting with this already existing Docker Image for your tests:

https://hub.docker.com/r/gsauthof/fedora-mingw64-devel/

[lyssdod]

Is it possible to add an option to output just optimised C sources, like in Cython? This would solve cross compilation problem (assuming being in charge of compiler flags)

[kayhayen]

@lyssdod This is exactly where Nuitka is supposed to be different than Cython, to Nuitka the C code is machine code replacement only, could also be LLVM bytecode, but it is supposed to not be portable to another machine.

Since the Nuitka design doesn't lend to these things, and it's moving further away from it, I do not intend to work in this direction, other than the approach outlined above, where a natively running Python is used for compile time evaluations on the target platform.

[lyssdod]

@kayhayen I'm pretty sure it's still technically possible by overriding $CC and $LDFLAGS (as in aforementioned patch in the mailing list), just in many cases it's not feasible to use target hardware to perform compilation directly...

[kayhayen]

@lyssdod You don't understand that e.g. os.path will be from the compiling platform, resulting in os.path.join("a","b") to be evaluated withposixpathtoa/b, wherentpathwould createab` on the resulting platform.

With improved optimization, chances for a working binary will quickly decrease.

My intended solution is to execute compile time evaluations like this on a Python process running on the target platform (preserving results in a cache), so to cross compile, Nuitka would then run on both Linux and Windows optionally, and then create C sources for Windows, which your cross compiler could handle.

Running a python on an ARM arch could work the same.

This is heavy change to Nuitka though, and not going to be undertaken for some time. It could be material got a GSoC student though.

[mcondarelli]

I would also be interested in this, but from a completely different vantage point:
I work in the embedded systems area, mainly using Buildroot to prepare my root-filesystems (sometimes I'm forced to use Yocto, though).
In such environments I already have CPython both as target program (running on target, e.g.: an ARM processor) and as a "host utility" (running on host (x86_64), but behaving as if on target (ARM)).
Both the development and target are Linux, so there should be a much better compatibility in "os" package (and friends).
Aim would be to cross-compile to executable, possibly doing completely without the python interpreter on target.
Do You think this would be feasible?

[kayhayen]

@mcondarelli that would be perfect

[mcondarelli]

@kayhayen ... meaning I can try to use Nuitka as-is?

Are there any special caveats to compilation/usage of Nuitka in this environment?
Is it enough to have my cross-CPython earlier in PATH or should I point Nuitka to it, in some way?

[kayhayen]

@mcondarelli no it means you have all the pre-conditions to change Nuitka so it works.

[mcondarelli]

@kayhayen Ok, fair enough.

Do You have any special recommendations?

Main reason why I need (?) this is to shorten startup time of my application: Some modules (notably requests and jinja2) take a very long time (>20s) just for the initial "import". After that they run fast enough, but initialization add-up, so my application is "alive" after about one minute from power-on, which is unacceptable.
Can I hope to shorten this kind of delays with Nuitka?

[kayhayen]

@mcondarelli I have not yet measure start up time with Nuitka compiled binary, so far I have only aimed at acceleration.

[mcondarelli]

A first, very crude, attempt at cross compilation bombs with:

PYTHONPATH="/home/mcon/vocore/Buildroot/output/target/usr/lib/python3.6/" cd /home/mcon/vocore/Buildroot/output/target && /home/mcon/vocore/Buildroot/output/host/bin/python3.6 /home/mcon/vocore/Buildroot/buildroot/support/scripts/pycompile.py  usr/lib/python3.6
Cannot compile usr/lib/python3.6/site-packages/nuitka/build/inline_copy/lib/scons-2.3.2/SCons/Action.py
Traceback (most recent call last):
  File "/home/mcon/vocore/Buildroot/output/host/lib/python3.6/py_compile.py", line 125, in compile
    _optimize=optimize)
  File "<frozen importlib._bootstrap_external>", line 713, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "usr/lib/python3.6/site-packages/nuitka/build/inline_copy/lib/scons-2.3.2/SCons/Action.py", line 677
    except EnvironmentError, e:
                           ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mcon/vocore/Buildroot/buildroot/support/scripts/pycompile.py", line 69, in <module>
    compileall.compile_dir(args.target, force=args.force, quiet=ReportProblem())
  File "/home/mcon/vocore/Buildroot/output/host/lib/python3.6/compileall.py", line 90, in compile_dir
    legacy, optimize):
  File "/home/mcon/vocore/Buildroot/output/host/lib/python3.6/compileall.py", line 152, in compile_file
    if quiet >= 2:
  File "/home/mcon/vocore/Buildroot/buildroot/support/scripts/pycompile.py", line 52, in __ge__
    return not self < other
  File "/home/mcon/vocore/Buildroot/buildroot/support/scripts/pycompile.py", line 26, in operator
    raise value
  File "/home/mcon/vocore/Buildroot/output/host/lib/python3.6/compileall.py", line 149, in compile_file
    optimize=optimize)
  File "/home/mcon/vocore/Buildroot/output/host/lib/python3.6/py_compile.py", line 129, in compile
    raise py_exc
py_compile.PyCompileError:   File "usr/lib/python3.6/site-packages/nuitka/build/inline_copy/lib/scons-2.3.2/SCons/Action.py", line 677
    except EnvironmentError, e:
                           ^
SyntaxError: invalid syntax

What am I doing wrong?

Shouldn't that be except EnvironmentError as e:?

[kayhayen]

There is two inline copies of scons, one for Python2 and one for Python3. You can't run the former with Python3 at all, so do't do it. There is a 3.0.1 or so as an inline copy too.

[mcondarelli]

I tries a second time to use Nuitka in a cross-compilation system (Buildroot) and it failed with the following output (I dumped environment because the build system sets a lot of variables).
I need guidance to track this down.

============= THIS IS FOR DEBUG =============================
/home/mcon/vocore/Buildroot/output/host/bin/python --version
Python 3.6.3
set
BASE_DIR=/home/mcon/vocore/Buildroot/output
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="4" [2]="23" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.4.23(1)-release'
BINARIES_DIR=/home/mcon/vocore/Buildroot/output/images
BR2_CONFIG=/home/mcon/vocore/Buildroot/output/.config
BR2_EXTERNAL_VOCORE2_DESC='VoCore2 board setup'
BR2_EXTERNAL_VOCORE2_PATH=/home/mcon/vocore/Buildroot/input
BR2_REPRODUCIBLE=
BR2_VERSION=2018.08.2
BR2_VERSION_FULL=2018.08.2-dirty
BR_COMPILER_PARANOID_UNSAFE_PATH=enabled
BR_NO_CCACHE=
BR_NO_CHECK_HASH_FOR='  canfestival-7740ac6fdedc23e1ed6908d3d7db54833c88572b.tar.gz dvb-apps-3d43b280298c39a67d1d889e01e173f52c12da35.tar.gz fis-2892.tar.gz open2300-12.tar.gz opentyrian-9c9f0ec3532b.tar.gz python-nfc-212.tar.gz pygame-d61ea8eabd56.tar.gz'
BUILDROOT_CONFIG=/tmp/deprecated/The-BUILDROOT_CONFIG-environment-variable-was-renamed-to-BR2_CONFIG
BZR=bzr
CDPATH=
CINNAMON_VERSION=3.8.8
COLORTERM=truecolor
CONFIG_SHELL=/bin/bash
CVS=cvs
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DESKTOP_SESSION=lightdm-xsession
DIRSTACK=()
DISPLAY=:0
EUID=1000
GDMSESSION=lightdm-xsession
GIT=git
GJS_DEBUG_OUTPUT=stderr
GJS_DEBUG_TOPICS='JS ERROR;JS LOG'
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/81977ca4_fe8a_4164_a5ad_8a7f32c368a8
GNOME_TERMINAL_SERVICE=:1.86
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
GROUPS=()
GTK_MODULES=gail:atk-bridge
GTK_OVERLAY_SCROLLING=0
HG=hg
HISTCONTROL=ignoreboth
HOME=/home/mcon
HOSTAR=/usr/bin/ar
HOSTARCH=x86_64
HOSTAS=/usr/bin/as
HOSTCC=/usr/bin/gcc
HOSTCC_NOCCACHE=/usr/bin/gcc
HOSTCFLAGS=
HOSTCXX=/usr/bin/g++
HOSTCXX_NOCCACHE=/usr/bin/g++
HOSTLD=/usr/bin/ld
HOSTNAME=cinderella
HOSTTYPE=x86_64
HOST_DIR=/home/mcon/vocore/Buildroot/output/host
IFS=$' \t\n'
KBUILD_VERBOSE=0
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LOCALFILES=cp
LOGNAME=mcon
LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
MACHTYPE=x86_64-pc-linux-gnu
MAKEFLAGS=' --no-print-directory -- '
MAKELEVEL=2
MAKE_TERMERR=/dev/pts/1
MAKE_TERMOUT=/dev/pts/1
MC_SID=32024
MC_TMPDIR=/tmp/mc-mcon
MFLAGS=--no-print-directory
OLDPWD=/home/mcon/vocore/Buildroot
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
PERL=/usr/bin/perl
PERL5LIB=/home/mcon/vocore/Buildroot/output/host/lib/perl
PPID=21675
PS4='+ '
PWD=/home/mcon/vocore/Buildroot/buildroot
Q=@
QT_ACCESSIBILITY=1
SCP=scp
SESSION_MANAGER=local/cinderella:@/tmp/.ICE-unix/4154,unix/cinderella:/tmp/.ICE-unix/4154
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=3
SSH_AGENT_PID=4190
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
STAGING_DIR=/home/mcon/vocore/Buildroot/output/host/mipsel-buildroot-linux-musl/sysroot
SVN=svn
TARGET_DIR=/home/mcon/vocore/Buildroot/output/target
TERM=xterm-256color
UID=1000
USER=mcon
VTE_VERSION=0.48.3
WGET='wget --passive-ftp -nd -t 3'
XAUTHORITY=/home/mcon/.Xauthority
XDG_CURRENT_DESKTOP=X-Cinnamon
XDG_DATA_DIRS=/usr/share/gnome:/usr/local/share/:/usr/share/
XDG_GREETER_DATA_DIR=/var/lib/lightdm/data/mcon
XDG_RUNTIME_DIR=/run/user/1000
XDG_SEAT=seat0
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_SESSION_CLASS=user
XDG_SESSION_DESKTOP=lightdm-xsession
XDG_SESSION_ID=2
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
XDG_SESSION_TYPE=x11
XDG_VTNR=7
_=/usr/bin/make
============= END OF DEBUG OUTPUT ===========================
( cd /home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client && /home/mcon/vocore/Buildroot/output/host/bin/python  -m nuitka --follow-imports --standalone --plugin-enable=pylint-warnings stream.py )
Nuitka:WARNING:There is a problem with detecting imports, CPython said:
b'import _frozen_importlib # frozen'
b'import _imp # builtin'
b'import sys # builtin'
b"import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>"
b"import '_thread' # <class '_frozen_importlib.BuiltinImporter'>"
b"import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>"
b"import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>"
b"import '_io' # <class '_frozen_importlib.BuiltinImporter'>"
b"import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>"
b"import 'posix' # <class '_frozen_importlib.BuiltinImporter'>"
b"import _thread # previously loaded ('_thread')"
b"import '_thread' # <class '_frozen_importlib.BuiltinImporter'>"
b"import _weakref # previously loaded ('_weakref')"
b"import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>"
b'# installing zipimport hook'
b"import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>"
b'# installed zipimport hook'
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/__init__.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/__init__.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/__init__.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/codecs.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/codecs.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/codecs.pyc'"
b"import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>"
b"import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61ff550>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/aliases.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/aliases.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/aliases.pyc'"
b"import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6213f98>"
b"import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61ff0b8>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/utf_8.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/utf_8.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/utf_8.pyc'"
b"import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6227da0>"
b"import '_signal' # <class '_frozen_importlib.BuiltinImporter'>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/latin_1.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/latin_1.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/latin_1.pyc'"
b"import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a622d8d0>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/io.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/io.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/io.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/abc.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/abc.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/abc.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/_weakrefset.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/_weakrefset.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/_weakrefset.pyc'"
b"import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6232828>"
b"import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a622deb8>"
b"import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a622db00>"
b'Python 3.6.3 (default, Dec 20 2018, 10:12:44) '
b'[GCC 8.2.0] on linux'
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/ascii.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/ascii.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/ascii.pyc'"
b"import 'encodings.ascii' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61c4358>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/base64_codec.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/base64_codec.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/base64_codec.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/base64.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/base64.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/base64.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/re.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/re.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/re.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/enum.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/enum.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/enum.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/types.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/types.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/types.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/functools.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/functools.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/functools.pyc'"
b"import '_functools' # <class '_frozen_importlib.BuiltinImporter'>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/collections/__init__.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/collections/__init__.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/collections/__init__.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/_collections_abc.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/_collections_abc.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/_collections_abc.pyc'"
b"import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6179358>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/operator.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/operator.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/operator.pyc'"
b"import '_operator' # <class '_frozen_importlib.BuiltinImporter'>"
b"import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a618d128>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/keyword.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/keyword.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/keyword.pyc'"
b"import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a613a4e0>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/heapq.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/heapq.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/heapq.pyc'"
b"# extension module '_heapq' loaded from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so'"
b"# extension module '_heapq' executed from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so'"
b"import '_heapq' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f55a6140278>"
b"import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a613ad30>"
b"import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/reprlib.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/reprlib.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/reprlib.pyc'"
b"import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6140358>"
b"import '_collections' # <class '_frozen_importlib.BuiltinImporter'>"
b"import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6156908>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/weakref.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/weakref.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/weakref.pyc'"
b"import 'weakref' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6156d30>"
b"import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61f1128>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/collections/abc.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/collections/abc.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/collections/abc.pyc'"
b"import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61f1f98>"
b"import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61e83c8>"
b"import 'enum' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61d5278>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_compile.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_compile.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_compile.pyc'"
b"import '_sre' # <class '_frozen_importlib.BuiltinImporter'>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_parse.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_parse.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_parse.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_constants.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_constants.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/sre_constants.pyc'"
b"import 'sre_constants' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a60f2940>"
b"import 'sre_parse' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a60ed470>"
b"import 'sre_compile' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61f3c18>"
b"import '_locale' # <class '_frozen_importlib.BuiltinImporter'>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/copyreg.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/copyreg.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/copyreg.pyc'"
b"import 'copyreg' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6104780>"
b"import 're' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61caef0>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/struct.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/struct.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/struct.pyc'"
b"# extension module '_struct' loaded from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so'"
b"# extension module '_struct' executed from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so'"
b"import '_struct' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f55a6104c88>"
b"import 'struct' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a6104a90>"
b"# extension module 'binascii' loaded from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so'"
b"# extension module 'binascii' executed from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so'"
b"import 'binascii' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f55a6104da0>"
b"import 'base64' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61c4978>"
b"import 'encodings.base64_codec' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a61c44a8>"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/big5.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/big5.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/big5.pyc'"
b'# /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/cp437.pyc matches /home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/cp437.py'
b"# code object from '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/cp437.pyc'"
b"import 'encodings.cp437' # <_frozen_importlib_external.SourceFileLoader object at 0x7f55a610e400>"
b"# zipimport: found 158 names in '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/setuptools-38.4.0.post20181220-py3.6.egg'"
b'Traceback (most recent call last):'
b'  File "/tmp/tmps2k6bggl", line 1, in <module>'
b"    import sys; sys.path = ['/home/mcon/vocore/Buildroot/output/host/lib/python36.zip', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/lib-dynload', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/setuptools-38.4.0.post20181220-py3.6.egg', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/cffi-1.11.2-py3.6-linux-x86_64.egg', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Cython-0.27.3-py3.6-linux-x86_64.egg', '/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg']; sys.real_prefix = sys.prefix;import encodings.aliases;import encodings.ascii;import encodings.base64_codec;import encodings.big5;import encodings.big5hkscs;import encodings.bz2_codec;import encodings.charmap;import encodings.cp037;import encodings.cp1006;import encodings.cp1026;import encodings.cp1125;import encodings.cp1140;import encodings.cp1250;import encodings.cp1251;import encodings.cp1252;import encodings.cp1253;import encodings.cp1254;import encodings.cp1255;import encodings.cp1256;import encodings.cp1257;import encodings.cp1258;import encodings.cp273;import encodings.cp424;import encodings.cp437;import encodings.cp500;import encodings.cp720;import encodings.cp737;import encodings.cp775;import encodings.cp850;import encodings.cp852;import encodings.cp855;import encodings.cp856;import encodings.cp857;import encodings.cp858;import encodings.cp860;import encodings.cp861;import encodings.cp862;import encodings.cp863;import encodings.cp864;import encodings.cp865;import encodings.cp866;import encodings.cp869;import encodings.cp874;import encodings.cp875;import encodings.cp932;import encodings.cp949;import encodings.cp950;import encodings.euc_jis_2004;import encodings.euc_jisx0213;import encodings.euc_jp;import encodings.euc_kr;import encodings.gb18030;import encodings.gb2312;import encodings.gbk;import encodings.hex_codec;import encodings.hp_roman8;import encodings.hz;import encodings.idna;import encodings.iso2022_jp;import encodings.iso2022_jp_1;import encodings.iso2022_jp_2;import encodings.iso2022_jp_2004;import encodings.iso2022_jp_3;import encodings.iso2022_jp_ext;import encodings.iso2022_kr;import encodings.iso8859_1;import encodings.iso8859_10;import encodings.iso8859_11;import encodings.iso8859_13;import encodings.iso8859_14;import encodings.iso8859_15;import encodings.iso8859_16;import encodings.iso8859_2;import encodings.iso8859_3;import encodings.iso8859_4;import encodings.iso8859_5;import encodings.iso8859_6;import encodings.iso8859_7;import encodings.iso8859_8;import encodings.iso8859_9;import encodings.johab;import encodings.koi8_r;import encodings.koi8_t;import encodings.koi8_u;import encodings.kz1048;import encodings.latin_1;import encodings.mac_arabic;import encodings.mac_centeuro;import encodings.mac_croatian;import encodings.mac_cyrillic;import encodings.mac_farsi;import encodings.mac_greek;import encodings.mac_iceland;import encodings.mac_latin2;import encodings.mac_roman;import encodings.mac_romanian;import encodings.mac_turkish;import encodings.palmos;import encodings.ptcp154;import encodings.punycode;import encodings.quopri_codec;import encodings.raw_unicode_escape;import encodings.rot_13;import encodings.shift_jis;import encodings.shift_jis_2004;import encodings.shift_jisx0213;import encodings.tis_620;import encodings.undefined;import encodings.unicode_escape;import encodings.unicode_internal;import encodings.utf_16;import encodings.utf_16_be;import encodings.utf_16_le;import encodings.utf_32;import encodings.utf_32_be;import encodings.utf_32_le;import encodings.utf_7;import encodings.utf_8;import encodings.utf_8_sig;import encodings.uu_codec;import encodings.zlib_codec;import locale;import inspect;"
b'  File "<frozen importlib._bootstrap>", line 971, in _find_and_load'
b'  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked'
b'  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked'
b'  File "<frozen importlib._bootstrap_external>", line 650, in exec_module'
b'  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed'
b'  File "/home/mcon/vocore/Buildroot/output/host/lib/python3.6/encodings/big5.py", line 7, in <module>'
b'    import _codecs_tw, codecs'
b'  File "<frozen importlib._bootstrap>", line 971, in _find_and_load'
b'  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked'
b"ModuleNotFoundError: No module named '_codecs_tw'"
b'# clear builtins._'
b'# clear sys.path'
b'# clear sys.argv'
b'# clear sys.ps1'
b'# clear sys.ps2'
b'# clear sys.last_type'
b'# clear sys.last_value'
b'# clear sys.last_traceback'
b'# destroy encodings.big5'
b'# clear sys.path_hooks'
b'# clear sys.path_importer_cache'
b'# clear sys.meta_path'
b'# clear sys.__interactivehook__'
b'# clear sys.flags'
b'# clear sys.float_info'
b'# restore sys.stdin'
b'# restore sys.stdout'
b'# restore sys.stderr'
b'# cleanup[2] removing builtins'
b'# cleanup[2] removing sys'
b'# cleanup[2] removing _frozen_importlib'
b'# cleanup[2] removing _imp'
b'# cleanup[2] removing _warnings'
b'# cleanup[2] removing _thread'
b'# cleanup[2] removing _weakref'
b'# cleanup[2] removing _frozen_importlib_external'
b'# cleanup[2] removing _io'
b'# cleanup[2] removing marshal'
b'# cleanup[2] removing posix'
b'# cleanup[2] removing zipimport'
b'# cleanup[2] removing encodings'
b'# cleanup[2] removing codecs'
b'# cleanup[2] removing _codecs'
b'# cleanup[2] removing encodings.aliases'
b'# cleanup[2] removing encodings.utf_8'
b'# cleanup[2] removing _signal'
b'# cleanup[2] removing __main__'
b'# destroy __main__'
b'# destroy encodings'
b'# cleanup[2] removing encodings.latin_1'
b'# cleanup[2] removing io'
b'# destroy io'
b'# cleanup[2] removing abc'
b'# destroy abc'
b'# cleanup[2] removing _weakrefset'
b'# destroy _weakrefset'
b'# cleanup[2] removing encodings.ascii'
b'# cleanup[2] removing encodings.base64_codec'
b'# cleanup[2] removing base64'
b'# cleanup[2] removing re'
b'# cleanup[2] removing enum'
b'# cleanup[2] removing types'
b'# destroy types'
b'# cleanup[2] removing functools'
b'# cleanup[2] removing _functools'
b'# cleanup[2] removing collections'
b'# cleanup[2] removing _collections_abc'
b'# cleanup[2] removing operator'
b'# destroy operator'
b'# cleanup[2] removing _operator'
b'# cleanup[2] removing keyword'
b'# destroy keyword'
b'# cleanup[2] removing heapq'
b'# cleanup[2] removing _heapq'
b'# cleanup[2] removing itertools'
b'# cleanup[2] removing reprlib'
b'# destroy reprlib'
b'# cleanup[2] removing _collections'
b'# cleanup[2] removing weakref'
b'# destroy weakref'
b'# cleanup[2] removing collections.abc'
b'# cleanup[2] removing sre_compile'
b'# cleanup[2] removing _sre'
b'# cleanup[2] removing sre_parse'
b'# cleanup[2] removing sre_constants'
b'# destroy sre_constants'
b'# cleanup[2] removing _locale'
b'# cleanup[2] removing copyreg'
b'# cleanup[2] removing struct'
b'# cleanup[2] removing _struct'
b'# cleanup[2] removing binascii'
b'# cleanup[2] removing encodings.cp437'
b'# destroy zipimport'
b'# destroy _signal'
b'# cleanup[3] wiping _frozen_importlib'
b'# destroy _frozen_importlib_external'
b'# cleanup[3] wiping _imp'
b'# cleanup[3] wiping _warnings'
b'# cleanup[3] wiping _thread'
b'# cleanup[3] wiping _weakref'
b'# cleanup[3] wiping _io'
b'# cleanup[3] wiping marshal'
b'# cleanup[3] wiping posix'
b'# cleanup[3] wiping codecs'
b'# cleanup[3] wiping _codecs'
b'# cleanup[3] wiping encodings.aliases'
b'# cleanup[3] wiping encodings.utf_8'
b'# cleanup[3] wiping encodings.latin_1'
b'# cleanup[3] wiping encodings.ascii'
b'# cleanup[3] wiping encodings.base64_codec'
b'# destroy base64'
b'# cleanup[3] wiping re'
b'# destroy enum'
b'# destroy sre_compile'
b'# destroy copyreg'
b'# cleanup[3] wiping functools'
b'# cleanup[3] wiping _functools'
b'# cleanup[3] wiping collections'
b'# destroy _collections_abc'
b'# destroy heapq'
b'# cleanup[3] wiping _operator'
b'# cleanup[3] wiping _heapq'
b'# cleanup[3] wiping itertools'
b'# cleanup[3] wiping _collections'
b'# destroy _collections'
b'# cleanup[3] wiping collections.abc'
b'# cleanup[3] wiping _sre'
b'# cleanup[3] wiping sre_parse'
b'# cleanup[3] wiping _locale'
b'# destroy _locale'
b'# cleanup[3] wiping struct'
b'# cleanup[3] wiping _struct'
b'# destroy _struct'
b'# cleanup[3] wiping binascii'
b'# cleanup[3] wiping encodings.cp437'
b'# cleanup[3] wiping sys'
b'# cleanup[3] wiping builtins'
b'# destroy re'
b'# destroy struct'
b'# destroy binascii'
b'# destroy _functools'
b'# destroy _operator'
b'# destroy _sre'
b'# destroy sre_parse'
b'# destroy functools'
b'# destroy collections.abc'
b'# destroy _heapq'
b'# destroy collections'
b'# destroy itertools'
b''
Error, please report the issue with above output.

Thanks in Advance!

[kayhayen]

The core of the error is this "ModuleNotFoundError: No module named '_codecs_tw'" which means it fails to load that module, which probably won't be possible if we are talking cross platform.

[kayhayen]

The reason Nuitka does this, is to find the module location from the verbose output of Python, where the importer will report where it came from. Although with Python3 I think, it uses __file__ after it has been loaded anyway.

To avoid executing, you would have to make that code not do __import__ but rather a proper finding of the module, which is something that Nuitka implements somewhat, but it is not really well exposed by library code.

[mcondarelli]

Ok. Fixed.
Problem actually was in my build environment Python3 is built without support for Asian codecs, but corresponding files were not removed (e.g.: .../encodings/big5.py).

Now I have the problem to convince Nuitka to use the cross-compiler.
Without any options it produces x86_64 code which won't run on target; I tried supplying a bunch of Environment variables:

( cd /home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client && PATH="/home/mcon/vocore/Buildroot/output/host/bin:/home/mcon/vocore/Buildroot/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games" AR="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-ar" AS="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-as" LD="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-ld" NM="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-nm" CC="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-gcc" GCC="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-gcc" CPP="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-cpp" CXX="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-g++" FC="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-gfortran" F77="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-gfortran" RANLIB="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-ranlib" READELF="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-readelf" STRIP="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-strip" OBJCOPY="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-objcopy" OBJDUMP="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-objdump" AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as" CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc" CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld" CPPFLAGS_FOR_BUILD="-I/home/mcon/vocore/Buildroot/output/host/include" CFLAGS_FOR_BUILD="-O2 -I/home/mcon/vocore/Buildroot/output/host/include" CXXFLAGS_FOR_BUILD="-O2 -I/home/mcon/vocore/Buildroot/output/host/include" LDFLAGS_FOR_BUILD="-L/home/mcon/vocore/Buildroot/output/host/lib -Wl,-rpath,/home/mcon/vocore/Buildroot/output/host/lib" FCFLAGS_FOR_BUILD="" DEFAULT_ASSEMBLER="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-as" DEFAULT_LINKER="/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-ld" CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os " CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os " LDFLAGS="" FCFLAGS=" -Os " FFLAGS=" -Os " PKG_CONFIG="/home/mcon/vocore/Buildroot/output/host/bin/pkg-config" STAGING_DIR="/home/mcon/vocore/Buildroot/output/host/mipsel-buildroot-linux-musl/sysroot" INTLTOOL_PERL=/usr/bin/perl /home/mcon/vocore/Buildroot/output/host/bin/python  -m nuitka --follow-imports --standalone --plugin-enable=pylint-warnings stream.py )
Nuitka:WARNING:stream.py:8: Cannot find 'serial' as absolute import.
Nuitka:WARNING:stream.py:506: Cannot find 'anagrafica' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/scanner/__init__.py:2: Cannot find 'evdev' in package 'scanner' as absolute import (tried evdev).
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/commands.py:7: Cannot find 'lxml' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/commands.py:91: Cannot find 'logging_tree' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/template.py:10: Cannot find 'jinja2' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/dazerolab.py:307: Cannot find 'requests' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/memento.py:4: Cannot find 'jsonpickle' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/client.py:17: Cannot find 'urllib3' as absolute import.
Nuitka:WARNING:/home/mcon/vocore/Buildroot/output/build/Applicazione-custom/Communications/Client/client.py:287: Cannot find 'signxml' as absolute import.
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.__main__.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.utils.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.dazerolab.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.commands.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from stream.build/__frozen.c:3:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.updater.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.agenzia_entrate.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.client.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.scanner.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
scons: *** [stream.build/module.utils.o] Error 1
scons: *** [stream.build/__frozen.o] Error 1
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/__helpers.c:4:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/__constants.c:2:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
In file included from /home/mcon/vocore/Buildroot/output/host/include/python3.6m/Python.h:50:0,
                 from /home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include/nuitka/prelude.h:43,
                 from stream.build/module.intermediario.c:19:
/home/mcon/vocore/Buildroot/output/host/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
scons: *** [stream.build/module.agenzia_entrate.o] Error 1
scons: *** [stream.build/__constants.o] Error 1
scons: *** [stream.build/__helpers.o] Error 1
scons: *** [stream.build/module.intermediario.o] Error 1
scons: *** [stream.build/module.scanner.o] Error 1
scons: *** [stream.build/module.client.o] Error 1
scons: *** [stream.build/module.updater.o] Error 1
scons: *** [stream.build/module.dazerolab.o] Error 1
scons: *** [stream.build/module.__main__.o] Error 1
scons: *** [stream.build/module.commands.o] Error 1
make[1]: *** [package/pkg-generic.mk:232: /home/mcon/vocore/Buildroot/output/build/Applicazione-custom/.stamp_built] Error 1
make: *** [Makefile:16: _all] Error 2

I can see two distinct problems:

1. Nuitka does not seem to pick up the right library path (that should be matter of setting the right PYTHONPATH).
2. Something doesn't match in compiler/includes (here I would need Your help).

TiA!

[mcondarelli]

I did some digging and found what is the problem, but I have no idea how to solve it.

In my cross-development system I have two instances of Python3:

host-python3: running on host, used to compile (.py -> .pyc) all packages before installation on Target. This is compiled with x86_64 gcc and runs on host; SIZEOF_LONG = 8 (64bit)

python3: running on Target, used to actually run Python code on Target, uses the .pyc files compiled. This is cross-compiled with mipsel-buildroot-linux-musl-gcc to run on Target; SIZEOF_LONG = 4 (32bit).

My attempt at cross-compilation (run Nuitka on host, using host-python3) is failing because I can produce the .c files this way, but compilation should use python3 (Target) for cross-compilation... and don't know how to explain this to Nuitka. PYTHONPATH for cross-compilation should be different from PYTHONPATH for code generation.

I also tried separating the two steps, but --generate-c-only does not seem to setup the necessary symlinks (at least).

Can you give advice, please?
TiA!

[mcondarelli]

I did some further step forward (in a very ugly way).
Current errors are:

vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include -I/home/mcon/vocore/Buildroot/output/target/usr/include/python3.6m -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os   module.intermediario.c module.template.c InspectPatcher.c CompiledCellType.c module.scanner.c module.updater.c module.memento.c __constants.c CompiledFrameType.c module.client.c CompiledFunctionType.c CompiledMethodType.c CompiledCoroutineType.c __internal__.c module.__main__.c module.dazerolab.c MetaPathBasedLoader.c module.LCtlWifi.c CompiledGeneratorType.c __frozen.c __helpers.c module.agenzia_entrate.c module.prot.c module.fattura24.c module.utils.c CompiledAsyncgenType.c module.serial_asyncio.c CompiledCodeHelpers.c MainProgram.c module.provider.c module.commands.c module.device.c -o ../stream.dist/stream
CompiledGeneratorType.c:34:56: fatal error: CompiledGeneratorTypeUncompiledIntegration.c: No such file or directory
 #include "CompiledGeneratorTypeUncompiledIntegration.c"
                                                        ^
compilation terminated.
CompiledCodeHelpers.c:30:28: fatal error: HelpersBuiltin.c: No such file or directory
 #include "HelpersBuiltin.c"
                            ^
compilation terminated.
MainProgram.c: In function ‘main’:
MainProgram.c:282:30: error: ‘_NUITKA_SYSFLAG_NO_SITE’ undeclared (first use in this function)
     Py_NoUserSiteDirectory = _NUITKA_SYSFLAG_NO_SITE;
                              ^~~~~~~~~~~~~~~~~~~~~~~
MainProgram.c:282:30: note: each undeclared identifier is reported only once for each function it appears in
MainProgram.c:284:22: error: ‘_NUITKA_SYSFLAG_VERBOSE’ undeclared (first use in this function)
     Py_VerboseFlag = _NUITKA_SYSFLAG_VERBOSE;
                      ^~~~~~~~~~~~~~~~~~~~~~~
MainProgram.c:285:27: error: ‘_NUITKA_SYSFLAG_BYTES_WARNING’ undeclared (first use in this function)
     Py_BytesWarningFlag = _NUITKA_SYSFLAG_BYTES_WARNING;
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MainProgram.c:351:5: warning: implicit declaration of function ‘_initBuiltinOriginalValues’ [-Wimplicit-function-declaration]
     _initBuiltinOriginalValues();
     ^~~~~~~~~~~~~~~~~~~~~~~~~~

The missing includes should be just a missing "-I..." option, but the_NUITKA_SYSFLAG_XXX seem to be some variables directly passed to scons.

Tomorrow I will resume fight
g'nite!

[mcondarelli]

Further update:
I managed to compile my code with the following (hand-crafted) command:

/home/mcon/vocore/Buildroot/output/host/bin/mipsel-buildroot-linux-musl-gcc -I. -I/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/include -I/home/mcon/vocore/Buildroot/output/host/lib/python3.6/site-packages/Nuitka-0.6.0.6-py3.6.egg/nuitka/build/static_src -I/home/mcon/vocore/Buildroot/output/target/usr/include/python3.6m -D_NUITKA_EXE=1 -D_NUITKA_SYSFLAG_NO_SITE=0 -D_NUITKA_SYSFLAG_VERBOSE=0 -D_NUITKA_SYSFLAG_BYTES_WARNING=0 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os   module.intermediario.c module.template.c InspectPatcher.c CompiledCellType.c module.scanner.c module.updater.c module.memento.c __constants.c CompiledFrameType.c module.client.c CompiledFunctionType.c CompiledMethodType.c CompiledCoroutineType.c __internal__.c module.__main__.c module.dazerolab.c MetaPathBasedLoader.c module.LCtlWifi.c CompiledGeneratorType.c __frozen.c __helpers.c module.agenzia_entrate.c module.prot.c module.fattura24.c module.utils.c CompiledAsyncgenType.c module.serial_asyncio.c CompiledCodeHelpers.c MainProgram.c module.provider.c module.commands.c module.device.c -L/home/mcon/vocore/Buildroot/output/staging/usr/lib -lpython3 -l python3.6m -o ../stream.dist/stream

now two problems (probably strictly connected) remain:

• how to convince scons to issue the right commands for cross-compilation.
• how to convince scons to cross compile dependencies (all .so in stream.dist are x86_64 binaries).

I was not able to find where the intermediate source files for the dependency modules are (otherwise I could hand-cross-compile-them, at least to confirm the beast can actually run on target!).

To this end I suppose I will have to modify Nuitka itself and Your guidance would be needed.

[mingwandroid]

I really like what I have seen of Nuitka so far even if I've not managed to get a successful build yet!

Can someone detail how compilers are found? Does it defer to sysconfig at all? Would that be a welcome addition? I see there are options for e.g. --clang, would e.g. --cc-from-sysconfig be acceptable? If so I can work on a patch.

Many thanks for working on this project!

[kayhayen]

@mingwandroid that would be great, looking forward to a PR and a welcome addition.

[mingwandroid]

Hello again @kayhayen, I've made some progress on this, I'd be happy to share a WIP PR with you to make sure I'm on the right track though.

It works like a dream on Linux, but on macOS it fails because (somehow) clang is being called with CXXFLAGS and that fails. The same thing happens on Linux, but instead of failing, GCC just complains. Do you have any pointers as to why this might be happening? I know that you use a C++ compiler when c11_mode is False but in these instances, c11_mode is True.

[DrTerr]

@mingwandroid I am interested in what have you made.
Cross compilation support would be great for Nuitka.
I have a project which would greatly benefit from it.
I have a build-root environment ready, and being able to use to compile my project for embeded system would be great.

[mingwandroid]

Hi @DrTerr, well, I did do some hacking on this but haven't finished it yet unfortunately. I will try to track down the latest of my work on this (I could have lost it actually!) and will get back to you (if I forget please ping me).

[DrTerr]

Hi @mingwandroid, pinging you as I am still interested.
I didn't have time yet to investigate what needs to be updated to enable cross-compilation in Nuitka, but if have a git copy with changes, it also would greatly help with seeing what further work needs to be done.

[jayvdb]

@mingwandroid, I am also quite interested in what you have so far. It doesnt matter if it is incomplete; I could help get it finished.

[mingwandroid]

I was unable to locate my work. I'm really sorry.

[jayvdb]

I would appreciate if somebody tried installing wine, CPython for Windows into it, MinGW64 (32 bits or is wine 64 bits good already, maybe try both) for Windows into it, then run a compilation with it, and report.

I have MinGW cross compilation happening on openSUSE Open Build System, with the same collection of packages being worked on at https://github.com/Nuitka/Nuitka/issues/522 .

The cross-compile project is at https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37 (ignore wine-python3, it was an early attempt).

Ordinary Python packages built on MinGW at https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages (bootstrap + Nuitka) and https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages-p2 (the rest of the core packages) , but not much effort put into this yet.

Nuitka compiled packages at https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages-Nuitka

It required a few patches, which I will start polishing & pushing

[mingwandroid]

Any reason not to use msys2/mingw-w64-python or at least our patches? It
would gain you compatibility with that.

On Tue, 1 Oct 2019, 04:34 John Vandenberg, [e-mail] wrote:

I would appreciate if somebody tried installing wine, CPython for Windows
into it, MinGW64 (32 bits or is wine 64 bits good already, maybe try both)
for Windows into it, then run a compilation with it, and report.

I have MinGW cross compilation happening on openSUSE Open Build System,
with the same collection of packages being worked on at #522
https://github.com/Nuitka/Nuitka/issues/522 .

The cross-compile project is at
https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37
(ignore wine-python3, it was an early attempt).

Ordinary Python packages built on MinGW at
https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages
(bootstrap + Nuitka) and
https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages-p2
(the rest of the core packages) , but not much effort put into this yet.

Nuitka compiled packages at
https://build.opensuse.org/project/show/home:jayvdb:wine-python3:py37:packages-Nuitka

It required a few patches, which I will start polishing & pushing

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Nuitka/Nuitka/issues/43?email_source=notifications&email_token=AAH6S5FLUY7G45JB6EMP7M3QMKZLLA5CNFSM4EI5UTL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD77W3EQ#issuecomment-536833426,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6S5HYR72ZAKHCD6YL5A3QMKZLLANCNFSM4EI5UTLQ
.

[jayvdb]

@mingwandroid , I did use those patches, refreshed a bit; see .spec in https://build.opensuse.org/package/show/home:jayvdb:wine-python3:py37/mingw32-python3 , which is very similar to the .spec used by RedHat.

[mingwandroid]

Very cool @jaydb. Pinging @tmelman as this is something we started to explore years ago. Do your changes break compilation on msys2? If not we could review a PR there.

[jayvdb]

I kept the entire series of patches, even the ones irrelevant to the openSUSE cross-compile.
I left notes above each patch entry in the .spec . Mostly it was fuzz updates because openSUSE is still on py3.7.3 while msys is py3.7.4, and rpm doesnt allow any fuzz, but I did drop one hunk of 2050-undo-venv-redirector.patch

https://build.opensuse.org/package/view_file/home:jayvdb:wine-python3:py37/mingw32-python3/mingw32-python3.spec?expand=1

I tried to get gdbm working, but it failed, as I think your team has also encountered.