Nuitka: Compiled app does not work if the path contains unicode characters

[BoboTiG]

• Nuitka version

0.6.3rc4
Python: 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 12:45:02) [MSC v.1900 32 bit (Intel)]
Executable: C:\Users\Windows10\Desktop\deploy-dir\3.6.7\Scripts\python.exe
OS: Windows
Arch: x86

• How did you install Nuitka and Python

Via pip on the factory branch.

• SSCCE

from win32com.shell import shell, shellcon

# Simply print out the current username home folder (equivalent to %userprofile%)
path = shell.SHGetFolderPath(0, shellcon.CSIDL_PROFILE, None, 0)
print("path =", path)

Compiled with:

python -m nuitka --standalone --trace-execution --debug test.py

This works fine if the username is latin, but not with such names abc こん ツリー.

I attached ok.log and bad.log to show where is fails; but this is the final error:

Traceback (most recent call last):
  File "C:\Users\abc \u3053\u3093 \u30c4\u30ea\u30fc\Desktop\test.dist\test.py", line 1, in <module>
    from win32com.shell import shell, shellcon
  File "C:\Users\abc \u3053\u3093 \u30c4\u30ea\u30fc\Desktop\test.dist\win32com\__init__.py", line 5, in <module win32com>
ImportError

[kayhayen]

I managed to reproduce this easily by mkdirん and trying to run standalone program inside there. For me it was with MSVC. This can sometimes depend on the C runtime. I think one of the reasons which non-MinGW64, i.e. normal MinGW is not supported, was that it doesn't handle it.

[kayhayen]

It basically fails to load DLLs from there. With -python-flag=-v one gets some debug information, part of which says this:

import _ctypes # LoadLibraryEx("C:\Users\kayha\repos\Py2C\?\MiniPyCryptodomeX.dist\_ctypes.pyd");

Notice the question mark, that is most probably the issue here. Seems determining the binary directory fails.

Yours,
Kay

[kayhayen]

Well, I am exploring to shift all API usage to unicode and wchar_t, I think I tried ANSI so far, and while it once worked, apparently it does cause too many complexities. I fear however, that for Python2, due to older run time environment, there was an issue driving me to ANSI in the first place, but I will find out I guess.

[BoboTiG]

Thanks for those answers. Indeed moving to wchar_t would be a good thing :)
I know you have lot of work, but do you have a timeline or an idea for a date/release of such possible migration?

[kayhayen]

I try to get away from timelines as much as I can for a project dependent on spare time of mine. But the good news is, a prototype is working, with relative ease. But now I am fighting Scons and Python2, which apparently hates to run inside such folders, but maybe the newer scons would do. I will keep you posted about progress, if only so I can be interrupted and resume later and not forget what I did.

[kayhayen]

Yeah, Scons running Python3 doesn't have a problem, --python-for-scons is our friend there. Just using the newer inline copy of Scons was not enough. But e.g. pip will also refuse it too, so it's not surprising I guess.

[kayhayen]

So, Python2 is working too, but many things Python fail to work, e.g. cffi won't load its DLLs, so that is not working. The standalone DLLs done by Nuitka work though. I think I also tried MinGW64 sucessfully. Now on to Linux.

[kayhayen]

Turns out, Linux was the real reason not going with wchar_t, as e.g. dlopen and may other things will not work with it. I now made it so that Linux is using char * and Windows is wchar_t *, and kind of special case there.

On Windows, even for Python2, it's slightly more healthy to use the wchar_t, but it's not really going to work to run inside strange folder names. But I guess, we are just not going to care about that too much, although there may be a way using "short folder names". I have had to use it for the commit hook, where spaces are not allowed for the #! in git. And it may be the solution to do this, but it won't be too urgent yet, and definitely not blocking right now.

Right now I am back to Windows, getting it working again after the Linux changes.

[kayhayen]

So I updated git factory branch with a version that may even produce binaries that work nicely for Python2 on Windows too. Compiling inside non-unicode paths will be a problem, but I expect there is a chance, things could work for Python2 too.

One or two things are not pretty. Too many of the buffers used are large fixed size and not actual size, and that makes things not robust. I only started to clean that up.

And then, for Python2 to be able to compile in Scons, I think we only need to switch to the short named directory. Luckily I recently had to level for the git commit hooks, where #! can only use short paths to the sh.exe, so that code already exists. I only moved it to where it can be used.

For Scons under Python2, the directories passed, and potentially the current directory, need to become the short names. Then it ought to be as if no unicode path was used. I guess that will solve a major issue for good.

[kayhayen]

Seems the factory code worked on everything but MinGW64, where host encoded strings were used, and the new code has different name for that on Windows and Linux, which is something I now removed, should be better now.

@Nuitka/nuitka-pushers I plan to make a release 0.6.3 mostly right after I merged this into develop. It's very crucial issue for compatibility with Windows, esp. to Asia. Lets try and also have the most important fixes in it, but nothing risky anymore. There is always a next release. Interestingly no hotfixes were necessary last time, and I like it that way. :)

[BoboTiG]

Successfully tested on Windows 10.
Thank you very much, this was a big one and you resolved it sooooo quickly! :champagne: :muscle:

[kayhayen]

Released as 0.6.3