Nuitka: How do you use multiprocessing with this?

[joveice]

OS: Windows 10 64 bit.
Python 3.6.6 v3.6.6:4cf1f54eb7 64 bit
Nuitka: First tried stable then tried the development Nuitka 0.5.33rc5 Python3.6 64 bit MSI
I installed it with that.
Short example:

import multiprocessing
import time

class Test(object):

    def __init__(self):
        self.process = multiprocessing.Process(target=self.run, args=())
        self.process.daemon = True

    def start(self):
        self.process.start()

    def stop(self):
        self.process.terminate()

    def run(self):
        while True:
            print('Never happening')
            time.sleep(1.5)

Test().start()

The error changes based on what the process is doing.
Example:

Traceback (most recent call last):
  File "test.py", line 28, in <module __parents_main__>
  File "test.py", line 15, in start
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
TypeError: Can't pickle instancemethod objects

My program (closed source):

Traceback (most recent call last):
  File "app.py", line 233, in <module __parents_main__>
  File "multiprocessing\spawn.py", line 74, in freeze_support
  File "multiprocessing\spawn.py", line 99, in spawn_main
  File "multiprocessing\reduction.py", line 82, in steal_handle
OSError: [WinError 87] The parameter is incorrect

Whats the correct way to build a standalone with multiprocessing?

I used this (and many other combinations before that)

nuitka test.py --standalone --recurse-all --python-flag=no_site --plugin-enable=multiprocessing --recurse-plugin=multiprocessing --show-modules

Awesome work!

[kayhayen]

I will try your enable, once I get a chance. You didn't call "freeze_support()" which might be mandatory for it to work.

If yes, add this to your program, and it should work.

if __name__ == '__main__':
    freeze_support()

My guess is many people who used this successfully and provided examples have had things that already worked with e.g. py2exe, and therefore have this.

Does this help?

Yours,
Kay

[joveice]

I will try to use that and I'll come back with what I find :)

[joveice]

It didn't solve the issue :(

[kayhayen]

I moved the "Test.start" info the __main__ test branch, and then it works for Python 2.7, but it does not for 3.6:

Common is this:

Nuitka:INFO:Injecting plug-in based pre load code for module 'multiprocessing.forking':
Nuitka:INFO: Monkey patching "multiprocessing" load environment.

Only 2.7 does this:

Nuitka:INFO:Injecting plug-in based post load code for module 'multiprocessing.forking':
Nuitka:INFO: Monkey patching "multiprocessing" for compiled methods.

Anyway, then 3.6 not doing the above, getting this is not surprising, is it:

File "C:\Python36_32\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: Can't pickle instancemethod objects

Seems also that the existing multiprocessing test of Nuitka somehow doesn't cover those, although it doesn't look all that different to me.

Anyway, I will look into why there is a difference, and if it's easy to resolve.

Yours,
Kay

[kayhayen]

Ok, somehow "multiprocessing.reduction" is not included, and then not monkey patched. The plugin was made portable across a multiprocessing change for "forking" and "reduction", and tried to monkey patch either one, post loading that one.

I added "reduction" to the list of modules to be included and made the pre/post load code of "multiprocessing" itself do it all. That way it ought to be really robust, and the distinction is now a try/except ImportError handling, that finds the pickler that multiprocessing uses in one of the two possible places.

With this in place, your program works. I am going to include this in a hotfix later today.

[joveice]

Really awesome man!

[kayhayen]

Fix is on factory branch now.

[kayhayen]

Released as part of 0.5.32.7

[joveice]

Yep, seems to have fixed it. Now I found another issue with connection and url getting, I will test more so I know what is wrong so I can make a issue on it :)

[kayhayen]

I pushed a hotfix related to multiprocessing yesterday to the factory branch. Seems plain functions calls were not working with Python3 there. Maybe that is your issue there too.

[joveice]

Do I have to build factory or is there anywhere I can download a pre build one?

[kayhayen]

Check this out: http://nuitka.net/doc/factory.html

[joveice]

Seems to have fixed it. Is there a way to fill the details of the .exe file? Such as version, copyright, product etc?

[kayhayen]

Can you make the details stuff be a separate issue?

[joveice]

Sure, will do.