Cython: Spawning-Problem when Cython-Compiling (to executable) DASK or JOBLIB multiprocessing on Windows

[Barry1]

I tried to optimize some of my jobs with cython and compiled them as executables for use in Windows (sorry, I can not use another OS).
Every time when using DASK or JOBLIB, I could not run the job, but my executable behaves as a Multiprocessing bomb.
At first I found out, that after compilation, the workers could no longer be identified (by __name__). Further the Workers did not get startet.
Further details to the problem can be found (and some solution ideas are given also over there) at my thread StackOverflow .

[jdodds]

If I understand the behavior you're seeing correctly, it may also be alleviated by a block similar to this:

if __name__ == '__main__':
    setattr(sys, 'frozen', True)
    multiprocessing.freeze_support()
    main()

As multiprocessing does some funky things to support "frozen" (i.e. made by py2exe or a few other tools) executables on windows systems, and it predicates this by checking for sys.frozen which is just kinda a convention some tools follow from what I understand.

[Barry1]

Hi @jdodds ,
I just tried with your block.
While running from python it won't work - it results in many unsuccessful tries to start the subprocesses - with the command output

Unknown option: --
usage: C:\Users\EbelingB\AppData\Local\Continuum\Anaconda3\python.exe [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.

This messages pops up unlimited until control-C.

But now the good news: compiling the code (with your block) with cython and gcc runs fine. That is fine - do you have an idea how to find a solution running from python and compiled?
And would it be possible to include something in the cython code to solve that issue?

Another think I'm wondering about is:
Wouldn't there be a way to distinguish the (cython-compiled) processes as within python? Running from intertreter python.exe the __name__ reads

__main__
__mp_main__
__mp_main__
__mp_main__
__mp_main__
__mp_main__

while after compiling it reads

__main__
__main__
__main__
__main__
__main__
__main__

Thx

[Barry1]

I further tried around with your input @jdodds resulting in the following solution

if __name__ == '__main__':
    if sys.argv[0][-4:] == '.exe':
        setattr(sys, 'frozen', True)
    multiprocessing.freeze_support()
    YOURMAINROUTINE()

which I also posted on StackOverflow.
That one seems to be most elegant for me and it is working for python call and beeing compiled as an windows-executable.

[scoder]

Closing as solved. Also, this is more of a support question, for which the bug tracker is the wrong place.