Joblib: Joblib with Loky multiprocessing backend unable to use in frozen executable

[Sieboldianus]

This issue has bugged me for a week now, I tried to isolate it as good as possible and finally created this GitHub repo as a minimal reproducible example.

---

Short read: when joblib is used in a dependency (eg. in my example, hdbscan), multiprocessing doesn't work in frozen Python executables (cx_freeze) on Windows - neither with threaded execution, nor with Queue or with multiprocessing.Pool. Standard pool execution results in OSError: [WinError 87] The parameter is incorrect, Queue and ThreadPool lead to a multiprocessing bomb, using the existing mp.Pool with multiprocessing.get_context("spawn").Pool() doesn't crash but is unusable slow.

See complete logs for the approaches above.

---

Considerations:

• all of this only happens in the frozen package, not when used regularly with Python interpreter (tested in WSL Ubuntu 18.04 and Windows 10).
• this didn't happen with the old multiprocessing backend from joblib (back then, HDBSCAN was using joblib from sklearn.externals.joblib), only with the new Loky backend. I've moved from python 3.6 to 3.7 and can't go back to old joblib/sklearn versions
• in my original app, I also get unrecognized arguments: --multiprocessing-fork 3832 (likely because I'm processing args, which I am not doing in the expackage repo)

I've stumbled across a lot of similar reports, and I link those here for completeness:

• Joblib Parallel + Cython hanging forever: seems different to the issue described here
• Multiprocessing backed parallel loops cannot be nested below threads: nesting below threads has worked for me, as far as I can tell, but it doesn't work with the Loky backend?
• uWSGI and joblib Semaphore: Joblib will operate in serial mode - I cannot relate this to this issue, but Loky issue 207 also results in loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. and there's also joblib issue 827 with a similar traceback
• according the joblib docs, one can switch back to the old multiprocessing backend of loky, but I don't know how to do this inside my dependency hdbscan
• a similar spawning problem is also described in cython issue 2014, but the solutions don't work for me
• best reference so far was Compiling Executable with dask or joblib multiprocessing with cython results in errors, provided solutions didn't work for me
• there's also this bug report for frozen apps in Python, for which the fix apparently didn't make it into 3.7 release

---

The expackage repository I set up has 4 branches, with different approaches to execute a function async. Check out the cluster.py in each of these branches:

• multiprocessing.Pool
• multiprocessing.ThreadPool
• multiprocessing Queue and Thread (perhaps from a code formatting point of view the best approach)
• multiprocessing existing context pool

All of these branches work when executed directly in python, but not when frozen. It's overkill, but I've provided links to the frozen executables (300MB each zip).

---

I would be very grateful for any hints to solve the problem. I understand the argumentation "don't use Windows" or "don't freeze Python environments" - in this case I have no choice, the executable I am working on is for end users on Windows. The reason I need to run the fit_cluster in async mode (on a different thread or process) is that the GUI otherwise freezes and crashes. I've excluded all the GUI stuff and made sure that this is not the cause of the issue (it isn't, as is visible in the example repo). This also shouldn't matter: generally, I would expect that multiprocessing in frozen apps is still supported in Python 3.7, as it includes an extra hook for it (multiprocessing.freeze_support()) - then this would be an issue of joblib/loky - which is why I am reporting it here.

If anyone has an idea, but can't test this due to missing Windows, I'm happy to test Pull Requests on the linked Repo. Thanks so much!

[Sieboldianus]

.. I found a workaround in the rabbit hole.

1. Clone hdbscan
2. Replace Parallel(n_jobs=self.n_jobs) with Parallel(n_jobs=self.n_jobs, prefer="threads") in lines 411 and 1015, as suggested by Manish Dash (here's my fork)
3. Errors from joblib/loky on frozen expackage from branch queuethread and threadpool gone!

I am wondering if there is any hook that I could add to my parent package to force joblib in my dependency toprefer="threads"? Otherwise, I would need to maintain hdbscan in parallel.

[mx2048]

It appears to me you want to monkey-patch a class in a third-party module. If so, below is the hook:

import joblib

def change_jolib_parallel_prefer(prefer='threads'):
    _original_init = joblib.parallel.Parallel.__init__

    def _monkey_patched_init(self, *args, **kwargs):
        kwargs['prefer'] = prefer
        _original_init(self, *args, **kwargs)

    joblib.parallel.Parallel.__init__ = _monkey_patched_init

Call the function before any third-party modules call :class: joblib.parallel.Parallel.

[Sieboldianus]

Nice, thanks a lot! Don't know why I didn't think of this solution. I think this issue can be closed then.

[mx2048]

I am glad to help you, however, 'loky' backend still doesn't work in frozen executables on Windows so far. Only 'threading' and 'multiprocessing' backends work properly to the best of my knowledge.

[Sieboldianus]

Yes, I wasn't sure whether this is can be fixed at all in loky, or if it is just a situation that needs a workaround, like forcing threading instead of loky backend when working with frozen executables - anyway, happy to leave this open.

[ogrisel]

@Sieboldianus @mx2048 Could you please give us a minimal reproducing example? In particular how is the frozen executable generated?

[ogrisel]

Note: to change the active backend you can do:

from joblib import parallel_backend

parallel_backend("threading")
# my code here

alternatively with a context manager:

To change the active backend you can do:

from joblib import parallel_backend

with parallel_backend("threading"):
    # my code here

[Sieboldianus]

@Sieboldianus @mx2048 Could you please give us a minimal reproducing example? In particular how is the frozen executable generated?

Thanks @ogrisel for the hint!

Have a look at the expackage I created at the time as a minimal reproducible example (albeit it perhaps isn't really considered as one): There's a cx_setup.py that I used with cx_Freeze installed with conda(-forge) on Windows to produce the executable (e.g. python cx_setup.py build).

I hope that helps!

[mx2048]

@ogrisel, below is a minimal reproducing example.

1. Create and activate a new virtual environment.
2. Upgrade pip:
   python -m pip install --upgrade pip
3. Install required packages:
   pip install joblib -U
pip install cx_freeze==6.1
4. Create a file sample.py

from joblib import Parallel, delayed

# Expected output: 0123456789
Parallel(n_jobs=-1, backend='loky')(delayed(print)(i, end='') for i in range(10))

1. Create a file setup.py

from cx_Freeze import setup, Executable


executables = [Executable('sample.py')]

build_exe_options = {
    "build_exe": './build/frozen_exe',
    "excludes": ['tkinter'],
    'include_msvcr': True}

setup(name='joblib_issue_1002',
      version='0.1',
      options={"build_exe": build_exe_options},
      executables=executables
      )

1. Change directory to the root of the project
2. Run python setup.py build

Tests
Test with backend='threading': OK, prints 0123456789 to console.
Test with backend='loky': BAD, outputs nothing, when pressing keyboard interrupt Ctrl+C, shows walls of errors, cannot be stopped, haave to terminate the whole process. Careful: 'loky' creates hundreds of processes, uses CPU and RAM up to 100%.

Additional info:

Virtual environment:
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 22:45:29) [MSC v.1916 32 bit (Intel)] on win32

pip list

Package    Version
---------- -------
cx-Freeze  6.1
joblib     0.16.0
pip        20.1.1
setuptools 40.8.0

You can download a virtual machine for Windows 10 development environment.

[mx2048]

@ogrisel, were you able to reproduce the issue?

[haugs2]

I have the exact same issue and was able to reproduce it with a minimal example similar to the one listed above.
Thank you for the suggested workaround @Sieboldianus and @ogrisel!
The workaround with using the 'threads' backend is however not a satisfying solution for me as there is an immense slow down in processing time when processing larger amounts of data (4GB of data are loaded by each process).
In addition the processes manipulate shared python objects, which probably due to the GIL is also a big slowdown when using the threads backend.

Is there any other workaround or planned solution to this problem?

[patrick-reinhard]

Note: to change the active backend you can do:

from joblib import parallel_backend

parallel_backend("threading")
# my code here

alternatively with a context manager:

To change the active backend you can do:

from joblib import parallel_backend

with parallel_backend("threading"):
    # my code here

Was also facing issues with scikit-learn using loky backend when running from a frozen pyinstaller executable.

This worked for me. Thanks a lot!

[yangfan0356]

Thanks for the above questions and answers, they are very helpful! I would like to add addtional result I got when using scikit-learn with pyinstaller, when processing parallel tasks. Actually it is possible to use the "old" backend "multiprocessing" of joblib together with the recent scikit-learn and python version e.g, 3.8. "Multiprocessing" is faster than "thread" doing cpu intensive work. The following two steps shall work.

1. Use "multiprocessing" parameter
   https://scikit-learn.org/stable/modules/generated/sklearn.utils.parallel_backend.html

from joblib import parallel_backend
with parallel_backend("multiprocessing"):
      #my code here

or

from joblib import parallel_backend
parallel_backend("multiprocessing")
#my code here

1. Modify your entry script for pyinstaller and add freeze_support like this:

from multiprocessing import freeze_support

if __name__ == '__main__':

    # Pyinstaller fix
    freeze_support()
    main()

The above two steps are sufficient to run frozen executable with joblib, tested on windows 10.

1. An example that I use GridSearchCV method with multiprocessing.

        grid = GridSearchCV(... n_jobs = -1, ...) 
        with parallel_backend("multiprocessing"):
            grid.fit(X, y) 

Fan