Distributed: Python 3.9 compatibility

[jcrist]

Opening this issue to track Python 3.9 compatibility issues.

[jcrist]

Currently distributed fails to start worker processes on Python 3.9 with a local cluster (at least on osx). I see

In [3]: client = Client()
Task exception was never retrieved
future: <Task finished name='Task-396' coro=<_wrap_awaitable() done, defined at /opt/miniconda3/envs/prefect39/lib/python3.9/asyncio/tasks.py:678> exception=ImportError("cannot import name 'Popen' from partially initialized module 'multiprocessing.popen_spawn_posix' (most likely due to a circular import) (/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/popen_spawn_posix.py)")>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/asyncio/tasks.py", line 685, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/core.py", line 305, in _
    await self.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 295, in start
    response = await self.instantiate()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 378, in instantiate
    result = await self.process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 575, in start
    await self.process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/context.py", line 283, in _Popen
    from .popen_spawn_posix import Popen
ImportError: cannot import name 'Popen' from partially initialized module 'multiprocessing.popen_spawn_posix' (most likely due to a circular import) (/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/popen_spawn_posix.py)

[madkinsz]

I get the same error in the python3.9 docker image (Debain 10 - Linux version 4.19.76) so I presume this is not limited to OSX

[mrocklin]

cc @itamarst this seems like something that might be up your alley

[itamarst]

Always impressed at your nerd sniping skills :grin:

[mrocklin]

I think that my main value to society today is indexing people by technical interest

[mrocklin]

Also, thank you for taking a look, I appreciate it

(I'm interpretting your :grin: as enthusiasm)

[itamarst]

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

No idea what's going on yet, but I suspect it's a bug in Python.

[mrocklin]

Thank you for investigating this @itamarst

[mrocklin]

Short term, would a solution be to add that import to the appropriate file within distributed?

[itamarst]

I would guess so, yes (on POSIX platforms, anyway).

[lingfeiwang]

And a similar command line workaround:

dask-worker --preload-nanny multiprocessing.popen_spawn_posix

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

[mrocklin]

Would you be interested in opening a PR Itamar?

Nice solution Lingfei Wang!

On Wed, Nov 4, 2020 at 4:30 PM Lingfei Wang [e-mail]
wrote:

And a similar command line workaround:

dask-worker --preload-nanny multiprocessing.popen_spawn_posix

A workaround:

import multiprocessing.popen_spawn_posix>>> from distributed import Client>>> Client()

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/distributed/issues/4168#issuecomment-722049470,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTBDSD6NGWPZZ7G44SDSOHW2TANCNFSM4SRDXDTQ
.

[stonebig]

hi all,

The workaround proposed doesn't seem to work on Windows / Python-3.9.1 / Distributed-2020.12.0 ... I notice it may only be ok for "NOT WINDOWS"

Any hope on this for non-windows, then Windows ?
Any issue/fix identified/asked to cpython for next Python-3.9.2 I can follow ?

without the POSIXpatch

with the POSIX patch (so not ok either)

[stonebig]

hum, looking in bugs.cpython.org, maybe it's this cpython issue https://bugs.python.org/issue41567 , due to https://bugs.python.org/issue35943

if it's https://bugs.python.org/issue38884, Ansible defined a workaround: https://github.com/ansible/awx/pull/6093/files

[SunMaungOo]

For Windows workaround

>>> from dask.distributed import Client
>>> import multiprocessing.popen_spawn_win32
>>> Client()
<Client: 'tcp://127.0.0.1:7182' processes=4 threads=8, memory=8.54 GB>

[stonebig]

Thanks a lot @SunMaungOo , it works nicely.

[yannikschaelte]

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

No idea what's going on yet, but I suspect it's a bug in Python.

Thanks @itamarst for the patch! Also solved my issue of Client() just getting stuck without any output on python 3.9.

[mr-september]

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

No idea what's going on yet, but I suspect it's a bug in Python.

PyCharm gives me an error: No module named 'popen_spawn_posix'

Do I need to pip install something? Sorry very new to python.

[crusaderky]

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

No idea what's going on yet, but I suspect it's a bug in Python.

PyCharm gives me an error: No module named 'popen_spawn_posix'

Do I need to pip install something? Sorry very new to python.

What's your version of Python?

[crusaderky]

@mr-september also, in windows you have to import multiprocessing.popen_spawn_win32 instead

[mr-september]

What's your version of Python?

3.9, and I'm on Ubuntu.

but sorry I don't think I had an issue afterall, I just ignored PyCharm's warning and ran the script, things seems to have worked well.

[pitrou]

I think this is definitely a CPython regression. Taking this upstream.

[pitrou]

https://bugs.python.org/issue43517

[mrocklin]

Thanks @pitrou

On Tue, Mar 16, 2021 at 11:20 AM Antoine Pitrou @.*>
wrote:

https://bugs.python.org/issue43517

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/distributed/issues/4168#issuecomment-800405034,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTERE46B67ZQW5QZHF3TD6ALTANCNFSM4SRDXDTQ
.