Pipx: Unable to run 'pipx list' on Termux (Android)

[z0rc]

Running pipx list on Termux results in following stack trace:

venvs are in /data/data/com.termux/files/home/.local/pip
x/venvs
apps are exposed on your $PATH at /data/data/com.termux/
files/home/.local/bin
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multipr
ocessing' (/data/data/com.termux/files/usr/lib/python3.7
/lib-dynload/_multiprocessing.cpython-37m.so)

During handling of the above exception, another exceptio
n occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/home/.local/bin/pipx
", line 10, in <module>
    sys.exit(cli())
  File "/data/data/com.termux/files/home/.local/pipx/ven
vs/pipx/lib/python3.7/site-packages/pipx/main.py", line 
547, in cli
    exit(run_pipx_command(parsed_pipx_args))
  File "/data/data/com.termux/files/home/.local/pipx/ven
vs/pipx/lib/python3.7/site-packages/pipx/main.py", line 
198, in run_pipx_command
    commands.list_packages(venv_container)
  File "/data/data/com.termux/files/home/.local/pipx/ven
vs/pipx/lib/python3.7/site-packages/pipx/commands.py", l
ine 638, in list_packages
    with multiprocessing.Pool() as p:
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/pool.py", line 158, in __init__
    self._setup_queues()
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/pool.py", line 251, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/queues.py", line 332, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.7/mu
ltiprocessing/synchronize.py", line 32, in <module>
    " synchronization primitives needed will not" +
ImportError: This platform lacks a functioning sem_open 
implementation, therefore, the required synchronization 
primitives needed will not function, see issue 3770.

This is on OnePlus 5T with Android 9.0.

Technically it's a Python and/or Android issue, where multiprocessing module isn't working because Android kernel is lacking in semaphores syscalls. Also see https://bugs.python.org/issue3770

AFAIK there is still a way to workaround this, but it would require switching multiprocessing to threading module. For example, this is how black dealt with similar issue: https://github.com/psf/black/pull/533

[cs01]

Thanks for the suggestion on the workaround. I am guessing that is the way to go. Pull requests welcome if you want to give it a shot.

[pschmitt]

Sorry for necrobumping but I still encounter the error on Termux with pipx 0.15.1.3.
Did I miss something obvious?

[cs01]

This is the PR that closed it is https://github.com/pipxproject/pipx/pull/277. The diff is small, maybe you could take a look and see if it looks like it should work on your platform. Are you still getting the same error?

[pschmitt]

On a fully updated version of Termux you get the following:

Click to expand! (Spoiler: it's the same error as reported by OA)

pipx --version; pipx list                                                                                                                                                   
0.15.1.3
venvs are in /data/data/com.termux/files/home/.local/pipx/venvs
apps are exposed on your $PATH at /data/data/com.termux/files/home/.local/bin
<class 'function'>
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/home/.local/bin/pipx", line 8, in <module>
    sys.exit(cli())
  File "/data/data/com.termux/files/home/.local/lib/python3.8/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/data/data/com.termux/files/home/.local/lib/python3.8/site-packages/pipx/main.py", line 195, in run_pipx_command
    return commands.list_packages(venv_container)
  File "/data/data/com.termux/files/home/.local/lib/python3.8/site-packages/pipx/commands/list_packages.py", line 35, in list_packages
    with Pool() as p:
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/dummy/__init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

I dug a little further and it turns out that multiprocessing.dummy.Pool is not usable on Termux;

$ python3 -c "from multiprocessing.dummy import Pool; Pool()"                                                                                                                  
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/dummy/__init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

[cs01]

Thanks. Let me know if this branch fixes it
https://github.com/pipxproject/pipx/tree/cs01/multiprocessing

[pschmitt]

That won't work since both multiprocessing.Pool and multiprocessing.dummy.Pool are unavailable.

[pschmitt]

I commented too early. Hang on.

[pschmitt]

Jup. That works @cs01 ! Thanks.

[pschmitt]

$ pip install -U pipx $ pipx list venvs are in /data/data/com.termux/files/home/.local/pipx/venvs apps are exposed on your $PATH at /data/data/com.termux/files/home/.local/bin package flake8 3.8.1, Python 3.8.3 - flake8

Thanks, @cs01 !

[cs01]

Glad it’s working. Thanks for following back up so we could get this fixed.