Termux-packages: Python: This platform lacks a functioning sem_open implementation

Created on 30 Nov 2016 · 15Comments · Source: termux/termux-packages

$ python
Python 3.5.2 (default, Sep 26 2016, 20:12:03)
[GCC 4.9.x 20150123 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> Queue()
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.5/multiprocessing/synchronize.py", line 29, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.5/multiprocessing/context.py", line 101, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.5/multiprocessing/queues.py", line 39, in __init__
    from .synchronize import SEM_VALUE_MAX as maxsize
  File "/data/data/com.termux/files/usr/lib/python3.5/multiprocessing/synchronize.py", line 34, in <module>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
>>>

Source

tonyseek

Most helpful comment

@BKubi You can google for python multiprocessing android to find some information, and perhaps open an issue against the Python project to support multiprocessing for Android if there isn't one already.

The only workaround I know of is to use threading instead of multiprocessing.

fornwall on 15 Feb 2017

👍3

All 15 comments

Android does not support sem_open() or related functions, so there is not much to do from the Termux side here - support for multiprocessing on Python would need upstream work.

fornwall on 1 Dec 2016

@fornwall ain't there any work arounds or guides or any resourceful links that can help enlighten us or even solve it

dekubi on 15 Feb 2017

❤2

The only workaround I know of is to use threading instead of multiprocessing.

fornwall on 15 Feb 2017

👍3

Python looks like it could utilize libandroid-shmem, which could make this a termux thing. Please reopen @fornwall

traverseda on 25 Jul 2017

@traverseda libandroid-shmem just emulates sysv shared memory functionality, not semaphores

vishalbiswas on 25 Jul 2017

Ahh, fair enough.

traverseda on 25 Jul 2017

@fornwall Not expert though, how to address this to python community? (Programming on mobile is cool idea for budding programmers). Pardon for irrelevant post.

RSwarnkar on 7 Dec 2019

This seems to break pip 20.1.

Running pip list --outdated results in:

ERROR: Exception:
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 160, in run
    packages = self.get_outdated(packages, options)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 167, in get_outdated
    return [
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 167, in <listcomp>
    return [
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 216, in iter_packages_latest_infos
    pool = Pool(DEFAULT_POOLSIZE)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/dummy/__init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 922, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

YtvwlD on 29 Apr 2020

This seems to break pip 20.1.

Yes, me too. I then use python -m pip install pip==20.0.2 to downgrade to the normal version on my phone.

sqliuchang on 2 May 2020

@traverseda libandroid-shmem just emulates sysv shared memory functionality, not semaphores

you should be able to emulate named semaphores with named shared memory and unnamed semaphores, so ashmem plus sem_init.

oaken-source on 15 Jul 2020

Solution to above problem. Use proot distros in termux. Android support semaphores but the termux being based on aarch64 thats new architecture thee are restrictions. But not on proot based distros as they use arm64 repos l. So install arch or ubuntu on termux

tathastu871 on 12 Sep 2020

👀1

Android support semaphores but the termux being based on aarch64 thats new architecture thee are restrictions. But not on proot based distros as they use arm64 repos

AArch64-specific restrictions are not exists.
AArch64 equals ARM-64. https://stackoverflow.com/questions/31851611/differences-between-arm64-and-aarch64/47274698#47274698

CPU architecture cannot apply restrictions on things that are purely software-implemented.

xeffyr on 12 Sep 2020

Android support semaphores but the termux being based on aarch64 thats new architecture thee are restrictions. But not on proot based distros as they use arm64 repos

AArch64-specific restrictions are not exists.

AArch64 equals ARM-64. https://stackoverflow.com/questions/31851611/differences-between-arm64-and-aarch64/47274698#47274698

CPU architecture cannot apply restrictions on things that are purely software-implemented.

Thats true now both are combined. And on android we can install and run both aarch64 and arm64 versions of programs. However termux natively follows aarch64 while all proot distros made for termux whether its arch, alpine, ubuntu etc installs arm64 versions from their repo.

There's no difrerence between them but one thing i have seen is all python ML programs requiring multiprocessing not work natively on termux because of error mentioned in this issue, but same programs run in ubuntu installed in termux. Also audacity,libreoffice, works in ubuntu termux but not in termux directly. Also using ffmpeg for 4k processing that requires high ram, termux normally crashes, but at same time if i use the same in ubuntu within termux it works. So it might be that proot providing some extra functionality for multiprocessing.

I was using opencv for K means clustering images and it gave above error. But same work in proot ubuntu

tathastu871 on 13 Sep 2020

@tathastu871 Normal semaphores are supported, that's true. Problem is only with named ones, i.e. using sem_open, sem_unlink etc.

Proof: https://github.com/aosp-mirror/platform_bionic/blob/29cff99e08746f8060f32eba03df3b95acf1163d/libc/bionic/semaphore.cpp#L124-L137

sem_t* sem_open(const char*, int, ...) {
  errno = ENOSYS;
  return SEM_FAILED;
}

int sem_close(sem_t*) {
  errno = ENOSYS;
  return -1;
}

int sem_unlink(const char*) {
  errno = ENOSYS;
  return -1;
}

If doubt whether mirror is legit, see this https://android.googlesource.com/platform/bionic/+/master/libc/bionic/semaphore.cpp#124.

As you can see, sem_open, sem_close, sem_unlink is just a stub returning -1 and errno set to ENOSYS meaning that system call is not supported.

Android kernel may support them though, yet on OS level they are not supported by libc and also can be blocked by SELinux and/or seccomp. To avoid issues, Termux does not provide their support to inform packages during configuring/build time that such functionality is unavailable.

sem.h is being removed from NDK include dir by our script: https://github.com/termux/termux-packages/blob/master/scripts/build/termux_step_setup_toolchain.sh#L198

Proot distribution works differently. First of all, there is used GNU libc which is natively supported by all GNU tools and 99.99% of existing open source software. For second, you can have a /dev/shm as non-root. Named semaphores should work there, unless related system calls are blocked by seccomp.

xeffyr on 13 Sep 2020

Dear MR Plyush,
thank you very much for yours professional explanation.