Opencv-python: ARM support

[skvark]

ARM builds would be needed to support for example Raspberry Pi.

[skvark]

...but manylinux tag does not support ARM?

[gkvoelkl]

Thanks for making that great Python installer.

When will there be a version for raspberry pi?

Best regards

Gerhard

[skvark]

Pre-built wheel distributions which target ARM based Linux devices can't be done with current toolchains. The PEP (https://www.python.org/dev/peps/pep-0513/) which defines the manylinux tag recognizes only certain architectures (x86, i686).

Please see also this issue: https://github.com/pypa/manylinux/issues/84

However, Python's packaging system is evolving all the time and this might be possible at some point in the future.

[gkvoelkl]

I hope that time will come fast.

I have seen some packages that could be installed with pip

Examp. https://github.com/samjabrahams/tensorflow-on-raspberry-pi

Do they create their own toolchain?

[skvark]

Yes, they have created their own special Docker image with all the correct dependencies and it runs on actual RPi hardware. Tensorflow itself has some tools to generate the package after the build.

What I would need to make this happen:

• similar Docker image as the Cent OS 5 manylinux image is but with some RPi /ARM baseline which is common for all devices - this will probably require QEMU based image or some image with cross-compilation toolkit like Scratchbox2
• I must be able to run the images on Travis
• afaik the resulting wheels cannot be distributed via PyPI so I would need to host them manually somewhere

This is not a task that can be done over a weekend, it will require most probably several weeks, maybe months, of work.

[illume]

Raspberry pi have a build system where they try to build all the pypi packages -- piwheels.

piwheels website: https://www.piwheels.hostedpi.com/
blogpost about what it is: http://bennuttall.com/piwheels-building-a-faster-python-package-repository-for-raspberry-pi-users/
FAQ: https://www.piwheels.hostedpi.com/faq.html

They are receptive to PRs to include Debian dependencies here: https://github.com/bennuttall/piwheels/

I guess all your dependencies aren't in the Debian they use though? Also, compiling on real hardware can take a long time for big C++ libraries.

[sctv]

In the latest raspian it automatically downloads the piwheels if they exist.
(opencv-python piwheels for rpi arm)] for python 3, no python 2. Is that a problem or a feature?

[skvark]

I have no idea. I do not maintain those packages and I don't know how they were created.

[bennuttall]

We haven't built Python 2 wheels - we might do in future. But there's not much future left.

[sctv]

@bennuttall only use case I have for python2 wheels is for dependencies for Inkscape extensions, as Inkscape still uses python2.

[popjxc]

I try to build this project on QEMU based docker image, it quite slow and often make docker build timeout. it's really painful...

[cooldeaaap1]

when Can I "pip install opencv_contrib_python" in Raspberry Pi?

[skvark]

PyPi does not support ARM wheels yet. I have no idea if it will get support for ARM wheels and if it does when it happens.

[bennuttall]

It does! But I wouldn't recommend publishing Arm wheels, it's a bit messy. See https://github.com/bennuttall/piwheels/issues/66

Raspberry Pi users can use piwheels (configured by default in Raspbian) to get pre-built packages which work on all Raspberry Pi models. We just haven't built for Python 2 so far.

[waveform80]

I should add, I recently added the *-headless builds from this repo too (thanks to @skvark's fine work!), so you can "pip install opencv_contrib_python_headless" too. I watch this repo, so when there's a new release I try to add it to piwheels as soon as I'm able, but there'll usually be some delay.

[cooldeaaap1]

@bennuttall I am sorry that :
I have use pip install opencv_contrib_python-3.2.0.7-cp35-cp35m-linux_armv7l.whl and It seems OK, but when I
python import cv2
It told me that:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pi/.pyenv/versions/3.5.2/lib/python3.5/site-packages/cv2/__init__.py", line 7, in <module> from . import cv2 ImportError: libhdf5_serial.so.100: cannot open shared object file: No such file or directory
even
sudo apt-get update sudo apt-get install libhdf5-dev sudo apt-get update sudo apt-get install libhdf5-serial-dev
It didnot work ...
How can I makr it work ?

[waveform80]

@cooldeaaap1 libhdf5-dev should be pulling in libhdf5-100 which in turn provides /usr/lib/arm-linux-gnueabihf/libhdf5_serial.so.100 (you don't actually need the *-dev packages for installation - those are for building - but they pull in the underlying library anyway, so no harm).

[waveform80]

@cooldeaaap1 when you say sudo apt-get update; sudo apt-get install libhdf5-dev libhdf5-serial-dev didn't work, do you mean installing them didn't work, or importing cv2 after installing them didn't work? If the former, then I'd need to see the error from apt-get. If the latter I'd have to ask if you were running the latest Raspbian or not?

[cooldeaaap1]

@waveform80
Thank for your help , I change to the latest Raspbian , I want opencv-contrib-python==3.2.0.7;but I got error

ImportError: /usr/local/lib/python3.5/dist-packages/cv2/cv2.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: hb_buffer_get_glyph_infos
change to others ,like 3.3.0.9 ,it works, Is it a bug ?

[walchko]

PyPi does not support ARM wheels yet.

@skvark, thanks for the great work. I am confused when you say pypi doesn't support arm, it does. Unless I am misunderstanding you, if you look at something I published sslam-0.0.3-cp27-cp27mu-linux_armv7l.whl it is a wheel file for armv7. I did nothing special, here is my setup.py, just a simple c extension. I did build the package on a real raspberry pi, not travis, because that is only x86. Also, my stuff is trivial compared to opencv. Again, I might be misunderstanding your comment. :smile:

[waveform80]

@cooldeaaap1 could be a bug in 3.2 (there's something vaguely similar mentioned in https://github.com/opencv/opencv_contrib/issues/1175) but if 3.3 works then I'd just stick with that (or later).

@walchko See last FAQ at https://www.piwheels.org/faq.html - it's slightly out of date in that pypi.org is now the "next gen" PyPI that's talked about but the rest of it still stands (there's tons of ARM architectures and stuff compiled on Raspbian on a Pi isn't "true" ARMv7 because it still has to be compatible with the ARMv6 userland).

[cooldeaaap1]

@waveform80 I am sorry that : I had better to use "3.2.0.7" ,because I want to use a prject https://github.com/AirtestProject/Airtest
and it's requirements contains "opencv-contrib-python 3.2.0.7" https://github.com/AirtestProject/Airtest/blob/master/requirements.txt

[skvark]

@skvark, thanks for the great work. I am confused when you say pypi doesn't support arm, it does. Unless I am misunderstanding you, if you look at something I published sslam-0.0.3-cp27-cp27mu-linux_armv7l.whl it is a wheel file for armv7. I did nothing special, here is my setup.py, just a simple c extension. I did build the package on a real raspberry pi, not travis, because that is only x86. Also, my stuff is trivial compared to opencv. Again, I might be misunderstanding your comment. 😄

Yeah, you can upload arbitrary ARM builds to PyPI nowadays. My comment was a bit misleading. What I mean is that there is no universal support for different ARM based boards. Yes, I could start building these packages for ARM on a Raspberry Pi but that does not mean that the packages are compatible with other boards.

Therefore, it does not make any sense for this repository to start supporting one product line (RasPi) as there are many different ARM based boards. For example almost all mobile phones which are running Android are ARM based (!). It would be cool to run pip install opencv-python on my Sailfish OS phone, but that's not just yet possible with the current state of Python package management. Please see https://github.com/pypa/manylinux/issues/84.

However, thanks to @bennuttall's and @waveform80's great work, you can use pre-built wheels on Raspberry Pi's. For more information, see https://www.piwheels.org/.

[Veilkrand]

@skvark I'm in the same situation described before. After installing opencv-contrib-python-3.4.2.17 using piwheels and importing cv2 in python3 it shows:
ImportError: libhdf5_serial.so.100: cannot open shared object file: No such file or directory

I have tried to reinstall libhdf5 and libhdf5_serial but it didn't resolved the problem. I'm using Ubuntu Mate in RPI. Any insights? Thanks

[skvark]

I don't maintain Raspberry Pi builds. Please open an issue to https://github.com/bennuttall/piwheels.

[native-api]

Can Travis build for ARM?

According to https://github.com/travis-ci/travis-ci/issues/2790, only cross-compiling to Android is available out of the box.

https://github.com/Pi4J/pi4j/blob/master/.travis.yml is an example showing it's possible to cross-compile for Pi with a downloaded "rpi-tools" package.

As per Running Travis CI tests on ARM architecture | tomaz.me, it's also possible to run tests with qemu.

Finally, https://bennuttall.com/piwheels-building-a-faster-python-package-repository-for-raspberry-pi-users/ and https://www.piwheels.hostedpi.com/ say that the piwheels project compile everything from PyPI (though, if the latter is correct, only for py3.4 and py3.5 -- looks strange, maybe the info is outdated). So all we need to do is provide an sdist that they can run pip wheel <pkg> on.

[waveform80]

Finally, https://bennuttall.com/piwheels-building-a-faster-python-package-repository-for-raspberry-pi-users/ and https://www.piwheels.hostedpi.com/ say that the piwheels project compile everything from PyPI (though, if the latter is correct, only for py3.4 and py3.5 -- looks strange, maybe the info is outdated). So all we need to do is provide an sdist that they can run pip wheel on.

We attempt to build everything from PyPI (and keep everything that succeeds :). As OpenCV is a little more tricky (as you noted there's no sdist that works "out of the box") we handle that separately; specifically I watch this repo and when @skvark makes a new release I build them on my Pis and import them into piwheels when they're done (usually a couple of days later!). Tensorflow is also handled separately (specifically, the maintainers send Ben the pre-built wheels and he imports them).

We only handle py34 and py35 at the moment because we (currently) only support Raspbian Jessie (py34) and Raspbian Stretch (py35). At some point, when I've got a spare couple of weekends, we'll add py27 to that mix but it's not a trivial addition (the build queue logic is already fairly complex and multiple major versions makes things even more complex).

We probably won't be supporting anything more recent until Raspbian Buster makes an appearance (there was a vague attempt at supporting py36 via Ubuntu images on the builders, but that just caused more trouble than it was worth because some py36 users had built their own on Raspbian, some were using Ubuntu, and thus each had different lib versions installed so we're sticking strictly to Raspbian for now).

[waveform80]

Anyway, as noted above - if you've any issues with the piwheels builds, do feel free to open issues at https://github.com/bennuttall/piwheels

[bennuttall]

To resolve the "cannot open shared object file" error, see https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/

[native-api]

Travis now supports natively building for the ARM architecture: https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

It's currently in alpha so be aware of possible problems.

[illume]

In particular, no language: python support at the moment (so you have to build/install python yourself and such).

[Red-Eyed]

I wonder, why not to use cross compilation?

[AshokBhat]

@skvark There is an option to get Arm servers via the WorksOnArm project. Please see the https://github.com/WorksOnArm/cluster/issues/209 for details. Tagging @vielmetti for visibility.

[vielmetti]

There has also been some progress in getting wheels built for some Python projects for arm64 - notably this report from numpy at https://twitter.com/pypackages/status/1274444742621888513 .

There's a couple of options through the Works on Arm effort mentioned above:

• shared CI on Drone Cloud w/native arm64 support
• shared CI on Travis CI w/native arm64 support
• dedicated CI on a dedicated machine directly through Works on Arm

the preference for dedicated systems is for projects that have some compute density needed - ideally you'd want to build more than one package, or be working on a system that is a critical piece not yet ported but that lots of other things depend on, etc. Much of the time, Drone or Travis is a more effective way of getting CI done than dedicated systems.

[nyanye]

Hello, I guess this entire thread is yet focused on embedded devices such as raspberry pi however is there any further plan to support ARM64 architecture and corresponding wheels for new Apple silicon since there is no wheel compiled for macOS / arm64? Thanks!

[skvark]

Please use the search function, there is a dedicated issue for arm64.

[skvark]

The latest release includes experimental arm64 (aarch64) Linux wheels. Older arm variants will not be supported. Please use PiWheels or compile from source if you need armv7 / armv6 wheels.

Linux arm64 (aarch64) tracking issue: https://github.com/skvark/opencv-python/issues/366
Apple Silicon arm64 tracking issue: https://github.com/skvark/opencv-python/issues/429