Opencv-python: Include headers to support binary linking?

[virtuald]

First off, thanks for this project, it saves a ton of time and I'm really happy that it exists. It makes it a lot easier to get OpenCV installed for the high school students I work with.

I'd like to take a C++ library that uses OpenCV, wrap it with cython, and link it to the OpenCV bits in the binary provided by the installed wheel. This would allow users to just install the wheel of the other library + the opencv-python wheel, as opposed to having to compile their own version.

Has anyone tried this yet? A cursory look at the linux .so files indicate that it exports all of the normal OpenCV symbols, so in theory if you just include the right headers and link to it then it should just work (though maybe not for Windows)... I think that one would add an additional cv2 module (cv2.distutils?) that could provide a list of includes/libraries, similar to how numpy works.

If there's no opposition to this, I might try this out in a week or three to see how it goes.

[skvark]

Thanks! I started this project because I wanted to make the installation as easy as possible and I'm happy to hear that I have reached that target.

I haven't tried that, it might be possible. Trying it shouldn't break anything so you may go ahead :)

[virtuald]

TL;DR: it's definitely possible, but it sucks and would need a bunch of integration work + testing

I played with this a bunch on Linux today, here's what I've found so far:

• I think this requires you to use a shared library version of opencv, because it doesn't seem like you can link directly against the statically linked cv2 python module... or at least I haven't found the right combination of flags to allow you to do so
• Assuming they all end up in site-packages in the cv2 directory, the shared libraries need to be able to find each other, so you have to set the rpath (can use a cmake arg: CMAKE_INSTALL_RPATH=$ORIGIN)
• Special magic required for a python module that tries to link to this stuff
  
  
  
  • Needs to find the special include file for the API (presumably in site-packages in the cv2 directory)
  
  
  • Needs to find the libraries on site-packages, and user needs to know which libraries they want to link to
  
  
  • Need to set the RPATH on the user's custom python module to $ORIGIN/../cv2 -- though, this magic varies on where your compiled module ends up, and it appears that different install mechanisms may put your binary in different places (setuptools vs distutils)
  
  
• I created a patch for OpenCV to export the numpy <-> cv2 array conversion routines, it's not so bad and I'll approach upstream about it even if this isn't a successful effort.

It's really surprising to me that nobody has tried to do this sort of thing before, but it seems that people don't mix OpenCV python and OpenCV c++, except as a custom exe. I'm undecided if I'm giving up yet, but I welcome any input from others who may have ideas.

[auscompgeek]

What about passing --allow-shlib-undefined to the linker?

[native-api]

@virtuald:

It's really surprising to me that nobody has tried to do this sort of thing before

There's no gain here, and you're tryng to use a Python module for something that it's not designed to -- that's why.

cv2.so/.pyd is a Python library and is supposed to be used via Python interface. Python's packaging system is intentionally not designed to ship C libraries.

• If you're writing a Python extension, use it through Python's C interface as a Python module should.
• If you wish to use OpenCV's C++ interface, use your own copy of OpenCV. There's absolutely no difference functionality-wise -- you'll get all the same functions.
  
  
  
  • The only functional difference would be if you wish to access the same OpenCV object instances both as wrapped into Python objects and as underlying C++ objects. But that will require a layer of middleware in cv2 to enforce correct shared access due to Python and C++ having different memory models, different object lifecycle management, locking etc -- all for questionable gain.
    
    
    
    
    
    • The only package I know of that exposes both Python and C interface is numpy. I'm sure that it does have such middleware layer so this is where you should look if you wish to pursue this further for whatever reason.
    
    
    
  
  

[virtuald]

@native-api I would like to use a library that is written using OpenCV/C++ from python. Not duplicating a ton of carefully written C++ code seems like a huge gain to me.

I agree that python's packaging system isn't designed to do this, and that accomplishing my goal using opencv-python would be tricky (and perhaps not possible).

My target audience are high school students primarily using Windows. If I could ship a wheel that bundled/wrapped my OpenCV/C++ dependency that depended on opencv-python, that would be fantastic and significantly lower the bar for users (particularly on Windows) to use such a library. The dependency I'm using is getting Windows support this year, so I'll probably dig into this yet again in the coming months.

[native-api]

This project ships cv2.so/.pyd exactly as OpenCV build script builds it -- statically linked. There's no way to change this via build options AFAICS, so you should go to the OpenCV project anyway if you wish to change that.

There are other package managers that work in Windows and _can_ ship C libraries separately -- e.g. Chocolatey and Anaconda -- and already have packages for Python and OpenCV. To achieve that, they are forced to put additional restrictions to avoid dependency hell that Python's ecosystem doesn't -- like same compilers and compiler options for everyone, only a single instance of a library per environment and/or strict file naming and placements conventions and little interoperability with other software installed on the system.
If that doesn't bother you, you can make your product as a package for one of those package managers.

[virtuald]

Yes, I'm aware of all of that. I'd rather use stock python + wheels, I think it's a much simpler proposition from a novice user perspective (though certainly more difficult from a packager perspective).

It is possible to build cv2.so/pyd to use shared libraries (perhaps you didn't read above?). And yes, as _also_ mentioned above, it is likely that there are some things that would need to be patched upstream to make this work.

Your input is welcome if you have other ideas on how one might be able to get this to work.

[native-api]

It is possible to build cv2.so/pyd to use shared libraries (perhaps you didn't read above?)

There's no way to change this via build options AFAICS, so you should go to the OpenCV project anyway if you wish to change that.

[virtuald]

Historically cmake -DBUILD_SHARED_LIBS=ON ... has built a cv2 python module that utilizes the libopencv*.so/dll libraries.

[virtuald]

Just tried it on my mac:

[100%] Linking CXX shared module ../../lib/python3/cv2.cpython-36m-darwin.so
[100%] Built target opencv_python3
user@host ~/dev/ext/opencv/build
$ otool -L lib/python3/cv2.cpython-36m-darwin.so 
lib/python3/cv2.cpython-36m-darwin.so:
    @rpath/libopencv_ml.3.4.dylib (compatibility version 3.4.0, current version 3.4.4)
    @rpath/libopencv_dnn.3.4.dylib (compatibility version 3.4.0, current version 3.4.4)
    @rpath/libopencv_shape.3.4.dylib (compatibility version 3.4.0, current version 3.4.4)
[snip]

[virtuald]

Looks like someone already filed a bug for creating a capsule: https://github.com/opencv/opencv/issues/8872

[virtuald]

Haven't done anything in particular towards this, but recently we've done something similar that is relevant.

Our new robot libraries now wrap existing C++ libraries, consisting of lots of different .so/.dll/.dylib files. Our new build system (https://github.com/robotpy/robotpy-build) generates python wrappers for the C++ libraries, embeds the shared library version of the C++ library in the python package, adds include files, and then autogens a snippet that looks like this:

from ctypes import cdll
_lib = cdll.LoadLibrary(join(_root, "lib", "libwpilibc.so"))

As long as that snippet is imported before the python bindings are imported, the library symbols are resolved and everything works great. We even are shipping wheels that depend on libraries shipped in other wheels. The only exception to this is on OSX, where we have to fix up the rpath to include a site-packages relative @loader_path section.

This technique could be adapted for opencv-python instead of using statically linked libraries. However, to be truly useful, would really want https://github.com/opencv/opencv/issues/8872 to be resolved (as mentioned above).

[lukacu]

Hi, I found this issue because I am interested in the same thing, it seems silly that you would include OpenCV two times. From my experience, PyTorch is doing something similar, there is an extension that links the core shared libraries, bundled in the Python package. The package also includes C++ headers and some python functions to locate them. You can then compile a PyTorch C++ extension and link it with torch C++ libraries. In Python code you have to first load torch Python C++ module that loads shared libraries and you can also load your extension. Would something like that work for OpenCV as well?