Pybind11: Ask opencv mat numpy array conversion
I try to add an Openv Numpy interface with type_caster
I start from @jampekka comments here
I ended to this :
#include <pybind11/numpy.h>
#include <opencv2/core/core.hpp>
#include <stdexcept>
namespace pybind11 { namespace detail {
template <> struct type_caster<cv::Mat> {
public:
/**
* This macro establishes the name 'inty' in
* function signatures and declares a local variable
* 'value' of type inty
*/
PYBIND11_TYPE_CASTER(cv::Mat, _("numpy.ndarray"));
/**
* Conversion part 1 (Python->C++): convert a PyObject into a inty
* instance or return false upon failure. The second argument
* indicates whether implicit conversions should be applied.
*/
bool load(handle src, bool)
{
array b(src, true);
buffer_info info = b.request();
int ndims = info.ndim;
decltype(CV_32F) dtype;
size_t elemsize;
if (info.format == format_descriptor<float>::format()) {
dtype = CV_32F;
elemsize = sizeof(float);
} else if (info.format == format_descriptor<double>::format()) {
dtype = CV_64F;
elemsize = sizeof(double);
} else if (info.format == format_descriptor<unsigned char>::format()) {
dtype = CV_8U;
elemsize = sizeof(unsigned char);
} else if (info.format == format_descriptor<unsigned char>::format()) {
if (ndims == 3) {
dtype = CV_8UC3;
} else {
dtype = CV_8U;
}
elemsize = sizeof(unsigned char);
} else {
throw std::logic_error("Unsupported type");
return false;
}
std::vector<int> shape = {info.shape[0], info.shape[1]};
std::vector<size_t> strides = {(size_t) elemsize * info.strides[0], (size_t) elemsize * info.strides[1], (size_t) elemsize * info.strides[2]};
value = cv::Mat(2,
&shape[0],
dtype,
info.ptr,
&strides[0]);
return true;
}
/**
* Conversion part 2 (C++ -> Python): convert an inty instance into
* a Python object. The second and third arguments are used to
* indicate the return value policy and parent object (for
* ``return_value_policy::reference_internal``) and are generally
* ignored by implicit casters.
*/
static handle cast(const cv::Mat &m, return_value_policy, handle defval)
{
std::cout << "m.cols : " << m.cols << std::endl;
std::cout << "m.rows : " << m.rows << std::endl;
std::string format = format_descriptor<unsigned char>::format();
size_t elemsize = sizeof(unsigned char);
int dim;
switch(m.type()) {
case CV_8U:
format = format_descriptor<unsigned char>::format();
elemsize = sizeof(unsigned char);
dim = 2;
break;
case CV_8UC3:
format = format_descriptor<unsigned char>::format();
elemsize = sizeof(unsigned char);
dim = 3;
break;
case CV_32F:
format = format_descriptor<float>::format();
elemsize = sizeof(float);
dim = 2;
break;
case CV_64F:
format = format_descriptor<double>::format();
elemsize = sizeof(double);
dim = 2;
break;
default:
throw std::logic_error("Unsupported type");
}
std::vector<size_t> bufferdim;
std::vector<size_t> strides;
if (dim == 2) {
bufferdim = {(size_t) m.rows, (size_t) m.cols};
strides = {elemsize * (size_t) m.cols, elemsize};
} else if (dim == 3) {
bufferdim = {(size_t) m.rows, (size_t) m.cols, (size_t) 3};
strides = {(size_t) elemsize * m.cols * 3, (size_t) elemsize * 3, (size_t) elemsize};
}
return array(buffer_info(
m.data, /* Pointer to buffer */
elemsize, /* Size of one scalar */
format, /* Python struct-style format descriptor */
dim, /* Number of dimensions */
bufferdim, /* Buffer dimensions */
strides /* Strides (in bytes) for each index */
)).release();
}
};
}} // namespace pybind11::detail
The cast part work, i'm able to transfert cv::Mat from C++ to numpy array in python.
but the load part is completely broken there is buffer issues. it seem strides were bad but I don't find how to fixe them.
have you an idea to fixe that ? thx
edmBernard
All 18 comments
{(size_t) elemsize * info.strides[0], (size_t) elemsize * info.strides[1], (size_t) elemsize * info.strides[2]};
What's the logic here? Strides are already expressed in bytes. That's probably the cause of your errors.
Besides, note that a lot of your code is not outdated / deprecated, there's new/better ways to do these things. Also, you may not need to do a .request() at all, see below.
- >
array b(src, true);
Direct constructors from (handle, bool), are deprecated; instead:
if (!isinstance<array>(src))
return false;
auto arr = reinterpret_borrow<array>(src);
- >
std::vector<int> shape = {info.shape[0], info.shape[1]};
const size_t* shape = arr.shape();
- >
std::vector<size_t> strides = {(size_t) elemsize * info.strides[0], (size_t) elemsize * info.strides[1], (size_t) elemsize * info.strides[2]};
const size_t* strides = arr.strides();
- >
int ndims = info.ndim;
int ndim = arr.ndim();
- >
info.ptr
arr.mutable_data() /* or arr.data() if you only need a const pointer */
aldanor
on 30 Nov 2016
thx for your answer i'll test it soon.
Just to complete my solution finally it work with small modifications (see below).
My issues come from the python side and mutable objects, when i split channel data here not continuous anymore
a = cv2.imread("my_image.png")
my_function(a) # worked
b=a[:,:,0]
my_function(b) # don't worked
c=copy.deepcopy(b)
my_function(c) # worked
PS: I'm aware my code use deprecated things but i don't find enough help on the non deprecated solutions.
working code :
#include <pybind11/numpy.h>
#include <opencv2/core/core.hpp>
#include <stdexcept>
namespace pybind11 { namespace detail {
template <> struct type_caster<cv::Mat> {
public:
/**
* This macro establishes the name 'inty' in
* function signatures and declares a local variable
* 'value' of type inty
*/
PYBIND11_TYPE_CASTER(cv::Mat, _("numpy.ndarray"));
/**
* Conversion part 1 (Python->C++): convert a PyObject into a inty
* instance or return false upon failure. The second argument
* indicates whether implicit conversions should be applied.
*/
bool load(handle src, bool)
{
/* Try a default converting into a Python */
array b(src, true);
buffer_info info = b.request();
int ndims = info.ndim;
decltype(CV_32F) dtype;
size_t elemsize;
if (info.format == format_descriptor<float>::format()) {
if (ndims == 3) {
dtype = CV_32FC3;
} else {
dtype = CV_32FC1;
}
elemsize = sizeof(float);
} else if (info.format == format_descriptor<double>::format()) {
if (ndims == 3) {
dtype = CV_64FC3;
} else {
dtype = CV_64FC1;
}
elemsize = sizeof(double);
} else if (info.format == format_descriptor<unsigned char>::format()) {
if (ndims == 3) {
dtype = CV_8UC3;
} else {
dtype = CV_8UC1;
}
elemsize = sizeof(unsigned char);
} else {
throw std::logic_error("Unsupported type");
return false;
}
std::vector<int> shape = {info.shape[0], info.shape[1]};
value = cv::Mat(cv::Size(shape[1], shape[0]), dtype, info.ptr, cv::Mat::AUTO_STEP);
return true;
}
/**
* Conversion part 2 (C++ -> Python): convert an inty instance into
* a Python object. The second and third arguments are used to
* indicate the return value policy and parent object (for
* ``return_value_policy::reference_internal``) and are generally
* ignored by implicit casters.
*/
static handle cast(const cv::Mat &m, return_value_policy, handle defval)
{
std::cout << "m.cols : " << m.cols << std::endl;
std::cout << "m.rows : " << m.rows << std::endl;
std::string format = format_descriptor<unsigned char>::format();
size_t elemsize = sizeof(unsigned char);
int dim;
switch(m.type()) {
case CV_8U:
format = format_descriptor<unsigned char>::format();
elemsize = sizeof(unsigned char);
dim = 2;
break;
case CV_8UC3:
format = format_descriptor<unsigned char>::format();
elemsize = sizeof(unsigned char);
dim = 3;
break;
case CV_32F:
format = format_descriptor<float>::format();
elemsize = sizeof(float);
dim = 2;
break;
case CV_64F:
format = format_descriptor<double>::format();
elemsize = sizeof(double);
dim = 2;
break;
default:
throw std::logic_error("Unsupported type");
}
std::vector<size_t> bufferdim;
std::vector<size_t> strides;
if (dim == 2) {
bufferdim = {(size_t) m.rows, (size_t) m.cols};
strides = {elemsize * (size_t) m.cols, elemsize};
} else if (dim == 3) {
bufferdim = {(size_t) m.rows, (size_t) m.cols, (size_t) 3};
strides = {(size_t) elemsize * m.cols * 3, (size_t) elemsize * 3, (size_t) elemsize};
}
return array(buffer_info(
m.data, /* Pointer to buffer */
elemsize, /* Size of one scalar */
format, /* Python struct-style format descriptor */
dim, /* Number of dimensions */
bufferdim, /* Buffer dimensions */
strides /* Strides (in bytes) for each index */
)).release();
}
};
}} // namespace pybind11::detail
@aldanor I try your modification but i'm not able to find a equivalent of :
info.format to got the data element type
edmBernard
on 30 Nov 2016
Yes, what I think happens is this: the array is non-contiguous, and as such the data pointer stops making much sense. PyArray_Check() works just fine, so the handle is happily accepted as a valid array. Note that array_t has some conversion logic through template-level flags, but array doesn't. As of right now, you probably want to manually check if the array is c-style and contiguous.
@wjakob ^ This is quite closely related to flags rework, how do you think cases like this should be solved, ideally/hypothetically? If instead of array one used e.g. generic_array<array::c_array>, would that help here in load()? (presumably, that would make a copy of data in case it's non-contiguous, but then who would own it once you leave the load() function?)
aldanor
on 30 Nov 2016
@aldanor I try your modification but i'm not able to find a equivalent of :
info.format to got the data element type
@edmBernard There's py::dtype::of<float>() and arr.dtype() which you can compare, which is the same as comparing numpy dtypes (this handles a few more corner cases, like int being the same as long on Windows, but with different format strings, structured types, etc). However, as of right now you would have to use PyArray_EquivTypes_() directly.
Maybe we should properly expose this so that pybind11::dtype either gains an operator==(), or equiv_to(py::dtype) method, or equiv_to<T>(), or something like that. Then you could hypothetically check if (arr.dtype().equiv_to<float>()) which would be quite nice.
aldanor
on 30 Nov 2016
When i try this api.PyArray_EquivTypes_(arr.dtype(), dtype::of<float>())
to check type, it fail with error: cannot convert ‘pybind11::dtype’ to ‘PyObject* {aka _object*}’ in argument passing
but there is an awfull solution : arr.dtype() == dtype(format_descriptor<unsigned char>::format()) that seem to work ^^ (i don't check if there is border effect)
edmBernard
on 30 Nov 2016
I put my 2 versions on git here :
https://github.com/edmBernard/pybind11_opencv_numpy
edmBernard
on 30 Nov 2016
I close as my solution finally work even if it not really clean
Thx for your help
edmBernard
on 1 Dec 2016
I've modified @edmBernard 's sample project and replaced the converter with extracts from https://github.com/yati-sagade/opencv-ndarray-conversion ... which really just copies OpenCV's internal cv::Mat <--> ndarray conversion routines so they can be used separate from the main module.
Ultimately, I'd like to add a capsule to OpenCV that exports it's conversion routines so that anyone can use them, instead of just using it internally. Would make this sort of thing a lot easier. Won't be able to finish that for awhile, but WIP for that at https://github.com/virtuald/opencv/commit/6db6d29b7f49354db9d9a3a0bd071e39c25960bb
Example project is at https://github.com/virtuald/pybind11_opencv_numpy, check it out, it seems to work so far.
virtuald
on 20 Jan 2017
@virtuald Using OpenCV's code is indeed probably the best approach. When I first started hacking the conversion, that was my main idea, but OpenCV doesn't make using just the conversion too easy. In fact its python bindings are such magic I couldn't even start to guess where to look.
jampekka
on 30 Jan 2017
@jampekka As I said above, I've extracted the necessary bits from OpenCV's python bindings and incorporated them into the example project above, you can find them at https://github.com/virtuald/pybind11_opencv_numpy/blob/master/ndarray_converter.cpp .
virtuald
on 30 Jan 2017
@virtuald thanks for your work, That's a really better solution.
edmBernard
on 30 Jan 2017
Is it not better to try to merge this solution in the repository?
bhack
on 15 Sep 2017
@bhack the real solution is adding a capsule to OpenCV for this. It probably doesn't belong in pybind11.
virtuald
on 15 Sep 2017
Capsule?
bhack
on 15 Sep 2017
See https://github.com/virtuald/opencv/commit/6db6d29b7f49354db9d9a3a0bd071e39c25960bb for an initial (crashing) implementation of a capsule, similar to how one imports numpy from a C/C++ project.
virtuald
on 15 Sep 2017
@vpisarev What do you think? Could it be upstreamed in opencv?
bhack
on 15 Sep 2017
Gently ping @vpisarev
bhack
on 27 Sep 2017
Related issues
neutralid
·
3Comments
CoinCheung
·
3Comments
c-f-h
·
3Comments
patrikhuber
·
3Comments
nschloe
·
3Comments
Most helpful comment
@jampekka As I said above, I've extracted the necessary bits from OpenCV's python bindings and incorporated them into the example project above, you can find them at https://github.com/virtuald/pybind11_opencv_numpy/blob/master/ndarray_converter.cpp .