Opencv_contrib: Using Facemark API (Python), Version 4.0.0 - pre : bad alloc error

[krshrimali]

System information (version)
OpenCV => 4.0.0 - pre
Operating System / Platform => Ubuntu 18.04

Detailed Description

I'm trying to implement FacemarkLBF testing version in Python (OpenCV Version : 4.0.0 - pre), but there is not enough documentation on it. Addition to that, when am trying to run, it shows std::bad alloc error.

Gist of the functions am using:

obj = cv2.face.createFacemarkLBF()
cv2.face_Facemark.loadModel(obj, <model_name>)
retVal, landmarks = cv2.face_Facemark.fit(obj, img, faces)

While trying to debug, the problem is mainly with roi.getObj() line in fit function definition (https://github.com/opencv/opencv_contrib/blob/master/modules/face/src/facemarkLBF.cpp : 376 line), which doesn't work because of which when faces[i] is accessed, there's an error of bad allocation.

Any solutions / comments on this, please?

[berak]

hi, @krshrimali

well, i added the python / java bindings, but could only test the java ones properly, so this is probably my bad here.

indeed the roi.getObj() code is the problem:

https://github.com/opencv/opencv_contrib/blob/2231018c839d728811a39556ec83741bf9a27614/modules/face/src/facemarkLBF.cpp#L373-L376

the generated python wrappers for the fit() function produce this:

static PyObject* pyopencv_cv_face_face_Facemark_fit(PyObject* self, PyObject* args, PyObject* kw)
{
    using namespace cv::face;

    cv::face::Facemark* _self_ = NULL;
    if(PyObject_TypeCheck(self, &pyopencv_face_Facemark_Type))
        _self_ = dynamic_cast<cv::face::Facemark*>(((pyopencv_face_Facemark_t*)self)->v.get());
    if (_self_ == NULL)
        return failmsgp("Incorrect type of self (must be 'face_Facemark' or its derivative)");
    {
    PyObject* pyobj_image = NULL;
    Mat image;
    PyObject* pyobj_faces = NULL;
    Mat faces;
    PyObject* pyobj_landmarks = NULL;
    vector_Mat landmarks;
    bool retval;

    const char* keywords[] = { "image", "faces", "landmarks", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|O:face_Facemark.fit", (char**)keywords, &pyobj_image, &pyobj_faces, &pyobj_landmarks) &&
        pyopencv_to(pyobj_image, image, ArgInfo("image", 0)) &&
        pyopencv_to(pyobj_faces, faces, ArgInfo("faces", 0)) &&
        pyopencv_to(pyobj_landmarks, landmarks, ArgInfo("landmarks", 1)) )
    {
        ERRWRAP2(retval = _self_->fit(image, faces, landmarks));
        return Py_BuildValue("(NN)", pyopencv_from(retval), pyopencv_from(landmarks));
    }

see the Mat faces there ? the situation is similar to this:

// vector<Rect> f1; // would be ok.
Mat f1;            // problem !
InputArray f2(f1);
void *o = f2.getObj(); // o has the address of a Mat, not a vector<Rect>
std::vector<Rect> & faces = *(std::vector<Rect> *)o;  // oh noes, bad cast!
cout << faces.size(); // entire garbage

the java wrappers are a bit more lucky here:

        std::vector<Rect> faces;
        Mat& faces_mat = *((Mat*)faces_mat_nativeObj);
        Mat_to_vector_Rect( faces_mat, faces ); // explicit copy

but they had to be told, to do so:

https://github.com/opencv/opencv_contrib/blob/2231018c839d728811a39556ec83741bf9a27614/modules/face/misc/java/gen_dict.json#L12-L13

solution ? i have no immediate one.
maybe the python wrapppers can be made smarter ?
maybe the interface can be refactored (swap InputArray for a vector<Rect> ) ?
maybe we can find a way to retrieve a vector from an InputArray without a horrible cast ?
none of it nice or easy.

[krshrimali]

Hi, @berak
Thanks for your response. Yes, none of the solutions look that easy.

May be, I'll give it a shot once. Let's see how it goes. Thanks for your analysis and details.

[berak]

i might have found a solution: just accept, that we have a Mat here, and treat it like that.

instead of casting a void pointer:

 bool FacemarkLBFImpl::fit( InputArray image, InputArray roi, OutputArrayOfArrays  _landmarks ) 
 { 
     // FIXIT 
    std::vector<Rect> & faces = *(std::vector<Rect> *)roi.getObj(); 

retrieve the vector<Rect> from a Mat:

bool FacemarkLBFImpl::fit( InputArray image, InputArray roi, OutputArrayOfArrays  _landmarks )
{
    std::vector<Rect> faces;
    Mat roimat = roi.getMat(); // see issue #1661
    if ((!roimat.empty()) && (roimat.type()==CV_32SC4))
        faces.insert(faces.begin(), roimat.begin<Rect>(), roimat.end<Rect>());
    if (faces.empty()) return false;

this would work with both vector<Rect> and Mat also add a bit of type safety, imho.
(ofc. the AAM and kazemi classes would need the same treatment)

@krshrimali , could you try it ? also, i'd be glad about any comments !

[berak]

@krshrimali i cannot find it ! do you have a line number for that ?

[krshrimali]

@berak
Am sorry, it was my bad. It builds successfully as of now. Trying to run the code now.
Is this the right code sample am using?

obj = cv2.face.createFacemarkLBF()
obj.loadModel("lbfmodel.yaml")
landmarks = obj.fit(img, faces)

(faces is an array, which in my case it is: [[138 102 171 171]])

Running the above sample, returns False from the fit function.
EDIT: And looks like : it's returning false because faces.empty() outputs True.

[krshrimali]

Update:

Earlier I was using cascade detector to detect faces, this time I used getFacesHAAR() function and since it returns a tuple, I had to input list(faces)[1] to fit function.

This returns segmentation fault though, but passes faces.empty() stage.

EDIT :
The segmentation fault is in fitImpl() function, in the beginning where it accesses landmarks.size(). And there is further segmentation fault in:

landmarks = Mat(shape.reshape(2) + Scalar(min_x, min_y)

[berak]

thanks for all your help, btw !

it's weird, roimat.type() is CV_32SC4 when the input comes from c++ or java, but CV_32S, when it's from python. i guess, it's better, to check

 roimat.depth() == CV_32S

instead.

(that's why it returned an empty landmarks list)

[krshrimali]

@berak Thanks! I did try replacing
roimat.type() == CV_32SC4
with roimat.depth() == CV_32S. There seems to be a segmentation fault error. I think it must be because of the edits I had made earlier to the source code. I'll try and undo them.

And can you please explain why difference in C++/Java with Python for roimat.type()?

[berak]

yes, i get the segfault too, still inquiring.
java & python wrappers are handling std::vector differently (unfortunately)

for a single face, i get a [1 x 1] Mat with type CV_32SC4, while from python i get [4 x 1] with type CV_32S (so, channels vs. columns)

and the python version crashes then, because the MatIterator tries to makes 4 Rects of it, and goesout of bounds this way.

i'll try with a reshape(4,matroi.rows)

[krshrimali]

@berak
Everything runs!!!!
Except this segmentation fault doesn't go. I tried printing something just before the fit() function returns boolean value in the source code, and it does print. Also the landmarks are also printing (as my image has only one face).

I tried reducing the call to:

params.detectROI = faces[0];
fitImpl(image.getMat(), landmarks[0]);

And when I do cout << landmarks[0] << endl;, it successfully gives me the 68 point landmarks.
But then it then gives segmentation fault.

I believe, there might be the problem with faces.size() and/or it's return type, may be?

[berak]

yea, it tries to call fitImpl() several times (see above). the 1st iteration is correct, just the next ones access illegal memory

[krshrimali]

Ohkay. So a reshape should do it? Can you tell the exact correction please?

[berak]

well, no solution yet.

while

Mat roimat = roi.getMat();
std::vector<Rect> faces = roimat.reshape(4,roimat.rows);
if (faces.empty()) return false;

would fix problem #1 ,it still segfaults with the landmarks now (similar problem).

c++ and java use a vector<vector<Point2f>>, while python has a vector_Mat there (look at the gennerated code above). and it segfaults after leaving the fit() function, in pyopencv_to().

there are even correct pyopencv_to() versions for vector_Rect and vector_vector_Point2f in cv2.cpp , but as long as the parser generates wrong signatures, they can't do the right thing.

[berak]

finally:

https://github.com/berak/opencv_contrib/blob/b6e631bdc87c40260995080abce5f52cdbd453d9/modules/face/src/facemarkLBF.cpp#L375-L397

and all my tests pass (travis code here)

but it's ugly. fixing the deficiancies of the python bindings from the inside of opencv code -- can't be really the answer. however, it's far less invasive, than changing the interfaces.

[krshrimali]

@berak This is great! Thanks a lot for your help. :+1
Closing the issue, as I believe it has been resolved. Please let me know if it has to remain open, till there is a sweeter solution!

[berak]

@krshrimali usually a pr (making a proper solution available to everyone else) should close this,
please rather leave this open for now (mainly, -- so other folks with the same problem have a better chance getting here).

till there is a sweeter solution!

exactly. it's a duct-tape fix.
it's mending the symptom (segfaults all over), not the problem (python bindings are not smart enough)

[efillman]

@berak I just wanted to say thank you! I had a fresh build of the 4.0.1 release but I still just had to add in your code snippets to facemarkLBF.cpp, facemarkAAM.cpp, and getlandmarks.cpp and recompile. Thanks again.

[berak]

@efillman, yes, sadly the problem is still unsolved. maybe this years gsoc efforts ....

[spmallick]

@berak : Thanks. Let me see if I can ask a student interested in GSOC to pick up this problem.

[tegusi]

Hello! I'm working on this issue now, and I am trying to rewrite the interface as:
bool FacemarkLBFImpl::fit( InputArray image, std::vector<Rect> & roi, CV_OUT std::vector<std::vector<Point2f> > & _landmarks )
by which the python wrapper is able to recognize the vector correctly. However I met some difficulies while test:

terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(4.1.0-pre) /home/tegusi/opencv-master/modules/core/src/copy.cpp:254: error: (-215:Assertion failed) channels() == CV_MAT_CN(dtype) in function 'copyTo'

Wish your help! @berak

[tegusi]

I made it! Since there is no explict convert function from numpy array to vector in cv2.cpp, I add it and everything works! I'll submit a pull request later!

[spmallick]

Very nice. Let's hope @berak likes it :).

[berak]

@tegusi , oh, nice ;)

unfortunately my test script still fails:

https://travis-ci.org/berak/tt/jobs/513404368#L2047-L2055

1.05s$ python landmarks.py
('faces', [(119, 46, 101, 101)])
testing LBF
loading data from : lbfmodel.yaml
Traceback (most recent call last):
  File "landmarks.py", line 14, in <module>
    ok, landmarks = obj.fit(img, faces)
TypeError: Expected cv::UMat for argument 'faces'
The command "python landmarks.py" exited with 1.

also, there's the similar problem with vector<vector<Point2f>> to solve ;)

[tegusi]

@berak Sorry, I just submitted the pr1 for opencv_contrib and pr2 for opencv, which should solve the problem and works fine on your tests. Since I'm not quite familiar with the whole process of pull request and it may take some time to merge into the master branch, you can try the one on my github page.

Thanks a lot for the hints you provided!

[berak]

@tegusi, ok, strike the last comment, and let me try again with both commits ;) (missed the opencv_contrib one)

you can try the one on my github page.

yep, that's what i'm doing now ;) (using both opencv and opencv_contrib from your github repos)

btw, you should make a new branch with a specific name for both changes, the buildbots can't test it properly, if the branch is named master

[berak]

@tegusi -- almost there !

can you check the kazemi version, again ?

[tegusi]

@tegusi -- almost there !

can you check the kazemi version, again ?

Maybe you miss the _face_landmark_model.dat_ download_link? The code has passed in my environment.

Besides it's my fault that I should have made a new branch... If it's possible to change the existing pr?

[berak]

yea, you're right, my kazemi model failed to dl correctly ;) (apologies for the noise)