Dlib: Using dlib with opencv to make "real time" face recognition: conflict between the two librairies?

[Toquet]

Hello,
I am currently working on a project (in c++) that aims at doing face recognition on images coming from a webcam. In order to do so, I use the opencv library which seems to cause issues during the dlib face encoding part:

Expected Behavior

The algorithm is as follow:

• Detect a face thanks to the dlib face_detector.
• Crop the image thanks to the shape predictor (shape_predictor_5_face_landmarks.dat).
• Use dlib_face_recognition_resnet_model_v1.dat to transform this image into an array.
• Compare this array to ones obtained with known faces in order to find the identity of the person detected.

As mentioned I need to store the arrays of the known persons before doing the recognition with the webcam. This is where strange things happen:

Current Behavior

• When I do not open the camera by calling cv::VideoCapture(), everything is working (known faces get always the same array, test pictures are well recognized).
• As soon as I declare the cv::VideoCapture() variable, it does not work anymore: known faces get always different arrays and the recognition does not work anymore on the test pictures.
  I checked everything and I figured out that in both cases the same picture was send to the neural network dlib_face_recognition_resnet_model_v1.dat, but different results are given. Is it possible that the face encoder is not compatible with opencv (especially the cv::VideoCapture function) ?

It may not be really clear with text so I hope the following code will be more explicit:

Steps to Reproduce

• Main, Create a recognizer class that I defined and then call the function that should save the arrays of the known faces:

  Recognizer *m_recognizer = RecognizerFactory::Get()->CreateRecognizer("facenet");
  m_recognizer->training_function("../modified");
  

Then as soon as I add the following line, the result of the encoding seems to be random:
cv::VideoCapture video(0);

• training_function (Loop on images present in a folder):

  ``` DIR* dirp;
  struct dirent * dp;
  // Trying to open the folder.
  if((dirp = opendir(name.c_str())) == NULL) {
  std::cout << "Error opening " << name << endl;
  return;
  }
  // Loop on the files that are in the folder.
  while ((dp = readdir(dirp)) != NULL) {
  // Do not take into account files that are smaller than 3 characters.
  if(std::string(dp->d_name).size() > 3) {
  // Load the image.
  dlib::matrix image_dlib;
  dlib::load_image(image_dlib, name + std::string("/") + std::string(dp->d_name));
  // Save the embedding.
  ROI location_face(0,0,image_dlib.nr(),image_dlib.nc());
  this->known_embeddings->push_back(face_encoding(image_dlib,location_face));
  cout << this->known_embeddings->at(this->known_embeddings->size()-1)(0) << endl;
  this->known_names->push_back(std::string(dp->d_name).substr(0,std::string(dp->d_name).find(".")));
  }
  }
  closedir(dirp);

• face_encoding function:
  ``` anet_type face_encoder;
  deserialize("dlib_face_recognition_resnet_model_v1.dat") >> face_encoder;
  std::vector> faces;
  // Landmarks.
  frontal_face_detector detector = get_frontal_face_detector();
  for (auto face : detector(img)) {
  auto shape = this->sp(img, face);
  matrix face_chip;
  extract_image_chip(img, get_face_chip_details(shape,150,0.1), face_chip);
  faces.push_back(move(face_chip));
  }
  // Embedding.
  std::vector> face_descriptors = face_encoder(faces);
  // Display (Check if the images are the same everytime I launch the program)
  dlib::image_window win(faces.at(0));
  win.wait_until_closed();
  I just have to print the first value of face_descriptors.at(0) to see that it is always the same for the same image when cv::VideoCapture video(0); is not present in the main function. Otherwise it seems random... I hope I made my issue understable !

  • Version: dlib: 19.9 or 19.10, opencv: 3.3.1 or 3.4.0
  • Where did you get dlib: Github repo
  • Platform: Ubuntu 16.04 LTS
  • Compiler: gcc 5.4.0

[davisking]

These things (the dlib stuff and openCV tools) have nothing to do with each other. You just have a bug in your code somewhere and it's corrupting the outputs. Maybe you have some kind of memory error. The code you posted has a bunch of raw pointers so that's not helping you. C++ isn't C. You shouldn't have raw pointers or be doing manual memory management.

[Toquet]

I thought it was a memory leak also. However I found an interesting thing:
when I run your example program dnn_face_recognition_ex.cpp, everything works fine, the outputs are always the same for the same inputs. However, when I include opencv and I declare a cv::VideoCapture in it, the outputs become random:

cv::VideoCapture video(0);
video.release();

I really believe that there is something strange between the two librairies but I can't find what.

[davisking]

Maybe there is a bug in opencv or you compiled it in some bad way. I don't
know. There are no interactions between dlib and opencv though.

[Toquet]

Ok thank you for the answers !

[Toquet]

Just to let you know the end of the story:
The issue was caused by the fact that my python libraries were interacting with the c++ ones. I just uninstalled them and reinstalled only the c++ lib and it solved the issue.
So you were right! Thank you.