Darknet: how to track objects using darknet

[anguoyang]

darknet has good performance on object detection, is it possible to track objects, e.g, with IDs for different objects in a video? thank you

[AlexeyAB]

@anguoyang

No, Yolo can't do this. There is ROLO, but it is only single object tracker and very slow: https://github.com/Guanghan/ROLO

But for tracking many objects you can use optical flow class PyrLKOpticalFlow from OpenCV : http://docs.opencv.org/2.4/modules/gpu/doc/video.html#gpu::PyrLKOpticalFlow

In loop:

1. Find absolute coordinates of centers of objects by using Yolo.
2. Initialize prev_pts_flow_gpu by cv::Point2fs - one point2f for each object found by Yolo.
3. Perform the function sync_PyrLKOpticalFlow_gpu.sparse();
4. If cv::Point2f from cur_pts_flow_gpu is equal (+-delta) to found by Yolo then this is the same object.

You can tracking something like this in the loop for each new frame.

cv::gpu::PyrLKOpticalFlow sync_PyrLKOpticalFlow_gpu;
cv::Mat err_cpu, tmp_pts_flow_cpu, prev_pts_flow_cpu, cur_pts_flow_cpu;
cv::gpu::GpuMat prev_frame_gpu, cur_frame_gpu;
cv::gpu::GpuMat prev_pts_flow_gpu, cur_pts_flow_gpu, status_gpu, err_gpu;

// get new frame
while(capture >> cur_frame_gpu)
{
// 1. here - find `cv::Point2f`s for objects found by Yolo
// 2. here - compare `cv::Point2f`s from (1) and compare with `prev_pts_flow_cpu` +-delta
// 3. here - set `prev_pts_flow_cpu` to the found `cv::Point2f`s from (1)
// Do following tracking...

  sync_PyrLKOpticalFlow_gpu.sparse(prev_frame_gpu, cur_frame_gpu, 
      prev_pts_flow_gpu, cur_pts_flow_gpu, status_gpu, &err_gpu);

  err_gpu.download(err_cpu);
  cur_pts_flow_gpu.download(cur_pts_flow_cpu);

  for (size_t i = 0; i < cur_pts_flow_cpu.cols; ++i) 
  {
    Point2f key_pt = cur_pts_flow_cpu.at<cv::Point2f>(0, i);
    if (err_cpu.at<float>(0, i) < 25)
        tmp_pts_flow_cpu.push_back(key_pt);
  }

  if (tmp_pts_flow_cpu.rows == 0) prev_pts_flow_cpu.release();
  else cv::transpose(tmp_pts_flow_cpu, prev_pts_flow_cpu);

  prev_frame_gpu = cur_frame_gpu.clone();
}

Also you can do it async in different threads: (A) find object by Yolo, (B) track object by optical-flow-PyrLK

[anguoyang]

Hi, @AlexeyAB, thank you so much for your kind help, I will try on it.

[AlexeyAB]

https://github.com/AlexeyAB/darknet/issues/49

[arvids-p]

But PyrLK is in C++. How do we add it to darknet?

[AlexeyAB]

@arvids-p On Windows it can be implemented in C++ DLL: https://github.com/AlexeyAB/darknet#how-to-use-yolo-as-dll

May be latter I'll add it: https://github.com/AlexeyAB/darknet/issues/85

[arvids-p]

Yes, as i understand, you work only on Windows. Any chance that you might look into the Linux branch? :)

[AlexeyAB]

@arvids-p Not soon )
I mostly use the modified Linux version, but that code is private and I can not show it )

[AlexeyAB]

@arvids-p Now this fork supports Linux.

This repository supports:

• both Windows and Linux
• both OpenCV 3.x and OpenCV 2.4.13
• both cuDNN 5 and cuDNN 6
• CUDA >= 7.5
• also create SO-library on Linux and DLL-library on Windows

How to compile on Linux: https://github.com/AlexeyAB/darknet#how-to-compile-on-linux

[Rahul-Venugopal]

@AlexeyAB @arvids-p @anguoyang
Now is there any solution for tracking multiple objects with YOLO ? Please let me know if i can refer any projects or resources where I can get a detailed picture of how to track multiple objects with YOLO ?

[buzdarbalooch]

@AlexeyAB same question as @Rahul-Venugopal .

[buzdarbalooch]

it would be great help, If YOLO object detection can be linked with tracking multiple objects.
There is object tracking https://www.pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/.
But it would be great how to exploit YOLO for that purpose.