Darknet: Problems about tracking pedestrians using optical flow

[RoseOnCollar]

@AlexeyAB l'm sorry to disturb you. i have tried to track pedestrians with optical flow that you suggest. It can not work very well. the problems are:
1.When occlusion occurd, such as two people walk towards each other and then separate, the optical flow will probably swap the labels of them.
2.YOLO2 seems can not detect two people exactly while occlusion occurd.

How about using color histogram to track pedestrians? the color of their clothes is different in most situations.

[AlexeyAB]

@RoseOnCollar Hi, yes, you can try to use mean shift with histogram.

There are at least 4 ways to do tracking:

1. Optical flow: gpu::PyrLKOpticalFlow::sparse()
2. Mean shift with Histograms: meanShift()
3. Phase correlation: phaseCorrelate()
4. Neural network LSTM Tracking: https://github.com/pjreddie/darknet/pull/64

To solve occlusion problem, you can get objects coords when they are detected by Yolo and store these coords-N and this frame-N. If in the next detection frame-M some objects are absence then try to track absence objects from (coords-N on frame-N) to (frame-M). If probability (error value of optical flow) is low, then try to track from (coords-N on frame-N) to next detection (frame-K). And so on
... If the object is not detected within next 1-10 frames, then delete it.

[RoseOnCollar]

@AlexeyAB ,Sorry, I think, my main problem is the matching of the same objects detected between two adjacent frames(N and N+1).
Do you have any good suggestions?
Thank you( ｰ̀εｰ́ )

[RoseOnCollar]

@AlexeyAB Is there a real-time pedestrian detection and tracking system using depth learning and running on CPU?

[AlexeyAB]

@RoseOnCollar I think that it can't be implement on CPU with a good result and solved occlusions.

But it can be implemented on K40 GPU with neural network layer LSTM: https://arxiv.org/abs/1506.04878v3

Video: https://www.youtube.com/watch?v=QeWl0h3kQ24

End-to-end people detection in crowded scenes: https://arxiv.org/pdf/1506.04878v3.pdf

Current people detectors operate either by scanning an image in a sliding win- dow fashion or by classifying a discrete set of proposals. We propose a modelthat is based ondecoding an image into a set of people detections. Our system takes an image as input and directly outputs a set of distinct detection hypotheses. Because we generate predictions jointly, common post-processing steps such as non-maximum suppression are unnecessary. We use a recurrent LSTM layer for sequence generation and train our model end-to-end with a new loss function thatoperates on sets of detections. We demonstrate the effectiveness of our approachon the challenging task of detecting people in crowded scenes.

[RoseOnCollar]

thankyou, please don't close this, I will try optical folw again.

[RoseOnCollar]

@AlexeyAB while I use optical flow for tracking people, I can't get the satisfactory result.
I have tried these parameters:

1.TermCriteria termcrit(CV_TERMCRIT_ITER | CV_TERMCRIT_EPS, 30, 0.01);
The maxcount is 20/30/100, and the epsilon is 0.01/0.001. This parameter has little effects.

2.Size winSize(15, 15);
The winsize is 15X15 / 31X31 / 101X101;

3.Then calcOpticalFlowPyrLK(prevGray, nextGray, point1, point2, status, err, winSize, 3, termcrit, 0);

Can you give me some suggestions? I have put the results on http://pan.baidu.com/s/1geVxSYn
Thank you!

[AlexeyAB]

@RoseOnCollar

1. I use GPU, and set from iters = 100 to 2000. So on CPU: TermCriteria termcrit(CV_TERMCRIT_ITER | CV_TERMCRIT_EPS, 100, 0.01);

2. Size winSize(15, 15); - The larger the object, the larger the window should be. But the window should not capture too much background - otherwise the background will be tracked instead of the object.

3. Also set from maxLevel = 5; to maxLevel = 8; - The further the object moved, the more this value should be

4. Also you can use Kalman Filter to generate initial estimations of object movement : http://docs.opencv.org/2.4/modules/video/doc/motion_analysis_and_object_tracking.html#KalmanFilter
   And pass it to cv::calcOpticalFlowPyrLK using nextPts and set flag OPTFLOW_USE_INITIAL_FLOW

    int stateSize = 10;
    int measSize = 10;
    int contrSize = 0;
    cv::KalmanFilter kf(stateSize, measSize, contrSize, CV_32F);

---

What I used: http://docs.opencv.org/2.4/modules/gpu/doc/video.html#gpu::PyrLKOpticalFlow

gpu::PyrLKOpticalFlow PyrLKOpticalFlow_gpu;
PyrLKOpticalFlow_gpu.maxLevel = 5;
PyrLKOpticalFlow_gpu.iters = 100;
PyrLKOpticalFlow_gpu.getMinEigenVals = true;

PyrLKOpticalFlow_gpu.sparse(prev_frame_gpu, cur_frame_gpu, prev_pts_flow_gpu, cur_pts_flow_gpu, status_gpu, &err_gpu);

err_gpu.download(err_cpu); // and control err_gpu
if(err_cpu.at<float>(0, i) < 6) // then track this object

[RoseOnCollar]

@AlexeyAB
Thank you for the reply.
I have tried the optical flow and the result is still unsatisfactory. Maybe something is wrong in my algorithm.
Could you please help me to think about one thing: While we want to match the same persons detected by YOLO between two frames(N and N+n), the problem maybe not person tracking but person re-identification, isn't it? Could you give me some suggestions?

[pribadihcr]

@RoseOnCollar,

You can see this as reference, it use tracking by person re-id https://github.com/nwojke/deep_sort

[RoseOnCollar]

@pribadihcr ,
Thank you, while i read the code of deep_sort, it use KF to update the tracker. I can't find the code of person re-id section.