Darknet: incorrect calculation of mAP for -points 0 (for own dataset)

[adityap27]

@AlexeyAB

• The code for calculation of Area-Under-Curve on PR-chart.(for -point 0) seems wrong.

• command:
  ./darknet detector map "data/obj.data" "cfg/yolov3.cfg" "backup/yolov3_last.weights" -dont_show -thresh 0.01 -points 0

Screenshot of code in detector.c

Problem: the code is not calculating the area of section between recall=0 and 1st recall value(from left on graph). i.e delta recall is only calculated uptill top 2 ranks.......not between the 1st rank and recall=0.

This problem occurs only if the recall at rank 1 is non-zero........in that case we need to calculate another delta_recall which isn't calculated.

output

[adityap27]

Hey @AlexeyAB I have solved this incorrect calculation by changing the detector.c file.

May I create a pull request so that map is calculated accurately ?? (I have noticed that old map calculation wasn't correct for less images so...I solved it for less images too.)

In next comment I have added comparison of mAP for old_detector.c vs new detector.c

[adityap27]

OLD CODE

NEW CODE

My Actual image that is used for mAP calculation.

We can clearly see that the aP for class mask should be 100% aP .....but the old code was giving 75% aP due to the reason I stated above.(less images)

[AlexeyAB]

• Do you mean that you get Area under PR-curve = 1 for all confidence thresh >= 0.005 ? https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L951

• Add between these lines: https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L1255-L1256
  this line
  printf("class_id = %d, point = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, point, last_recall, last_precision);
  and show me screenshot

• Yes, create a Pull request with minimal changes and refer to this issue.

[adityap27]

For 1st question,

Yes, according to my test image, I should get area under PR-curve=1 for all conf thresh >=0.005, but due the old detector.c code it gave me incorrect aP(0.75) and mAP.

[adityap27]

• Add between these lines: https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L1255-L1256

  this line
  printf("class_id = %d, point = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, point, last_recall, last_precision);
  and show me screenshot

hey @AlexeyAB ,

Adding this line gives me error, as point variable is not accessible in that scope....as it is not there for map calculation for custom dataset.

[AlexeyAB]

Try this line
printf("class_id = %d, rank = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, rank, last_recall, last_precision);

[adityap27]

printf("class_id = %d, rank = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, rank, last_recall, last_precision);

Hey @AlexeyAB , Tried this new line....checkout the screenshot below

[AlexeyAB]

Try to add this line between these two lines: https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L1254-L1255

[adityap27]

Try to add this line between these two lines:

https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L1254-L1255

Hey @AlexeyAB , check it out

note: this output is using my new detector.c ..........The old one would have given me 75%aP for class_id=1

[AlexeyAB]

Thanks!
Also add this line between these 2 lines: https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L1244-L1245

Do you get cur_precision=1.0 for rank=3?

[adityap27]

Thanks!
Also add this line between these 2 lines:

https://github.com/AlexeyAB/darknet/blob/00919eeb99f73484cf9f51b277a4e4ebacef70df/src/detector.c#L1244-L1245

Do you get cur_precision=1.0 for rank=3?

@AlexeyAB see now:

[AlexeyAB]

@adityap27 Thanks!

[adityap27]

@adityap27 Thanks!

Thank you @AlexeyAB for your support. :)