Darknet: calculating map

[Jin-S-Kim]

I train a new object(custom data) by using yolo_mark and dartknet and get new weights. And I want to see the result of training so I use './darknet detector map ~data ~cfg ~weights'.

If set repository of valid to images that used for training(these image files have txt files).

It calculate map.
And If I set repository of valid to images that for test(these images files have no txt files) it cannot calculate map and this is the result

So I make txt file of test images using new weights(that I get for training by training images) and put txt files in directory where test images is in.

And it calculate map.

Does the map have to be a class number of the object and a txt file containing information from the bounding box? Then, if I make a map by making a txt file of test images with the weight I trained, the detection result will be a ground trough. Shouldn't the map be 100 percent?
How can this command calculate map?

[AlexeyAB]

And If I set repository of valid to images that for test(these images files have no txt files) it cannot calculate map and this is the result

Both Training, Validation and Test datasets must have txt-labels files with ground-truth boxes.

---

So I make txt file of test images using new weights(that I get for training by training images) and put txt files in directory where test images is in.

Did you do these files by using pseudo-labeling using -save_labels flag?
./darknet detector test data/obj.data yolov3_obj.cfg backup/yolov3_obj_10000.weights -save_labels < valid.txt

This is the wrong approach.

---

Shouldn't the map be 100 percent?

No, it shouldn't. Because your labels are generated for the default Confidence-threshold -thresh 0.25,
but mAP is calculated for all possible Confidence-thresholds

[Sudhakar17]

@AlexeyAB
I checked the function validate_detector_map. There are two variables thresh_calc_avg_iou = 0.25 and iou_thresh = 0.5. I understand the second one which is compared with cur_iou of the detection. I don't understand the first one which is compared with prob. Can you please tell me why it is like that?

[AlexeyAB]

@Sudhakar17

thresh_calc_avg_iou = 0.25 - is a confidence-threshold (for probability), it is used only for output F1, FN/FP/TP, Precision, Recall, IoU for the specified confidence threshold

iou_thresh = 0.5 is a IoU-threshold, that is used for mAP calculation, it calculates [email protected] by default (iou_thresh = 0.5)

[Sudhakar17]

@AlexeyAB
Thanks a lot. I have one more question related to this one. I used test_detector to save the bbox co-ordinates, class prediction probability (-ext_output) in the text file. Then I used some python scripts to calculate mAP from the predicted results.

I am getting 54.37 % mAP for COCO-Val set.(C implementation) and 45.74% mAP for the same validation set (Python implementaion)

you calculated the results from validate_detector_map and test_detector(inference) functions are the same, right?

[AlexeyAB]

@Sudhakar17

You should run detector test command with flag -thresh 0.001 as it is done for mAP-calculation by default: https://github.com/AlexeyAB/darknet/blob/381f90ebb8d1518141e809521fe2985a2d506fc1/src/detector.c#L693

There is solved this issue: https://github.com/AlexeyAB/darknet/issues/2140#issuecomment-451721618

After including these detections and combining with the changes you suggested, I was able to obtain a mAP score from pycocotools that is very close to calling ./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.50.

---

So there are 3 thresholds in the ./darknet detector map:

• thresh_calc_avg_iou = 0.25 - confidence-threshold is used only for: F1, FN/FP/TP, Precision, Recall, IoU

• iou_thresh = 0.5 - IoU-threshold for mAP calculation ([email protected] by default)

• thresh = 0.005 - confidence-threshold for mAP to remove the most badly detections

---

Also you can calculate mAP on MS COCO test-dev evaluation server: https://github.com/AlexeyAB/darknet/issues/2145#issuecomment-451583623