Darknet: mAP and IOU calculations by the last commit (8474f49 )

[VanitarNordic]

Hi,

I tried to calculate the mAP for my own dataset. It delivers 100% for the mAP and 81.5% for the average IoU.

The command that I used was this:
darknet.exe detector map data/obj.data yolo-obj.cfg backup/yolo-obj_final.weights

I think the mAP is correct since it detects all validation sets correctly (I don't know in how much shift with annotation labels) but for the average IoU, does this number should be equal to the number which we get by running a command like this? because I get 83.56% in the last row by running the validation.

darknet.exe detector recall data/obj.data yolo-obj.cfg backup/yolo-obj_final.weights -i 0

[AlexeyAB]

Hi,

• In my code using detector map the average IoU is calculated correctly for default thesh = 0.24.

• Using detector recall the IoU is calculated for thresh = 0.001 and then only the best IoUs are taken without comparing the class id - i.e. network can detect apples when on images are only oranges - and IoU will be high - so this is purely an internal indicator for developers, Joseph never wrote that you should use it, I wrote about it only until I added a mAP.
  Code of detector recall:
  https://github.com/AlexeyAB/darknet/blob/8474f491f2be64222fb1662b088d87529a22670f/src/detector.c#L455-L468

[AlexeyAB]

I tried to calculate the mAP for my own dataset. It delivers 100% for the mAP and 81.5% for the average IoU.

• Also how many TP, FP, FN did you get?
• And how many AP for each class?

[VanitarNordic]

There is one class, there you go:

[AlexeyAB]

mAP is calculated from PR-curve that is calculated from (TP, FP, FN), where TP (true-positive) += 1 if current IoU > 0.5, so mAP can be 100% even if average IoU < 100%.

[VanitarNordic]

in the statistics, it shows one False positive. so by this, the effect is on the IoU since mAP is 100%

[VanitarNordic]

I have one question about small_object=1 effect also.

By defining this value, the images in the training dataset will be randomly scaled, isn't it?
if Yes, may I ask you to check it again because I couldn't detect any small object until I added 3-4 images from some small target objects into the training dataset. Therefore the scaling might not work.

[AlexeyAB]

in the statistics, it shows one False positive. so by this, the effect is on the IoU since mAP is 100%

You can un-comment this line, and re run mAP calculation, it will give you more info about 11 points on PR-curve: https://github.com/AlexeyAB/darknet/blob/4a7a076d0939a491af8fe3fd63e5012ffd5afebc/src/detector.c#L775

[AlexeyAB]

I have one question about small_object=1 effect also.

By defining this value, the images in the training dataset will be randomly scaled, isn't it?
if Yes, may I ask you to check it again because I couldn't detect any small object until I added 3-4 images from some small target objects into the training dataset. Therefore the scaling might not work.

No, small_object=1 isn't about scaling.

There was a problem during training, if size of ground truth of object less than 0.1-1% of image width or height - it leads to Nan. So these small ground truth were rejected during training and this led to the fact that the model was trained not to detect small objects: https://github.com/pjreddie/darknet/blob/80d9bec20f0a44ab07616215c6eadb2d633492fe/src/data.c#L435

The problem is in small truth boxes, but not in small objects.

So now, these small objects aren't rejected, but just bounded box size increased up to 1% x 1% of image size: https://github.com/AlexeyAB/darknet/blob/4a7a076d0939a491af8fe3fd63e5012ffd5afebc/src/data.c#L309-L314
Now small objects can be detected (only box will be not less than 1% x 1%)

[VanitarNordic]

Therefore scaling is not performed in the augmentation phase as rotation. if we do this (random scaling), it might increase the performance, yes?

[AlexeyAB]

Scaling performed in the augmentation phase if random=1. It increases precision.

[VanitarNordic]

As I realized, random=1 also upscale the images not just downscale them isn't it?

second, what is the effect of these values?

object_scale=5
noobject_scale=1
class_scale=1
coord_scale=1 

[VanitarNordic]

Assume the original image is this:

Do you like resizing and integration in the new image to feed the model with its minimum possible size also?

[AlexeyAB]

As I realized, random=1 also upscale the images not just downscale them isn't it?

Yes.

Do you like resizing and integration in the new image to feed the model with its minimum possible size also?

Try it. But it is better if the object is inserted without a superfluous background (without a table).

So if you insert an object with a background (table) into other images (forest+man), then the network can learn to search for an object only with this background (table) - this is overfitting. I.e. the probability of detection will be shifted to the case when apple will be detected only if around the apple there is piece of table.

[VanitarNordic]

you mean it is better to crop the bbox and then mix it with a new background?

if I crop it, still small parts of the table is around the apple in the bbox.

[AlexeyAB]

No, I mean it is better to crop an object by borders of object.

[VanitarNordic]

By borders of the object? I don't get. May I ask you to mention an example or picture?

[sivagnanamn]

@AlexeyAB means something similar to the idea mentioned in the paper below:

Use some kind of a slicing tool to select only the object of interest (without any background involved) & emboss it on different backgrounds.

This makes the NN to search for the objects without over-fitting to the background.

https://arxiv.org/pdf/1709.00849.pdf

[sivagnanamn]

@VanitarNordic Adding one more paper for your reference:

Synthesizing Training Data for Object Detection in Indoor Scenes