Pytorch-yolov3: True positive count may be wrong

[youzunzhi]

In get_batch_statistics(), it counts true_positives like this:

for pred_i, (pred_box, pred_label) in enumerate(zip(pred_boxes, pred_labels)):

 # If targets are found break
    if len(detected_boxes) == len(annotations):
        break

    # Ignore if label is not one of the target labels
    if pred_label not in target_labels:
        continue

    iou, box_index = bbox_iou(pred_box.unsqueeze(0), target_boxes).max(0)
    if iou >= iou_threshold and box_index not in detected_boxes:
        true_positives[pred_i] = 1
        detected_boxes += [box_index]

So it is possible that target_labels is [1, 2], and the iou of pred_boxes[0] and target_boxes[0] is over iou_threshold while pred_labels[0] is 2 (which should be 1 to make it a TP), but still here true_positives[0] will be 1.

I suggest you modify the third "if" to
if iou >= iou_thresh and box_index not in detected_boxes and pred_label == target_labels[box_index]:

[peaceiris]

Here's the link to the code.
https://github.com/eriklindernoren/PyTorch-YOLOv3/blob/1c5a58f47c9904a9fe9903b46c73bf93a230d8e7/utils/utils.py#L166

And, @youzunzhi 's suggestion is the following.

            for pred_i, (pred_box, pred_label) in enumerate(zip(pred_boxes, pred_labels)):

                # If targets are found break
                if len(detected_boxes) == len(annotations):
                    break

                # Ignore if label is not one of the target labels
                if pred_label not in target_labels:
                    continue

                iou, box_index = bbox_iou(pred_box.unsqueeze(0), target_boxes).max(0)
-                if iou >= iou_threshold and box_index not in detected_boxes:
+                if iou >= iou_thresh and box_index not in detected_boxes and pred_label == target_labels[box_index]:
                    true_positives[pred_i] = 1
                    detected_boxes += [box_index]

[H-YunHui]

@youzunzhi
@peaceiris
How much does this change affect the mAP?

[ujsyehao]

@H-YunHui It is hard to say.

[ujsyehao]

@youzunzhi @peaceiris
Another problem with calculating TP,
iou, box_index = bbox_iou(pred_box.unsqueeze(0), target_boxes).max(0)

The code just calculates max matched IOU, However, predict box may have the IOU(> iou_thresh) with another gt box, and also predict right.

[pandalgx]

@ujsyehao Hello, have you solved this problem? I have another question. For one ground-truth, we have found a matched predict box. In the next iteration, another predict box has a higher IOU than the already matched box. However, as this ground-truth already finds its matched box, we will ignore the second box, as in the code:
if iou >= iou_threshold and box_index not in detected_boxes:

In summary, for one ground-truth, the pred_box having the highest IOU is regarded as TP. So, I think we should iterate the target_boxes instead of pred_boxes.

[yinghuang]

@youzunzhi @peaceiris
Another problem with calculating TP,
iou, box_index = bbox_iou(pred_box.unsqueeze(0), target_boxes).max(0)

The code just calculates max matched IOU, However, predict box may have the IOU(> iou_thresh) with another gt box, and also predict right.

here's my solution.
`

             ...
            # Ignore if label is not one of the target labels
            if pred_label not in target_labels:
                continue    

            iou = bbox_iou(pred_box.unsqueeze(0), target_boxes)
        mask_matched = (target_labels == pred_label) & (iou >= iou_threshold)

        iou_matched = torch.where(mask_matched, iou, torch.zeros_like(iou))
        iou_max, box_index = iou_matched.max(0)

        if iou_max >= iou_threshold and box_index not in detected_boxes:
            true_positives[pred_i] = 1
            detected_boxes += [box_index]

`

[zxj11838]

I have another question. Suppose there are two target_box, when pred_box[0] matches target_box[0]、target_box[1], will a pred_box match two target box?

[youzunzhi]

I have another question. Suppose there are two target_box, when pred_box[0] matches target_box[0]、target_box[1], will a pred_box match two target box?

@zxj11838 In this implementation, pred_box[0] will be added to detected_boxes and it won't be count as another true positive any more.

[Romaosir]

i think this is correct

            iou_sort, box_index_sort = bbox_iou(pred_box.unsqueeze(0), target_boxes).sort(descending=True) 
            for iou,box_index in zip(iou_sort,box_index_sort):
                if iou >= iou_threshold and box_index not in detected_boxes and pred_label == target_labels[box_index]:
                    true_positives[pred_i] = 1
                    detected_boxes += [box_index]