Maskrcnn-benchmark: How to draw pr curve and output tested images?

[guanbin1994]

❓ Questions and Help

I can get the AP, but i do not know how to draw pr curve ?

And because the dataset i used to train is medical dataset, so I also want to look the detected test images which can show bbox on images, what should i do to output tested images into a folder.

Thanks!

[fmassa]

I have not yet added functionality to draw PR curves, nor to output the tested images predictions. It is not hard to add, but I didn't have the time yet.

[ChenjieXu]

question Questions and Help

I can get the AP, but i do not know how to draw pr curve ?

And because the dataset i used to train is medical dataset, so I also want to look the detected test images which can show bbox on images, what should i do to output tested images into a folder.

Thanks!

You may refer to maskrcnn_demo.ipynb in demo folder. Here is the code from predictor.py

`def compute_colors_for_labels(labels):
"""
Simple function that adds fixed colors depending on the class
"""
colors = labels[:, None] * torch.tensor([2 * 25 - 1, 2 * 15 - 1, 2 ** 21 - 1])
colors = (colors % 255).numpy().astype("uint8")
return colors

def overlay_boxes(image, predictions):
"""
Adds the predicted boxes on top of the image
Arguments:
image (np.ndarray): an image as returned by OpenCV
predictions (BoxList): the result of the computation by the model.
It should contain the field labels.
"""
labels = predictions.get_field("labels")
boxes = predictions.bbox

    colors = compute_colors_for_labels(labels).tolist()

    for box, color in zip(boxes, colors):
        box = box.to(torch.int64)
        top_left, bottom_right = box[:2].tolist(), box[2:].tolist()
        image = cv2.rectangle(
            image, tuple(top_left), tuple(bottom_right), tuple(color), 3
        )

    return image`

[guanbin1994]

question Questions and Help

I can get the AP, but i do not know how to draw pr curve ?
And because the dataset i used to train is medical dataset, so I also want to look the detected test images which can show bbox on images, what should i do to output tested images into a folder.
Thanks!

You may refer to maskrcnn_demo.ipynb in demo folder. Here is the code from predictor.py

`def compute_colors_for_labels(labels):
"""
Simple function that adds fixed colors depending on the class
"""
colors = labels[:, None] * torch.tensor([2 * 25 - 1, 2 * 15 - 1, 2 ** 21 - 1])
colors = (colors % 255).numpy().astype("uint8")
return colors

def overlay_boxes(image, predictions):
"""
Adds the predicted boxes on top of the image
Arguments:
image (np.ndarray): an image as returned by OpenCV
predictions (BoxList): the result of the computation by the model.
It should contain the field labels.
"""
labels = predictions.get_field("labels")
boxes = predictions.bbox

    colors = compute_colors_for_labels(labels).tolist()

    for box, color in zip(boxes, colors):
        box = box.to(torch.int64)
        top_left, bottom_right = box[:2].tolist(), box[2:].tolist()
        image = cv2.rectangle(
            image, tuple(top_left), tuple(bottom_right), tuple(color), 3
        )

    return image`

hello, my problem i s how to draw the precision-recall curve

[fmassa]

@guanbin1994 didn't https://github.com/facebookresearch/maskrcnn-benchmark/issues/353 address your question?

[milongo]

@fmassa issue I have a similar question. From issue #353 I can get the coco_eval object that has the precision and recall values. However, the shape of these two arrays is different, and it is not clear how to use them to plot a PR curve. I can see the arrays as being of shape T, R, K, A, M and T, K, A, M for precision and recall, respectively. So I have 101 precision values at each IoU threshold but only one recall value.
How can I plot the PR curve from these?

[ayman-saleh]

pycocotools gives you the eval at IoU thresholds staring at 0.5 up to 0.95 with step sizes of 0.05. The dimensions of the precision array correspond to the following:

• T: len([.5:.05:1.0]) the 10 IoU Thresholds
• R: len([0:0.01:1]) the 100 Recall Thresholds
• K: The number of category ids you have
• A: A=4 which corresponds to the object area ranges
• M: max detections

So, if you wanted to plot the curves, you could write something like this:

import numpy as np
import matplotlib.pyplot as plt 

all_precision = coco_eval.eval['precision']

pr_5 = all_precision[0, :, 0, 0, 2] # data for [email protected]
pr_7 = all_precision[4, :, 0, 0, 2] # data for [email protected]
pr_9 = all_precision[8, :, 0, 0, 2] # data for [email protected]

x = np.arange(0, 1.01, 0.01)
plt.plot(x, pr_5, label='[email protected]')
plt.plot(x, pr_7, label='[email protected]')
plt.plot(x, pr_9, label='[email protected]')

plt.show()

[isalirezag]

@ayman-saleh Thank you for your answer!
Just for my understanding, is M the confidence thresholds that is used? in other words, does it means that when M=100 we just consider the objects with confidence score of 100?
Hmmm is it possible to compute the mAP for a certain threshold?

[ayman-saleh]

@isalirezag M is a list, so by default, i believe it is M = [1, 10, 100]. This list corresponds to the thresholds on max detections per image.

https://github.com/cocodataset/cocoapi/blob/636becdc73d54283b3aac6d4ec363cffbb6f9b20/PythonAPI/pycocotools/cocoeval.py#L28

[xabirizar9]

@ayman-saleh sorry that I got here so late, but how would you go about plotting the precision against the recall? Here you are just plotting the precision against an array from 0 to 1.01, but to my knowledge, recall also has different values, which would make the plot different. Following your code, I plotted this in my program and this came out, which looks a bit odd to me compared to the typical precision vs recall curves.

[jpainam]

Hi @ayman-saleh how about the recall array?, the dimensions of precision array is 5 while the dimensions of the recall array is 4. ie. [TxKxAxM]. How to extract the recall in a shape that can be plot.
Thank you.

[Eldad27]

@ayman-saleh The code you provided computes precision. Please how about the recall points?

1. how to compute it? I tried to use your suggested code to generate PR curve on custom COCO dataset and had two extremely odd curves for two different classes (Please see below). The dimension of precision returned a shape of [10, 101, 2, 4, 3] against a recall dimension of [10, 2, 4, 3]. how does a 5 dimension shape plot against a 4 dimension shape? Could this be the reason for the Odd curves?
   2.
   

?

Btw, The array of precision returns values full of ones. something probably associated with my dataset. This seem a strange to me. Have you by any chance come across such behaviour?