Detectron2: How to evaluate a model based on input(list[dict]) and output(list[dict])

[Long-Xuan-Do]

Dear all,

Below are my input(list[dict]) and output(list[dict]).
[{'instances': Instances(num_instances=16, image_height=480, image_width=640, fields=[pred_boxes: Boxes(tensor([[1.3369e+02, 2.4598e+02, 4.6721e+02, 4.8000e+02],
[1.1462e+02, 2.6738e+02, 1.4828e+02, 3.9925e+02],
[2.5317e+02, 1.5696e+02, 3.3915e+02, 4.0868e+02],
[4.9911e+01, 2.7487e+02, 8.0228e+01, 3.4673e+02],
[1.0428e-01, 2.7798e+02, 7.5537e+01, 4.7822e+02],
[5.6061e+02, 2.7294e+02, 5.9683e+02, 3.6968e+02],
[3.8512e+02, 2.7218e+02, 4.1292e+02, 3.0423e+02],
[4.0975e+02, 2.7291e+02, 4.5986e+02, 3.5668e+02],
[5.0734e+02, 2.6601e+02, 5.7158e+02, 3.0312e+02],
[3.4307e+02, 2.5171e+02, 4.1340e+02, 2.8463e+02],
[3.5624e+02, 2.6986e+02, 3.8623e+02, 2.9790e+02],
[1.1357e+02, 2.6242e+02, 1.5003e+02, 2.8444e+02],
[5.1642e+02, 2.7962e+02, 5.6163e+02, 3.4186e+02],
[3.3034e+02, 2.3068e+02, 3.9238e+02, 2.5672e+02],
[3.0565e+02, 1.4728e+02, 3.1542e+02, 1.6721e+02],
[5.9443e+02, 2.7041e+02, 6.1337e+02, 3.1423e+02]])), scores: tensor([0.9993, 0.9980, 0.9976, 0.9858, 0.9840, 0.9815, 0.9719, 0.9656, 0.9638,
0.9398, 0.9104, 0.8624, 0.8399, 0.8378, 0.7728, 0.7262]), pred_classes: tensor([17, 0, 0, 0, 0, 0, 0, 0, 25, 25, 0, 25, 0, 25, 29, 0])])}]
[{'image': tensor([[[140., 135., 128., ..., 62., 72., 78.],
[134., 131., 126., ..., 61., 71., 78.],
[125., 124., 122., ..., 59., 70., 78.],
...,
[ 17., 19., 22., ..., 174., 167., 162.],
[ 15., 18., 22., ..., 169., 164., 161.],
[ 14., 17., 22., ..., 166., 162., 160.]],

    [[160., 158., 156.,  ...,  88.,  95., 100.],
     [165., 160., 153.,  ...,  91.,  99., 106.],
     [173., 163., 149.,  ...,  95., 106., 114.],
     ...,
     [ 16.,  18.,  22.,  ..., 168., 160., 154.],
     [ 14.,  17.,  23.,  ..., 161., 155., 150.],
     [ 13.,  17.,  24.,  ..., 156., 151., 147.]],

    [[118., 119., 120.,  ...,  52.,  64.,  72.],
     [121., 119., 116.,  ...,  60.,  74.,  83.],
     [125., 119., 109.,  ...,  71.,  88., 100.],
     ...,
     [ 18.,  20.,  22.,  ..., 191., 183., 177.],
     [ 16.,  19.,  24.,  ..., 184., 178., 173.],
     [ 15.,  19.,  25.,  ..., 179., 174., 171.]]]), 'height': 480, 'width': 640}]

I have tried to implement the DatasetEvaluator and finally the function inference_on_dataset (https://detectron2.readthedocs.io/_modules/detectron2/evaluation/evaluator.html#DatasetEvaluator) but it is unsuccessfully. I don't know how to generate the arguments data_loader and evaluator. Could anyone help me in coding?

Thanks in advanced!

[ppwwyyxx]

I don't know how to generate the arguments data_loader and evaluator.

data_loader is any iterable of inputs, as the documentation https://detectron2.readthedocs.io/modules/evaluation.html#detectron2.evaluation.inference_on_dataset says.

evaluator is something you want to implement.
An example is given in https://detectron2.readthedocs.io/tutorials/evaluation.html

[Long-Xuan-Do]

Dear,

I understand your point but I don't have any COCO-format JSON file, I just have input and output like I mentioned above. So could you tell me how to do evaluation?

Thanks in advance!

[ppwwyyxx]

It does not require a COCO-format JSON file because data_loader can be any iterable of inputs, such as (naively) a list of inputs.

[Long-Xuan-Do]

Dear,

Sorry because I am a beginner. But I don't understand how to implement evaluation. As I checked the source code in this link: https://detectron2.readthedocs.io/_modules/detectron2/evaluation/coco_evaluation.html#COCOEvaluator
I understand that it would automatically convert my input and output in normal standard form but I don't know where to pass my input and output. Could you help me?

Thank you so much!

[ppwwyyxx]

Maybe you're looking for this:

e = COCOEvaluator(...)
e.reset()
e.process(inputs, outputs)
print(e.evaluate())

Or, if you have a model:

print(inference_on_dataset(
    model, dataloader_that_matches_model_input_format, COCOEvaluator(...))

[Long-Xuan-Do]

Dear,

I tried to run,:
e = COCOEvaluator(...)
e.reset()
e.process(inputs, outputs)
print(e.evaluate())

It raised an error:
__init__() missing 2 required positional arguments: 'cfg' and 'distributed'

Thank you so much!

[ppwwyyxx]

"..." is meant to be something to fill out according to the API of COCOEvaluator

[ppwwyyxx]

The input format also requires an "image_id" to make evaluation possible. It seems missing in the input examples in this issue.

[Long-Xuan-Do]

Dear,

Yes, I am quite confused since now I think that I need to write a function that converts a raw image to the standard input. The input I mentioned to you above is I extracted from the run_on_image steps but it seems does not work.

Thank you so much. I would try. Hope that today I can generate the matrix.
Thank you & BR

[Long-Xuan-Do]

So do you have any source code that converts from a raw image to the standard input? Could you provide me the link or st similar? Thank you so much!

[ppwwyyxx]

I'm also confused: if you only have raw image but don't have the data in (or can be converted to) the standard input format, then it is impossible to do any evaluation because there is no ground truth.

[Long-Xuan-Do]

Oh, now I figured out the main problem. So we cannot evaluate based on the images and the pre_trained model we just can when we have the exact results, and based on that results and the output we evaluate. I am so stupid. Thank you so much.
Thank you so much, you are as my mentor, I am just a first year student so a lot of definitions I am not clear. Thank you so much!