Detectron2: Evaluate existing coco model on own dataset

[yklose]

I want to evaluate a custom dataset with an existing coco model for reference. I followed the Colab Notebook for general prediction after training, but I always get AP of 0.00 even though I visualized the output and it does a pretty good job

Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

OrderedDict([('bbox',
              {'AP': 0.0,
               'AP50': 0.0,
               'AP75': 0.0,
               'APs': 0.0,
               'APm': -100.0,
               'APl': -100.0,
               'AP-1': 0.0,
               'AP-2': 0.0,
               'AP-3': nan,
               'AP-4': 0.0,
               'AP-5': nan,
               'AP-6': nan,
               'AP-7': nan,
                   ...
               'AP-75': nan,
               'AP-76': nan,
               'AP-77': nan,
               'AP-78': nan,
               'AP-79': nan,
               'AP-80': nan})])

My code is as follows. The dataset has, like the coco dataset 80 classes. It however detects the 3 classes in the small test dataset, because AP-1, AP-2, AP-4 have a score of 0.0 while the other classes are nan.

model = build_model(cfg)
DetectionCheckpointer(model).load(weights_path)

evaluator = COCOEvaluator("testset_val", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "testset_val")
inference_on_dataset(model, val_loader, evaluator)

Any suggestions how to evaluate the right way?

[ppwwyyxx]

The category ids in your dataset has to have the same meaning as the one in COCO for a COCO model to work.

[yklose]

Thank you for your answer, even though my category id was right when register the dataset, it was different from the one in my ground-truth. Now its working!

[Chi-Zaozao]

@yklose
I have the same problem, could you tell me how to solve this problem in detail?

[northeastsquare]

I also met this problem, can you share some hits?
@yklose

[northeastsquare]

OK,I change the according line as follows, evalutation not 0 or nan now:
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
trainer.resume_or_load(resume=True)

[dhaivat666]

I had the exact same problem. Whenever you run evaluation, I recommend you to delete contents of ./output/ directory. For me, it turned out that the new json file was not being created, causing it to fail. If you make any changes to the code, it's a good idea to delete ./output/ directory to ensure that you are saving correct files