Detectron2: How to get the evaluations

[ccssmania]

❓

Hello,
I already trained my own dataset that is in coco format, but it has different classes.

If I understand correctly I need to set MetadataCatalog.get(dataset_name).evaluator_type

and I try to run the following code to get the evaluations

MetadataCatalog.get("dataset_name").evaluator_type="coco"
cfg.DATASETS.TEST = ("dataset_name",)  
cfg.MODEL.WEIGHTS = os.path.join("output/model_final.pth")
model = DefaultTrainer.build_model(cfg)
DetectionCheckpointer(model, save_dir=cfg.OUTPUT_DIR).resume_or_load(cfg.MODEL.WEIGHTS)
res = DefaultTrainer.test(cfg, model)

But I got this error
NotImplementedError: Please either implement build_evaluator() in subclasses, or pass your evaluator as arguments to DefaultTrainer.test().

Thank you for your attention.

[wuyuanyi135]

try this

    test_datasets = ["name_of_registered_coco_datasets"]
    evaluator = [COCOEvaluator(test_set, cfg, False) for test_set in test_datasets]
    metrics = DefaultTrainer.test(cfg, model, evaluator)

[ccssmania]

Ok thank you so much, now it works but I gettings these results

WARNING [11/14 21:57:22 d2.evaluation.coco_evaluation]: No predictions from the model! Set scores to -1

I don't know what does it mean or why is showing:

[11/14 21:57:22 d2.evaluation.testing]: copypaste: Task: bbox
[11/14 21:57:22 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl
[11/14 21:57:22 d2.evaluation.testing]: copypaste: -1.0000,-1.0000,-1.0000,-1.0000,-1.0000,-1.0000
[11/14 21:57:22 d2.evaluation.testing]: copypaste: Task: segm
[11/14 21:57:22 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl
[11/14 21:57:22 d2.evaluation.testing]: copypaste: -1.0000,-1.0000,-1.0000,-1.0000,-1.0000,-1.0000

could you give me a little explanation or advice, I'll really appreciate that.

[wuyuanyi135]

You can try predictor = DefaultPredictor(cfg) and load one of your image as tensor or numpy array with cv2 or PIL, then try predictor(img) to see if it is predicting correctly

[ppwwyyxx]

If I understand correctly I need to set MetadataCatalog.get(dataset_name).evaluator_type

No, as the metadata documentation (https://detectron2.readthedocs.io/tutorials/datasets.html) says you do not need to use it.

The colab tutorial (https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=kblA1IyFvWbT) is now updated to include inference & evaluation.

[ccssmania]

Ok , Thank you so much for your quickly response.

when I run this:

evaluator = COCOEvaluator("data_test", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "data_test")
inference_on_dataset(trainer.model, val_loader, evaluator)

I got this error:

NameError: name 'unicode' is not defined

and this is the all traceback.

---> 14 inference_on_dataset(trainer.model, val_loader, evaluator)

4 frames
/content/detectron2_repo/detectron2/evaluation/evaluator.py in inference_on_dataset(model, data_loader, evaluator)
151 )
152
--> 153 results = evaluator.evaluate()
154 # An evaluator may return None when not in main process.
155 # Replace it by an empty dict instead to make it easier for downstream code to handle

/content/detectron2_repo/detectron2/evaluation/coco_evaluation.py in evaluate(self)
145 self._eval_box_proposals()
146 if "instances" in self._predictions[0]:
--> 147 self._eval_predictions(set(self._tasks))
148 # Copy so the caller can do whatever with results
149 return copy.deepcopy(self._results)

/content/detectron2_repo/detectron2/evaluation/coco_evaluation.py in _eval_predictions(self, tasks)
182 self._coco_api, self._coco_results, task, kpt_oks_sigmas=self._kpt_oks_sigmas
183 )
--> 184 if len(self._coco_results) > 0
185 else None # cocoapi does not handle empty results very well
186 )

/content/detectron2_repo/detectron2/evaluation/coco_evaluation.py in _evaluate_predictions_on_coco(coco_gt, coco_results, iou_type, kpt_oks_sigmas)
463 c.pop("bbox", None)
464
--> 465 coco_dt = coco_gt.loadRes(coco_results)
466 coco_eval = COCOeval(coco_gt, coco_dt, iou_type)
467 # Use the COCO default keypoint OKS sigmas unless overrides are specified

/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py in loadRes(self, resFile)
306 print('Loading and preparing results...')
307 tic = time.time()
--> 308 if type(resFile) == str or type(resFile) == unicode:
309 anns = json.load(open(resFile))
310 elif type(resFile) == np.ndarray:

NameError: name 'unicode' is not defined

Thank you for your help.
Regards.

[ppwwyyxx]

That means your pycocotools is not the latest version.

[Dayan-Zhanchi]

I quick fix is to change the unicode to bytes. The solution can be found here:
https://github.com/cocodataset/cocoapi/issues/49

Alter line 308 of coco.py at pycocotools library directory, from:
if type(resFile) == str or type(resFile) == unicode:
to:
if type(resFile) == str or type(resFile) == bytes:
Works perfectly for Python 3

Or as ppwwyyxx said update the pycocotools.

[1e100]

FYI to others experiencing this issue: the version of pycocotools currently published on PyPI is incompatible with Python3. This is why Detectron2 installation instructions want you to install pycocotools as follows, directly from master.

pip3 install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

Not ideal, but a reasonable workaround.
Most people will probably want to do

pip3 install --user 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

But if you try per-user installation, it will fail. Global installation works fine. You need sudo for it, though.

[djpecot]

Ok , Thank you so much for your quickly response.

when I run this:

evaluator = COCOEvaluator("data_test", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "data_test")
inference_on_dataset(trainer.model, val_loader, evaluator)

I got this error:

NameError: name 'unicode' is not defined

and this is the all traceback.

---> 14 inference_on_dataset(trainer.model, val_loader, evaluator)
4 frames
/content/detectron2_repo/detectron2/evaluation/evaluator.py in inference_on_dataset(model, data_loader, evaluator)
151 )
152
--> 153 results = evaluator.evaluate()
154 # An evaluator may return None when not in main process.
155 # Replace it by an empty dict instead to make it easier for downstream code to handle
/content/detectron2_repo/detectron2/evaluation/coco_evaluation.py in evaluate(self)
145 self._eval_box_proposals()
146 if "instances" in self._predictions[0]:
--> 147 self._eval_predictions(set(self._tasks))
148 # Copy so the caller can do whatever with results
149 return copy.deepcopy(self._results)
/content/detectron2_repo/detectron2/evaluation/coco_evaluation.py in _eval_predictions(self, tasks)
182 self._coco_api, self._coco_results, task, kpt_oks_sigmas=self._kpt_oks_sigmas
183 )
--> 184 if len(self._coco_results) > 0
185 else None # cocoapi does not handle empty results very well
186 )
/content/detectron2_repo/detectron2/evaluation/coco_evaluation.py in _evaluate_predictions_on_coco(coco_gt, coco_results, iou_type, kpt_oks_sigmas)
463 c.pop("bbox", None)
464
--> 465 coco_dt = coco_gt.loadRes(coco_results)
466 coco_eval = COCOeval(coco_gt, coco_dt, iou_type)
467 # Use the COCO default keypoint OKS sigmas unless overrides are specified
/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py in loadRes(self, resFile)
306 print('Loading and preparing results...')
307 tic = time.time()
--> 308 if type(resFile) == str or type(resFile) == unicode:
309 anns = json.load(open(resFile))
310 elif type(resFile) == np.ndarray:
NameError: name 'unicode' is not defined

Thank you for your help.
Regards.

I ran into a similar problem running in Google Colab. Seems like running this command
!pip install --user 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

and/or changing line 308 to
if type(resFile) == str or type(resFile) == bytes:

doesn't solve the issue. I'm guessing this a problem with Google Colab not updating? Not sure how to get around this because it still throws the same error:

NameError: name 'unicode' is not defined

[ppwwyyxx]

The official tutorial https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5 does not have this issue so you can follow what it does.

[djpecot]

For anyone using Colab, if you put this in at the front of your code with the rest of the detectron installs, seems to do the trick:
!pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

[junliu-cn]

test_datasets = ["name_of_registered_coco_datasets"]
evaluator = [COCOEvaluator(test_set, cfg, False) for test_set in test_datasets]
metrics = DefaultTrainer.test(cfg, model, evaluator)

it's not working! it gives me the wrong answer:

OrderedDict([('bbox', {'AP': 0.0, 'AP50': 0.0, 'AP75': 0.0, 'APs': nan, 'APm': 0.0, 'APl': 0.0, 'AP-gear': 0.0, 'AP-axis': 0.0}), ('segm', {'AP': 0.0, 'AP50': 0.0, 'AP75': 0.0, 'APs': nan, 'APm': 0.0, 'APl': 0.0, 'AP-gear': 0.0, 'AP-axis': 0.0})])