Detectron2: How to calculate the mAP, LAMR? Also printing testing accuracy in tensorboard?

[mefahimrahman]

I want to see my test accuracy of the Detectron2. When I train the Detectron2 I've written -
from detectron2.engine import DefaultTrainer from detectron2.config import get_cfg import os cfg = get_cfg() cfg.merge_from_file("./detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml") cfg.DATASETS.TRAIN = ("pedestrian",) cfg.DATASETS.TEST = () cfg.DATALOADER.NUM_WORKERS = 2 cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl" cfg.SOLVER.IMS_PER_BATCH = 2 cfg.SOLVER.BASE_LR = 0.02 cfg.SOLVER.MAX_ITER = 300 cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128 cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 os.makedirs(cfg.OUTPUT_DIR, exist_ok=True) trainer = DefaultTrainer(cfg) trainer.resume_or_load(resume=False) trainer.train()

And for testing, I've written -
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth") cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set the testing threshold for this model cfg.DATASETS.TEST = ("pedestrian_night", ) predictor = DefaultPredictor(cfg)

When I went to print the accuracy in tensorboard -
%load_ext tensorboard %tensorboard --logdir output

Since the output folder is created from the training the tensorboard shows the training accuracy. So How can I get the testing accuracy in tensorboard?

[ppwwyyxx]

Detectron2 calculates mAP using COCO API or other APIs provided by different datasets.
Detectron2 does not calculate LAMR.
Detectron2 already shows test accuracy in tensorboard, by default at the end of training. You can change cfg.TEST.EVAL_PERIOD to make it more frequent.

[alekhka]

Setting EVAL_PERIOD to low value (I set it to 5000) crashed my training after a while. It seems like the eval results after each eval run isn't deleted and after 10 or so evaluations it goes out of memory (on a 300GB RAM system). Is that expected or am I doing something wrong?

[ppwwyyxx]

It's not expected and if you need help with an unexpected issue, please include details following the issue template.

[mefahimrahman]

@ppwwyyxx after training the output folder structure -

output
-events.out.tfevents.1582636641.13f26e5de6d5.495.0
-last_checkpoint
-metrics.json
-model_final.pth

After doing the testing the output folder structure -
-coco_instances_results.json
-events.out.tfevents.1582636641.13f26e5de6d5.495.0
-instances_predictions.pth
-last_checkpoint
-metrics.json
-model_final.pth

So how can I get the testing accuracy in the tensorboard from this folder?