Detectron: NotImplementedError: No evaluator for dataset

[ChuhuaW]

Expected results

Hi there,
I was training custom data using

python2 tools/train_net.py --cfg configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml OUTPUT_DIR /l/vision/v3/cw234/train_results

The dataset was converted into COCO json format. The .yaml has been modified( TRAIN.DATASETS & TEST.DATASETS ) and the dataset was added to dataset catalog .

Actual results

I get an error in evaluating:

INFO task_evaluation.py:  75: Evaluating detections
Traceback (most recent call last):
  File "tools/train_net.py", line 281, in <module>
    main()
  File "tools/train_net.py", line 122, in main
    test_model(checkpoints['final'], args.multi_gpu_testing, args.opts)
  File "tools/train_net.py", line 277, in test_model
    test_net.main(multi_gpu_testing=multi_gpu_testing)
  File "/nfs/zapdos/home/data/vision3/cw234/detectron/tools/test_net.py", line 98, in main
    output_dir, ind_range=ind_range, multi_gpu_testing=multi_gpu_testing
  File "/nfs/zapdos/home/data/vision3/cw234/detectron/lib/core/test_engine.py", line 83, in run_inference
    results = parent_func(output_dir, multi_gpu=multi_gpu_testing)
  File "/nfs/zapdos/home/data/vision3/cw234/detectron/lib/core/test_engine.py", line 110, in test_net_on_dataset
    dataset, all_boxes, all_segms, all_keyps, output_dir
  File "/nfs/zapdos/home/data/vision3/cw234/detectron/lib/datasets/task_evaluation.py", line 59, in evaluate_all
    dataset, all_boxes, output_dir, use_matlab=use_matlab
  File "/nfs/zapdos/home/data/vision3/cw234/detectron/lib/datasets/task_evaluation.py", line 97, in evaluate_boxes
    'No evaluator for dataset: {}'.format(dataset.name)
NotImplementedError: No evaluator for dataset: ego_data

Sorry if the problem looks obvious. I guess I'm missing some steps when setting up the data. Can anyone help me? Thank you!

System information

• Operating system: Linux Server release 7.4 (Maipo)
• Compiler version: gcc/5.2.0
• CUDA version: 8.0
• cuDNN version: 7.0.5
• NVIDIA driver version: 390.30
• GPU models (for all devices if they are not all the same): TITAN X (Pascal)
• PYTHONPATH environment variable: /l/vision/v3/cw234/caffe2/build
• python --version output: Python 2.7.14

[kampelmuehler]

in your .yaml, add the FORCE_JSON_DATASET_EVAL: True option under TEST like

TEST:
  FORCE_JSON_DATASET_EVAL: True

to do COCO style evaluation of your json dataset, or implement your own evaluation script.

[ChuhuaW]

@kampelmuehler Thanks!!! It worked!!!

[youngxiao]

@kampelmuehler It works. Thanks. However I got a new issue, just as:

Loading and preparing results...
Traceback (most recent call last):
  File "tools/train_net.py", line 128, in <module>
    main()
  File "tools/train_net.py", line 113, in main
    test_model(checkpoints['final'], args.multi_gpu_testing, args.opts)
  File "tools/train_net.py", line 123, in test_model
    check_expected_results=True,
  File "/home/young/DeepLearning/detectron/detectron/core/test_engine.py", line 128, in run_inference
    all_results = result_getter()
  File "/home/young/DeepLearning/detectron/detectron/core/test_engine.py", line 108, in result_getter
    multi_gpu=multi_gpu_testing
  File "/home/young/DeepLearning/detectron/detectron/core/test_engine.py", line 164, in test_net_on_dataset
    dataset, all_boxes, all_segms, all_keyps, output_dir
  File "/home/young/DeepLearning/detectron/detectron/datasets/task_evaluation.py", line 60, in evaluate_all
    dataset, all_boxes, output_dir, use_matlab=use_matlab
  File "/home/young/DeepLearning/detectron/detectron/datasets/task_evaluation.py", line 80, in evaluate_boxes
    dataset, all_boxes, output_dir, use_salt=not_comp, cleanup=not_comp
  File "/home/young/DeepLearning/detectron/detectron/datasets/json_dataset_evaluator.py", line 138, in evaluate_boxes
    coco_eval = _do_detection_eval(json_dataset, res_file, output_dir)
  File "/home/young/DeepLearning/detectron/detectron/datasets/json_dataset_evaluator.py", line 192, in _do_detection_eval
    coco_dt = json_dataset.COCO.loadRes(str(res_file))
  File "build/bdist.linux-x86_64/egg/pycocotools/coco.py", line 318, in loadRes
IndexError: list index out of range

[mate-cape]

Got the same error. Detectron saves detections to a json file:
INFO json_dataset_evaluator.py: 162: Writing bbox results json to: <detectron output path>/test/<dataset name>/generalized_rcnn/bbox_<dataset name>_results.json

However, for some reason this file contains an empty list only ("[]"), and this causes an error for pycocotools.

Update: already detections.pkl is full of empty numpy arrays (np.array([], shape=(0, 5), dtype=float32))

[mate-cape]

I think I found the issue: all the results are filtered out in box_results_with_nms_and_limit(scores, boxes) as scores are below threshold.

I tried to evaluate a network after just 30 training iterations, just to check if everything worked well. But all results were filtered out, and down the line pycocotools threw an error because it got an empty array... It would be nice to

@youngxiao: if you still have this problem, try adding to the TEST section of your .yaml file SCORE_THRESH: 0.001

[ChuhuaW]

@mate-cape Are you using custom dataset?

[mate-cape]

@ChuhuaW I'm making experiments with a modified COCO dataset, using some custom dataset augmentation. But I got this error before the augmentation, so that was the standard 2017 COCO.