Detectron: KeyError in json_dataset.py when training model on own dataset

[gabriellapizzuto]

I am trying to train a model with my own dataset but I am getting the following error. Any suggestions?

Expected results

Training model for custom dataset

Actual results

Found Detectron ops lib: /usr/local/lib/libcaffe2_detectron_ops_gpu.so
Found Detectron ops lib: /usr/local/lib/libcaffe2_detectron_ops_gpu.so
INFO train_net.py: 104: Called with args:
INFO train_net.py: 105: Namespace(cfg_file='configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml', multi_gpu_testing=False, opts=['OUTPUT_DIR', '/tmp/detectron-output'], skip_test=False)
INFO train_net.py: 111: Training with config:
INFO train_net.py: 112: {'BBOX_XFORM_CLIP': 4.1351665567423561,
'CLUSTER': {'ON_CLUSTER': False},
'DATA_LOADER': {'NUM_THREADS': 4},
'DEDUP_BOXES': 0.0625,
'DOWNLOAD_CACHE': '/tmp/detectron-download-cache',
'EPS': 1e-14,
'EXPECTED_RESULTS': [],
'EXPECTED_RESULTS_ATOL': 0.005,
'EXPECTED_RESULTS_EMAIL': '',
'EXPECTED_RESULTS_RTOL': 0.1,
'FAST_RCNN': {'MLP_HEAD_DIM': 1024,
'ROI_BOX_HEAD': 'fast_rcnn_heads.add_roi_2mlp_head',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 2},
'FPN': {'COARSEST_STRIDE': 32,
'DIM': 256,
'EXTRA_CONV_LEVELS': False,
'FPN_ON': True,
'MULTILEVEL_ROIS': True,
'MULTILEVEL_RPN': True,
'ROI_CANONICAL_LEVEL': 4,
'ROI_CANONICAL_SCALE': 224,
'ROI_MAX_LEVEL': 5,
'ROI_MIN_LEVEL': 2,
'RPN_ANCHOR_START_SIZE': 32,
'RPN_ASPECT_RATIOS': (0.5, 1, 2),
'RPN_MAX_LEVEL': 6,
'RPN_MIN_LEVEL': 2,
'ZERO_INIT_LATERAL': False},
'KRCNN': {'CONV_HEAD_DIM': 256,
'CONV_HEAD_KERNEL': 3,
'CONV_INIT': 'GaussianFill',
'DECONV_DIM': 256,
'DECONV_KERNEL': 4,
'DILATION': 1,
'HEATMAP_SIZE': -1,
'INFERENCE_MIN_SIZE': 0,
'KEYPOINT_CONFIDENCE': 'bbox',
'LOSS_WEIGHT': 1.0,
'MIN_KEYPOINT_COUNT_FOR_VALID_MINIBATCH': 20,
'NMS_OKS': False,
'NORMALIZE_BY_VISIBLE_KEYPOINTS': True,
'NUM_KEYPOINTS': -1,
'NUM_STACKED_CONVS': 8,
'ROI_KEYPOINTS_HEAD': '',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 0,
'UP_SCALE': -1,
'USE_DECONV': False,
'USE_DECONV_OUTPUT': False},
'MATLAB': 'matlab',
'MEMONGER': True,
'MEMONGER_SHARE_ACTIVATIONS': False,
'MODEL': {'BBOX_REG_WEIGHTS': (10.0, 10.0, 5.0, 5.0),
'CLS_AGNOSTIC_BBOX_REG': False,
'CONV_BODY': 'FPN.add_fpn_ResNet50_conv5_body',
'EXECUTION_TYPE': 'dag',
'FASTER_RCNN': True,
'KEYPOINTS_ON': False,
'MASK_ON': False,
'NUM_CLASSES': 12,
'RPN_ONLY': False,
'TYPE': 'generalized_rcnn'},
'MRCNN': {'CLS_SPECIFIC_MASK': True,
'CONV_INIT': 'GaussianFill',
'DILATION': 2,
'DIM_REDUCED': 256,
'RESOLUTION': 14,
'ROI_MASK_HEAD': '',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 0,
'THRESH_BINARIZE': 0.5,
'UPSAMPLE_RATIO': 1,
'USE_FC_OUTPUT': False,
'WEIGHT_LOSS_MASK': 1.0},
'NUM_GPUS': 1,
'OUTPUT_DIR': '/tmp/detectron-output',
'PIXEL_MEANS': array([[[ 102.9801, 115.9465, 122.7717]]]),
'RESNETS': {'NUM_GROUPS': 1,
'RES5_DILATION': 1,
'STRIDE_1X1': True,
'TRANS_FUNC': 'bottleneck_transformation',
'WIDTH_PER_GROUP': 64},
'RETINANET': {'ANCHOR_SCALE': 4,
'ASPECT_RATIOS': (0.5, 1.0, 2.0),
'BBOX_REG_BETA': 0.11,
'BBOX_REG_WEIGHT': 1.0,
'CLASS_SPECIFIC_BBOX': False,
'INFERENCE_TH': 0.05,
'LOSS_ALPHA': 0.25,
'LOSS_GAMMA': 2.0,
'NEGATIVE_OVERLAP': 0.4,
'NUM_CONVS': 4,
'POSITIVE_OVERLAP': 0.5,
'PRE_NMS_TOP_N': 1000,
'PRIOR_PROB': 0.01,
'RETINANET_ON': False,
'SCALES_PER_OCTAVE': 3,
'SHARE_CLS_BBOX_TOWER': False,
'SOFTMAX': False},
'RFCN': {'PS_GRID_SIZE': 3},
'RNG_SEED': 3,
'ROOT_DIR': '/home/gabriella/Detectron',
'RPN': {'ASPECT_RATIOS': (0.5, 1, 2),
'RPN_ON': True,
'SIZES': (64, 128, 256, 512),
'STRIDE': 16},
'SOLVER': {'BASE_LR': 0.0025,
'GAMMA': 0.1,
'LOG_LR_CHANGE_THRESHOLD': 1.1,
'LRS': [],
'LR_POLICY': 'steps_with_decay',
'MAX_ITER': 60000,
'MOMENTUM': 0.9,
'SCALE_MOMENTUM': True,
'SCALE_MOMENTUM_THRESHOLD': 1.1,
'STEPS': [0, 30000, 40000],
'STEP_SIZE': 30000,
'WARM_UP_FACTOR': 0.3333333333333333,
'WARM_UP_ITERS': 500,
'WARM_UP_METHOD': u'linear',
'WEIGHT_DECAY': 0.0001},
'TEST': {'BBOX_AUG': {'AREA_TH_HI': 32400,
'AREA_TH_LO': 2500,
'ASPECT_RATIOS': (),
'ASPECT_RATIO_H_FLIP': False,
'COORD_HEUR': 'UNION',
'ENABLED': False,
'H_FLIP': False,
'MAX_SIZE': 4000,
'SCALES': (),
'SCALE_H_FLIP': False,
'SCALE_SIZE_DEP': False,
'SCORE_HEUR': 'UNION'},
'BBOX_REG': True,
'BBOX_VOTE': {'ENABLED': False,
'SCORING_METHOD': 'ID',
'SCORING_METHOD_BETA': 1.0,
'VOTE_TH': 0.8},
'COMPETITION_MODE': True,
'DATASETS': ('gods_2018_val',),
'DETECTIONS_PER_IM': 100,
'FORCE_JSON_DATASET_EVAL': True,
'KPS_AUG': {'AREA_TH': 32400,
'ASPECT_RATIOS': (),
'ASPECT_RATIO_H_FLIP': False,
'ENABLED': False,
'HEUR': 'HM_AVG',
'H_FLIP': False,
'MAX_SIZE': 4000,
'SCALES': (),
'SCALE_H_FLIP': False,
'SCALE_SIZE_DEP': False},
'MASK_AUG': {'AREA_TH': 32400,
'ASPECT_RATIOS': (),
'ASPECT_RATIO_H_FLIP': False,
'ENABLED': False,
'HEUR': 'SOFT_AVG',
'H_FLIP': False,
'MAX_SIZE': 4000,
'SCALES': (),
'SCALE_H_FLIP': False,
'SCALE_SIZE_DEP': False},
'MAX_SIZE': 833,
'NMS': 0.5,
'PRECOMPUTED_PROPOSALS': False,
'PROPOSAL_FILES': (),
'PROPOSAL_LIMIT': 2000,
'RPN_MIN_SIZE': 0,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 1000,
'RPN_PRE_NMS_TOP_N': 1000,
'SCALE': 500,
'SCORE_THRESH': 0.05,
'SOFT_NMS': {'ENABLED': False, 'METHOD': 'linear', 'SIGMA': 0.5},
'WEIGHTS': ''},
'TRAIN': {'ASPECT_GROUPING': True,
'AUTO_RESUME': True,
'BATCH_SIZE_PER_IM': 256,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'CROWD_FILTER_THRESH': 0.7,
'DATASETS': ('gods_2018_train',),
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'FREEZE_CONV_BODY': False,
'GT_MIN_AREA': -1,
'IMS_PER_BATCH': 2,
'MAX_SIZE': 833,
'PROPOSAL_FILES': (),
'RPN_BATCH_SIZE_PER_IM': 256,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 0,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 2000,
'RPN_STRADDLE_THRESH': 0,
'SCALES': (500,),
'SNAPSHOT_ITERS': 20000,
'USE_FLIPPED': True,
'WEIGHTS': u'/tmp/detectron-download-cache/ImageNetPretrained/MSRA/R-50.pkl'},
'USE_NCCL': False,
'VIS': False,
'VIS_TH': 0.9}
INFO train_net.py: 206: Building model: generalized_rcnn
WARNING cnn.py: 40: [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information.
WARNING memonger.py: 70: NOTE: Executing memonger to optimize gradient memory
I0329 10:47:07.330505 18443 memonger.cc:253] Remapping 83 using 19 shared blobs.
INFO memonger.py: 112: Memonger memory optimization took 0.0245258808136 secs
I0329 10:47:08.507196 18443 context_gpu.cu:321] GPU 0: 153 MB
I0329 10:47:08.507233 18443 context_gpu.cu:325] Total: 153 MB
I0329 10:47:08.578140 18443 context_gpu.cu:321] GPU 0: 310 MB
I0329 10:47:08.578156 18443 context_gpu.cu:325] Total: 310 MB
INFO train_net.py: 254: Loading dataset: ('gods_2018_train',)
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Traceback (most recent call last):
File "tools/train_net.py", line 280, in
main()
File "tools/train_net.py", line 119, in main
checkpoints = train_model()
File "tools/train_net.py", line 133, in train_model
setup_model_for_training(model, weights_file, output_dir)
File "tools/train_net.py", line 232, in setup_model_for_training
add_model_training_inputs(model)
File "tools/train_net.py", line 256, in add_model_training_inputs
cfg.TRAIN.DATASETS, cfg.TRAIN.PROPOSAL_FILES
File "/home/gabriella/Detectron/lib/datasets/roidb.py", line 61, in combined_roidb_for_training
roidbs = [get_roidb(*args) for args in zip(dataset_names, proposal_files)]
File "/home/gabriella/Detectron/lib/datasets/roidb.py", line 46, in get_roidb
crowd_filter_thresh=cfg.TRAIN.CROWD_FILTER_THRESH
File "/home/gabriella/Detectron/lib/datasets/json_dataset.py", line 113, in get_roidb
self._add_gt_annotations(entry)
File "/home/gabriella/Detectron/lib/datasets/json_dataset.py", line 216, in _add_gt_annotations
cls = self.json_category_id_to_contiguous_id[obj['category_id']]
KeyError: 1

Detailed steps to reproduce

python2 tools/train_net.py --cfg configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml OUTPUT_DIR /tmp/detectron-output

[ambigus9]

@gabriellap Maybe you can consider build Detectron using Docker file. And then, try again. Also, if you can post the dataset_catalog.py and tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml will help.

[DANNALI35]

I encountered the same issue, it was because that some of my 'category_id' in the training annotation json file is 'none' , but that should be class annotation instead of 'none'.
Problem solved when all 'category_id' are correctly set to int class id, just like the pic below. hope it helps.