Models: Object Detection : KeyError: 'groundtruth_classes'

[OverFlow7]

Hello, when i try to run eval.py i get the following error :

WARNING:root:The following classes have no ground truth examples: [1 2] /opt/models/research/object_detection/utils/metrics.py:144: RuntimeWarning: invalid value encountered in true_divide num_images_correctly_detected_per_class / num_gt_imgs_per_class) /opt/models/research/object_detection/utils/object_detection_evaluation.py:585: RuntimeWarning: Mean of empty slice mean_ap = np.nanmean(self.average_precision_per_class) /opt/models/research/object_detection/utils/object_detection_evaluation.py:586: RuntimeWarning: Mean of empty slice mean_corloc = np.nanmean(self.corloc_per_class) Traceback (most recent call last): File "eval.py", line 130, in <module> tf.app.run() File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "eval.py", line 126, in main FLAGS.checkpoint_dir, FLAGS.eval_dir) File "/opt/models/research/object_detection/evaluator.py", line 210, in evaluate save_graph_dir=(eval_dir if eval_config.save_graph else '')) File "/opt/models/research/object_detection/eval_util.py", line 381, in repeated_checkpoint_run save_graph_dir) File "/opt/models/research/object_detection/eval_util.py", line 269, in _run_checkpoint_once image_id=batch, groundtruth_dict=result_dict) File "/opt/models/research/object_detection/utils/object_detection_evaluation.py", line 167, in add_single_ground_truth_image_info standard_fields.InputDataFields.groundtruth_classes] KeyError: 'groundtruth_classes'

It used to work fine before.

I checked my annotation files and txt.files and they look fine .

[bignamehyp]

I apologize but I am having a hard time understanding what the problem is, where the problem is, and what version it affects. Please resubmit and pay attention to the issue template (https://github.com/tensorflow/tensorflow/issues/new) . Please provide all the information it asks. Thank you.

[izzrak]

Same issue here, the eval.py is not working after updated to this pr #2622

[OverFlow7]

Oh sorry and thanks for your answers , here are more details:

• Have I written custom code:No
• OS Platform and Distribution: Ubuntu 16.04
• TensorFlow installed from: binary
• TensorFlow version: 1.4.0
• Python version: 2.7
• CUDA/cuDNN version:8.0/6.0
• GPU model and memory: GTX 1060 , 6GB RAM
• Exact command to reproduce: CUDA_VISIBLE_DEVICES=1 python eval.py --logtostderr --pipeline_config_path="/home/frcnn.config" --checkpoint_dir="/home/frcnn/training" --eval_dir="/home/frcnn/eval"

Describe the problem

I used to be able to run object detection/eval.py , without any problems (no warning or error) but now i get the folowling warning and error
WARNING:root:The following classes have no ground truth examples: [1 2] /opt/models/research/object_detection/utils/metrics.py:144: RuntimeWarning: invalid value encountered in true_divide num_images_correctly_detected_per_class / num_gt_imgs_per_class) /opt/models/research/object_detection/utils/object_detection_evaluation.py:585: RuntimeWarning: Mean of empty slice mean_ap = np.nanmean(self.average_precision_per_class) /opt/models/research/object_detection/utils/object_detection_evaluation.py:586: RuntimeWarning: Mean of empty slice mean_corloc = np.nanmean(self.corloc_per_class) Traceback (most recent call last): File "eval.py", line 130, in <module> tf.app.run() File "/root/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "eval.py", line 126, in main FLAGS.checkpoint_dir, FLAGS.eval_dir) File "/opt/models/research/object_detection/evaluator.py", line 210, in evaluate save_graph_dir=(eval_dir if eval_config.save_graph else '')) File "/opt/models/research/object_detection/eval_util.py", line 381, in repeated_checkpoint_run save_graph_dir) File "/opt/models/research/object_detection/eval_util.py", line 269, in _run_checkpoint_once image_id=batch, groundtruth_dict=result_dict) File "/opt/models/research/object_detection/utils/object_detection_evaluation.py", line 167, in add_single_ground_truth_image_info standard_fields.InputDataFields.groundtruth_classes] KeyError: 'groundtruth_classes'

[jsigman]

I would look into your scripts to produce your TFRecord files. I was having this same problem after updating from an older version of the Object Detection API. You're getting this error because Tensorflow is returning an empty dictionary instead of one containing the records of all the detections in this evaluated image.

This might not be the same problem you're having, but for me it was failing because I had a bug where ground truth bounding boxes were drawn outside of the image, i.e. my normalized xmax/ymax were greater than 1.0.

To find this bug, I used this test script to inspect the test set TFRecord:

import tensorflow as tf
from google.protobuf.json_format import MessageToJson

file = "/path/to/tfrecord"
fileNum=1
for example in tf.python_io.tf_record_iterator(file):
    jsonMessage = MessageToJson(tf.train.Example.FromString(example))
    with open("RESULTS/image_{}".format(fileNum),"w") as text_file:
        print(jsonMessage,file=text_file)
    fileNum+=1

This will extract the data from the TFRecords and put each into a file for inspection. My guess is that something is wrong with your data processing.

[bignamehyp]

@tombstone can you please take a look? Thanks.

[izzrak]

This problem has been solved by update PR #2725 , however when I run the eval.py script, I got 0 mAP and the following warning

WARNING:root:image 1000 does not have groundtruth difficult flag specified

Then I went back and tried the old one, and it worked fine.

[OverFlow7]

Thank you @jsigman , i used your script to check my tfrecords, but couldn't find anything suspicous , bounding boxes values are between 0 and 1 , so it's not that.

Thank you @izzrak , but i reinstalled tensorflow/models, and i still get the same error.

[tombstone]

@OverFlow7 Could you tell a bit about your data. It will be great if you can identify the tf_record where this fails.

[tombstone]

@izzrak If you are using the models from the zoo, can you check the updated links and the code and tell use if that solves your problem ?

[izzrak]

@tombstone I just updated to the latest commit, the mAP is not 0 anymore but lower than the older commit I am using. The mAP calculated by the latest commit is 0.69, and the old one is 0.78. So I wonder if there is any change made to the evaluation metrics.

[OverFlow7]

@tombstone thank you.
About my data: they are in the pascal voc format. I had about 3000 images, and 5 classes, on my evaluation set when the error appeared .
I made new tfrecords for both training and evalutation,reduced the number of classes to 2, 200 images for training and 20 images for evaluation, and stiil got the error

[OverFlow7]

I decided to try again but using a very small subset of the pascal voc dataset,

so i took 4 images from label "aeroplane" 2 for training 2 for evaluating.

aeroplane_train.txt:
2008_000037 1
2008_000151 1

aeroplane_val.txt:
2008_000021 1
2008_001468 1

and label map
item {
id: 1
name: 'aeroplane'
}

I let it train a little to get a checkpoint, stop the training, start the evaluating and once again i get the same error .

When using @jsigman 's script to analyze the eval tfrecord , i get the following: ( i removed the image/encoded stuff because it takes a lot of room)
{
"features": {
"feature": {
"image/object/class/label": {
"int64List": {
"value": [
"1"
]
}
},
"image/object/bbox/xmax": {
"floatList": {
"value": [
0.949999988079071
]
}
},
"image/height": {
"int64List": {
"value": [
"375"
]
}
},
"image/object/truncated": {
"int64List": {
"value": [
"0"
]
}
},
"image/format": {
"bytesList": {
"value": [
"anBlZw=="
]
}
},
"image/encoded": {
"bytesList": {
"value": [
""
]
}
},
"image/object/difficult": {
"int64List": {
"value": [
"0"
]
}
},
"image/object/bbox/ymax": {
"floatList": {
"value": [
0.7680000066757202
]
}
},
"image/object/bbox/ymin": {
"floatList": {
"value": [
0.3946666717529297
]
}
},
"image/source_id": {
"bytesList": {
"value": [
"MjAwOF8wMDAwMjEuanBn"
]
}
},
"image/key/sha256": {
"bytesList": {
"value": [
"YTZkODJiNTlhNzhjYTBlZDIyM2NmYTRkYjkzMzAxOTQ2MGQwN2E1NjUwODhmNGUzNTg3YzhlMzQ3OTk2Yjg5NA=="
]
}
},
"image/object/view": {
"bytesList": {
"value": [
"RnJvbnRhbA=="
]
}
},
"image/object/class/text": {
"bytesList": {
"value": [
"YWVyb3BsYW5l"
]
}
},
"image/object/bbox/xmin": {
"floatList": {
"value": [
0.02800000086426735
]
}
},
"image/width": {
"int64List": {
"value": [
"500"
]
}
},
"image/filename": {
"bytesList": {
"value": [
"MjAwOF8wMDAwMjEuanBn"
]
}
}
}
}
}
{
"features": {
"feature": {
"image/width": {
"int64List": {
"value": [
"500"
]
}
},
"image/object/class/label": {
"int64List": {
"value": [
"1"
]
}
},
"image/object/bbox/xmax": {
"floatList": {
"value": [
1.0
]
}
},
"image/height": {
"int64List": {
"value": [
"375"
]
}
},
"image/object/truncated": {
"int64List": {
"value": [
"1"
]
}
},
"image/format": {
"bytesList": {
"value": [
"anBlZw=="
]
}
},
"image/encoded": {
"bytesList": {
"value": [
"..."
]
}
},
"image/object/difficult": {
"int64List": {
"value": [
"0"
]
}
},
"image/filename": {
"bytesList": {
"value": [
"MjAwOF8wMDE0NjguanBn"
]
}
},
"image/object/bbox/ymin": {
"floatList": {
"value": [
0.3199999928474426
]
}
},
"image/source_id": {
"bytesList": {
"value": [
"MjAwOF8wMDE0NjguanBn"
]
}
},
"image/key/sha256": {
"bytesList": {
"value": [
"YmIzZjI0YTY1NDFkMzIyMTM2ZTA3ZDExZDk5NThhZTA3MWRlNzVkYTA2NWMxMjAxMGMzYjNjNDcyNzJlN2Q4MQ=="
]
}
},
"image/object/view": {
"bytesList": {
"value": [
"RnJvbnRhbA=="
]
}
},
"image/object/class/text": {
"bytesList": {
"value": [
"YWVyb3BsYW5l"
]
}
},
"image/object/bbox/xmin": {
"floatList": {
"value": [
0.017999999225139618
]
}
},
"image/object/bbox/ymax": {
"floatList": {
"value": [
0.9573333263397217
]
}
}
}
}
}

[OverFlow7]

It seems the error come from _process_batch in evaluator.py , every time this exception is raised :

try:
  result_dict = sess.run(tensor_dict)
  counters['success'] += 1
except;tf.errors.InvalidArgumentError:
  logging.info('Skipping image')
  counters['skipped'] += 1
  return {}

InvalidArgumentError:
Incompatible shapes: [51300] vs. [20320]\n\t [[Node: Decode/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Decode/Exp, Decode/get_center_coordinates_and_sizes/sub)]]\n\t [[Node: ToAbsoluteCoordinates/Assert/AssertGuard/Assert/Switch_1/_1343 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_747_ToAbsoluteCoordinates/Assert/AssertGuard/Assert/Switch_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

So this node seems to be the one responsible.

name: "Decode/mul"
op: "Mul"
input: "Decode/Exp"
input: "Decode/get_center_coordinates_and_sizes/sub"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}

[OverFlow7]

Ok, closing the issue, the error was due to having

feature_extractor {
type: 'faster_rcnn_inception_resnet_v2'
first_stage_features_stride: 8
}
first_stage_anchor_generator {
grid_anchor_generator {
scales: [0.25, 0.5, 1.0, 2.0]
aspect_ratios: [0.5, 1.0, 2.0]
height_stride: 16
width_stride: 16
}

instead of having the same stride for the 3 .

[AswathKiruba]

@OverFlow7 Thanks for your response. I am getting the same error too. My question is that, why are we getting this error while running evalutor.py and not while training. Could you please let me know.

[alexpm94]

In the configuration pipeline, I didn't specify the evaluation metrics. In eval_config add the following line: metrics_set: "coco_detection_metrics"
It solved the problem for me.