Models: Object_Detection ValueError: operands could not be broadcast together with shapes (0,) (16,)

[ucesfpa]

Dear all,
I am trying to train the API object detection model on my dataset with just one class. The training ended and i tried to run the evaluation (eval.py) but it gives me an error that I am not able to fully understand:

python eval.py
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
2017-11-06 11:24:50.514416: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-11-06 11:24:50.710928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:05:00.0
totalMemory: 10.91GiB freeMemory: 10.56GiB
2017-11-06 11:24:50.710953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from ./models/model/train/model.ckpt-200000
INFO:tensorflow:Restoring parameters from ./models/model/train/model.ckpt-200000
2017-11-06 11:24:57.255021: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: FIFOQueue '_2_parallel_read/common_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: parallel_read/common_queue_Dequeue = QueueDequeueV2component_types=[DT_STRING, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
WARNING:root:The following classes have no ground truth examples: 1
/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/utils/metrics.py:144: RuntimeWarning: invalid value encountered in true_divide
num_images_correctly_detected_per_class / num_gt_imgs_per_class)
/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/utils/object_detection_evaluation.py:585: RuntimeWarning: Mean of empty slice
mean_ap = np.nanmean(self.average_precision_per_class)
/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/utils/object_detection_evaluation.py:586: RuntimeWarning: Mean of empty slice
mean_corloc = np.nanmean(self.corloc_per_class)
Traceback (most recent call last):
File "eval.py", line 130, in
tf.app.run()
File "/home/ucesfpa/ObjectDetection/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "eval.py", line 126, in main
FLAGS.checkpoint_dir, FLAGS.eval_dir)
File "/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/evaluator.py", line 210, in evaluate
save_graph_dir=(eval_dir if eval_config.save_graph else ''))
File "/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/eval_util.py", line 381, in repeated_checkpoint_run
save_graph_dir)
File "/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/eval_util.py", line 269, in _run_checkpoint_once
image_id=batch, groundtruth_dict=result_dict)
File "/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/utils/object_detection_evaluation.py", line 174, in add_single_ground_truth_image_info
standard_fields.InputDataFields.groundtruth_difficult, None))
File "/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/utils/object_detection_evaluation.py", line 447, in add_single_ground_truth_image_info
groundtruth_is_group_of_list.astype(dtype=bool))
File "/home/ucesfpa/ObjectDetection/TF_Models/models/research/object_detection/utils/object_detection_evaluation.py", line 527, in _update_ground_truth_statistics
& ~groundtruth_is_group_of_list] == class_index)
ValueError: operands could not be broadcast together with shapes (0,) (16,)

Can anyone of you explain the problem?

Thank you,
Fabio

[Robmosh91]

Got the same issue since today.
Is there a context with updating tensorflow to version 1.4?

[daavoo]

I'm having similar issues. Not sure if it's related with th 1.4 version or with this commit https://github.com/tensorflow/models/commit/8a72df2d8c62f7ffa2d58995fc5d484e47955546#diff-806f63cccd92084fdb6308e55be22439 wich was 10 days ago.

[angerson]

Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version. Also, did you compile from source or install a binary? Make sure you also include the exact command if possible to produce the output included in your test case. If you are unclear what to include see the issue template displayed in the Github new issue template.

We ask for this in the issue submission template, because it is really difficult to help without that information. Thanks!

[Robmosh91]

Platform is Ubuntu 16.04 with Cuda 8 and cudnn 6.
TensorFlow version is 1.4 and installed using pip.

Exact command is
python /usr/local/lib/python2.7/dist-packages/tensorflow/models/research/object_detection/eval.py
--logtostderr
--pipeline_config_path=${PATH_TO_YOUR_PIPELINE_CONFIG}
--checkpoint_dir=${PATH_TO_TRAIN_DIR}
--eval_dir=${PATH_TO_EVAL_DIR}

But also the training seems to not work anymore but doesn't give an error.

[Da-Capo]

It seem the value of self.groundtruth_is_difficult_list cannot be get rightly by some reason.
I have temporarily solve this problem by force the list of image/object/difficult to be None
If you dont need 'difficult' , you can add groundtruth_dict[standard_fields.InputDataFields.groundtruth_difficult] = None
at 168 line in object_detection_evaluation.py

[tombstone]

This is being worked on. Will send out a fix shortly.

[Robmosh91]

Hi,
can we reopen this issue?
I am again getting this exact same error while running the evaluation since a few days.
Have been doing some research where it's coming from but have no clue yet.
The data set and models are fine and already where trained and I just wanted to fine tune them with a newer and bigger data set.
On this new and the old data set I now get the error.

Thanks for your help.

Error Message:
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
2018-01-26 16:24:13.245096: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-01-26 16:24:13.467321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:05:00.0
totalMemory: 11.92GiB freeMemory: 3.10GiB
2018-01-26 16:24:13.635727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 1 with properties:
name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076
pciBusID: 0000:06:00.0
totalMemory: 11.92GiB freeMemory: 3.35GiB
2018-01-26 16:24:13.635861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix
2018-01-26 16:24:13.635887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1
2018-01-26 16:24:13.635893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0: Y Y
2018-01-26 16:24:13.635897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1: Y Y
2018-01-26 16:24:13.635904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0, compute capability: 5.2)
2018-01-26 16:24:13.635909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:06:00.0, compute capability: 5.2)
INFO:tensorflow:Restoring parameters from /home/rkoch/FuE/Projekte/Bilderkennung/TF_ObjectDetection/tf_run_7/train/model.ckpt-0
INFO:tensorflow:Restoring parameters from /home/rkoch/FuE/Projekte/Bilderkennung/TF_ObjectDetection/tf_run_7/train/model.ckpt-0
WARNING:root:The following classes have no ground truth examples: [ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32 33 34 35 36 37 38]
/usr/local/lib/python2.7/dist-packages/object_detection-0.1-py2.7.egg/object_detection/utils/metrics.py:144: RuntimeWarning: invalid value encountered in true_divide
/usr/local/lib/python2.7/dist-packages/object_detection-0.1-py2.7.egg/object_detection/utils/object_detection_evaluation.py:585: RuntimeWarning: Mean of empty slice
/usr/local/lib/python2.7/dist-packages/object_detection-0.1-py2.7.egg/object_detection/utils/object_detection_evaluation.py:586: RuntimeWarning: Mean of empty slice
Traceback (most recent call last):
File "object_detection/eval.py", line 133, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/eval.py", line 129, in main
FLAGS.checkpoint_dir, FLAGS.eval_dir)
File "build/bdist.linux-x86_64/egg/object_detection/evaluator.py", line 210, in evaluate
save_graph_dir=(eval_dir if eval_config.save_graph else ''))
File "build/bdist.linux-x86_64/egg/object_detection/eval_util.py", line 381, in repeated_checkpoint_run
save_graph_dir)
File "build/bdist.linux-x86_64/egg/object_detection/eval_util.py", line 269, in _run_checkpoint_once
image_id=batch, groundtruth_dict=result_dict)
File "build/bdist.linux-x86_64/egg/object_detection/utils/object_detection_evaluation.py", line 174, in add_single_ground_truth_image_info
File "build/bdist.linux-x86_64/egg/object_detection/utils/object_detection_evaluation.py", line 447, in add_single_ground_truth_image_info
File "build/bdist.linux-x86_64/egg/object_detection/utils/object_detection_evaluation.py", line 527, in _update_ground_truth_statistics
ValueError: operands could not be broadcast together with shapes (0,) (10,)

[arpita-saha]

@robmorh hi, I am facing the same error. Have you able to solve this error?

[EvaKristiansen]

@robmorh am facing the same issue as well. Did you find a solution?