Models: The error(Invalid argument: Incompatible shapes) got when running eval.py on xception_65 model on deeplab

[Fengzhonglouge]

The enviorment is 16G memmory,GTX 1660TI card,windows10 system,anaconda 3,python 3.6,tensorflow-gpu-1.15.0,CUDA10.0,CUDNN7.5
Training on my own dataset "breastseg" which has already been registered at the related py file
Successfully run train.py with instructions below:
python deeplab/train.py \
--logtostderr \
--training_number_of_steps=100000 \
--train_split="train" \
--model_variant="xception_65" \
--atrous_rates=6\
--atrous_rates=12 \
--atrous_rates=18\
--output_stride=16 \
--decoder_output_stride=4 \
--train_crop_size=321,321 \
--train_batch_size=4 \
--dataset="breastseg" \
--tf_initial_checkpoint='D:/models-master/research/deeplab/backbone/deeplabv3_pascal_trainval/model.ckpt' \
--train_logdir='D:/models-master/research/deeplab/datasets/pascal_voc_seg/VOCdevkit/VOC2012/exp' \
--dataset_dir='D:/models-master/research/deeplab/datasets/pascal_voc_seg/VOCdevkit/VOC2012/tfrecord'
Then I run eval.py to check the miou, instructions:
python deeplab/eval.py \
--logtostderr \
--eval_split="val" \
--model_variant="xception_65" \
--atrous_rates=6 \
--atrous_rates=12 \
--atrous_rates=18 \
--output_stride=16 \
--decoder_output_stride=4 \
--eval_crop_size=513,513 \
--dataset="breastseg" \
--checkpoint_dir='D:/models-master/research/deeplab/datasets/pascal_voc_seg/VOCdevkit/VOC2012/exp' \
--eval_logdir='D:/models-master/research/deeplab/datasets/pascal_voc_seg/VOCdevkit/VOC2012/eval' \
--dataset_dir='D:/models-master/research/deeplab/datasets/pascal_voc_seg/VOCdevkit/VOC2012/tfrecord' \
--max_number_of_evaluations=1

And I got the error:

Traceback (most recent call last):
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
return fn(*args)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
target_list, run_metadata)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
run_metadata)

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Incompatible shapes: [263169] vs. [172261]
[[{{node false_negatives_2/Mul}}]]
[[ConstantFoldingCtrl/mean_iou/confusion_matrix/assert_less_1/Assert/AssertGuard/Switch_0/_4486]]
(1) Invalid argument: Incompatible shapes: [263169] vs. [172261]
[[{{node false_negatives_2/Mul}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "deeplab/eval.py", line 235, in
tf.app.run()
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\15122\Anaconda3\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\Users\15122\Anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "deeplab/eval.py", line 228, in main
eval_interval_secs=FLAGS.eval_interval_secs)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\contrib\training\python\training\evaluation.py", line 453, in evaluate_repeatedly
session.run(eval_ops, feed_dict)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1360, in run
raise six.reraise(original_exc_info)
File "C:\Users\15122\Anaconda3\lib\site-packages\six.py", line 703, in reraise
raise value
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1345, in run
return self._sess.run(args, *kwargs)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1418, in run
run_metadata=run_metadata)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1176, in run
return self._sess.run(args, **kwargs)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
run_metadata_ptr)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
run_metadata)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.

(0) Invalid argument: Incompatible shapes: [263169] vs. [172261]
[[node false_negatives_2/Mul (defined at C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
[[ConstantFoldingCtrl/mean_iou/confusion_matrix/assert_less_1/Assert/AssertGuard/Switch_0/_4486]]
(1) Invalid argument: Incompatible shapes: [263169] vs. [172261]
[[node false_negatives_2/Mul (defined at C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'false_negatives_2/Mul':
File "deeplab/eval.py", line 235, in
tf.app.run()
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\15122\Anaconda3\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\Users\15122\Anaconda3\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "deeplab/eval.py", line 187, in main
weights=weights)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\ops\metrics_impl.py", line 1561, in false_negatives
updates_collections)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\ops\metrics_impl.py", line 1501, in _count_condition
values = math_ops.multiply(values, weights)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
return target(args, *kwargs)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\ops\math_ops.py", line 331, in multiply
return gen_math_ops.mul(x, y, name)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\ops\gen_math_ops.py", line 6701, in mul
"Mul", x=x, y=y, name=name)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:\Users\15122\Anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()

Has anyone got the same issue please write some solutions which would be very grateful, looking forward to any help.

[Fengzhonglouge]

By the way, my images are 512*512 size

[SeanCho1996]

hi there, i have encountered the same problem, did you find any possible solutions?

[Papageno2]

this may result from array dimension inconsistence,
I have the same error,

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Incompatible shapes: [2097152] vs. [263169]
     [[node true_positives_11/LogicalAnd (defined at /home/mxue/anaconda3/envs/tf15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
     [[ConstantFoldingCtrl/mean_iou/confusion_matrix/assert_non_negative_1/assert_less_equal/Assert/AssertGuard/Switch_0/_4564]]
  (1) Invalid argument: Incompatible shapes: [2097152] vs. [263169]

but note that:

• [2097152] = [1024 * 2048]
• [263169] =[513 * 513]
  the original dataset for me is cityscapes with shape [1024, 2048,3], so I change the --eval_crop_size="1024,2048", everthing works !

try it , if it works, pls let know.

[SeanCho1996]

@Papageno2 Yes, I found this solution, too. Just need to change the crop size and make it bigger than input image size, it works well! Thx bro!