Models: InvalidArgumentError, Assign requires shapes of both tensors to match. lhs shape= [5] rhs shape= [1001]

Created on 25 Feb 2017 · 8Comments · Source: tensorflow/models

I am following the TF slim tutorial on the README, but I ran into some issues when trying to finetune Inception_v3 on the Flowers dataset.

I corrected the scopes as per the suggestions of others, since they are apparently wrong in the README.

I also downgraded my Tensorflow version to 0.11, since that gave fewer errors.

For some reason, the program is not able to reset the new size of the FC last layer, from 1000 to 5.

How do I go about correcting this?

`python train_image_classifier.py --train_dir=${TRAIN_DIR} --dataset_dir=${DATASET_DIR} --dataset_name=flowers --dataset_split_name=train --model_name=inception_v3 --checkpoint_path=${CHECKPOINT_PATH} \ --checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits --trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits

INFO:tensorflow:Fine-tuning from /tmp/checkpoints/inception_v3.ckpt
INFO:tensorflow:Error reported to Coordinator: , Assign requires shapes of both tensors to match. lhs shape= [5] rhs shape= [1001]
[[Node: save/Assign_8 = Assign[T=DT_FLOAT, _class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV3/AuxLogits/Conv2d_2b_1x1/biases, save/restore_slice_8)]]

Caused by op u'save/Assign_8', defined at:
File "train_image_classifier.py", line 585, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "train_image_classifier.py", line 575, in main
init_fn=_get_init_fn(),
File "train_image_classifier.py", line 370, in _get_init_fn
ignore_missing_vars=FLAGS.ignore_missing_vars)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 579, in assign_from_checkpoint_fn
saver = tf_saver.Saver(var_list, reshape=reshape_variables)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 986, in __init__
self.build()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1015, in build
restore_sequentially=self._restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 620, in build
restore_sequentially, reshape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 369, in _AddRestoreOps
assign_ops.append(saveable.restore(tensors, shapes))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 212, in restore
self.op.get_shape().is_fully_defined())
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 45, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [5] rhs shape= [1001]
[[Node: save/Assign_8 = Assign[T=DT_FLOAT, _class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV3/AuxLogits/Conv2d_2b_1x1/biases, save/restore_slice_8)]]

Traceback (most recent call last):
File "train_image_classifier.py", line 585, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "train_image_classifier.py", line 581, in main
sync_optimizer=optimizer if FLAGS.sync_replicas else None)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 753, in train
master, start_standard_services=False, config=session_config) as sess:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 969, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 797, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 386, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 958, in managed_session
start_standard_services=start_standard_services)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 715, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 235, in prepare_session
init_fn(sess)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 581, in callback
saver.restore(session, model_path)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1345, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [5] rhs shape= [1001]
[[Node: save/Assign_8 = Assign[T=DT_FLOAT, _class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV3/AuxLogits/Conv2d_2b_1x1/biases, save/restore_slice_8)]]

Source

nmehandru

Most helpful comment

just pay attention it is pointing to saver.py
which means somewhere in your directory you have some files from already trained model(or incomplete trained model i would say)
Go and just remove those files and try to run again....
it should work..

hrishi-aware on 31 Mar 2018

👍22 🎉4

All 8 comments

@nmehandru This looks like a duplicate of https://github.com/tensorflow/tensorflow/issues/7838

Please follow-up on that issue.

Also note that the following query for similar issues may help:
https://github.com/issues?utf8=%E2%9C%93&q=is%3Aissue+repo%3Atensorflow%2Fmodels+repo%3Atensorflow%2Ftensorflow+%22Assign+requires+shapes+of+both+tensors+to+match%22

tatatodd on 1 Mar 2017

hello sir, When i tried below line in Ubuntu 16.04 LTS command prompt then i found the respective error:

Code:
python guess.py --model_type inception --model_dir /home/mitul/rude-carnie/checkpoints/gender/inception/21936 --filename /home/mitul/Downloads/faceimages/jika.jpg

Error:
Executing on /cpu:0
selected (fine-tuning) inception model
/home/mitul/rude-carnie/checkpoints/gender/inception/21936/checkpoint-14999
Traceback (most recent call last):
File "guess.py", line 207, in
tf.app.run()
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "guess.py", line 166, in main
saver.restore(sess, model_checkpoint_path)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1388, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [8] rhs shape= [2]
[[Node: save/Assign_376 = Assign[T=DT_FLOAT, _class=["loc:@output/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](output/biases, save/RestoreV2_376)]]

Caused by op u'save/Assign_376', defined at:
File "guess.py", line 207, in
tf.app.run()
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "guess.py", line 165, in main
saver = tf.train.Saver()
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1000, in __init__
self.build()
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1030, in build
restore_sequentially=self._restore_sequentially)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 624, in build
restore_sequentially, reshape)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 373, in _AddRestoreOps
assign_ops.append(saveable.restore(tensors, shapes))
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 130, in restore
self.op.get_shape().is_fully_defined())
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
use_locking=use_locking, name=name)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/mitul/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [8] rhs shape= [2]
[[Node: save/Assign_376 = Assign[T=DT_FLOAT, _class=["loc:@output/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](output/biases, save/RestoreV2_376)]]

please help me to solve this type of errors.
thank you !!

mitulgiri on 19 Dec 2017

hrishi-aware on 31 Mar 2018

👍22 🎉4

@hrishi-aware wonderful solution, solved my problem

CRonaldo1997 on 21 Aug 2018

@hrishi-aware very thanks for your solution!

IamHimon on 30 Oct 2018

@hrishi-aware Hi! I'm kind of getting crazy because of this error, could you explain me better what do you mean about "incomplete trained model"? I have fine tuned the inception-resnet V2 network with my own dataset but when I try to to use my new checkpoint for extracting feature-layers it does not run (the network cannot be built up) cause of this error

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [5,5,128,768] rhs shape= [1,1,128,768]
[[node save/Assign_7 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py:748) = Assign[T=DT_FLOAT, _class=["loc:@InceptionResnetV2/AuxLogits/Conv2d_2a_5x5/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](InceptionResnetV2/AuxLogits/Conv2d_2a_5x5/weights, save/RestoreV2:7)]]

NB. I have not any other checkpoints in the directory to which I point

michirup on 30 Nov 2018

just pay attention it is pointing to saver.py
which means somewhere in your directory you have some files from already trained model(or incomplete trained model i would say)
Go and just remove those files and try to run again....
it should work..

how can I find those files?