Models: Not able to convert checkpoint models to Tensorflow graph proto

Created on 3 Sep 2018 · 12Comments · Source: tensorflow/models

System Information:
OS Platform and Distribution: Windows 10
Tensorflow version: 1.10.0
TensorFlow installed from: Installed through pip.
Bazel version: As I installed through pip and not binary, I cannot check the bazel version (?)
GPU model and memory: I do not have a GPU on my system.
CUDA/cUDNN version: Not being used

I created my own dataset of ~100 images and 1 class and ran the legacy/train.py script for the training.
I got the below checkpoint files:
model.ckpt-649.data-00000-of-00001
model.ckpt-649.index
model.ckpt-649.meta

However, I get the below error when I run the export_inference_graph.py file.
No custom code was written.

Exact commands to reproduce the error:
python .\object_detection\export_inference_graph.py --input_type image_tensor --pipeline_config_path .\object_detection\training\ssd_mobilenet_v1_pets.config --trained_checkpoint_prefix .\object_detection\training\model.ckpt-649.index --output_directory .\object_detection\trained-inference-graphs

Top level directory (as mentioned above already): object_detection

I ultimately want to use the .pb file to evaluate the model using the object_detection_tutorial notebook provided.

Logs:
Traceback (most recent call last):
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call
return fn(*args)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Tensor name "BoxPredictor_0/BoxEncodingPredictor/biases" not found in checkpoint fil
es .\object_detection\training\model.ckpt-649.index
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT
_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2
/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py", line 1725, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 877, in run
run_metadata_ptr)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run
run_metadata)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Tensor name "BoxPredictor_0/BoxEncodingPredictor/biases" not found in checkpoint fil
es .\object_detection\training\model.ckpt-649.index
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT
_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2
/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
File ".\object_detection\export_inference_graph.py", line 153, in
tf.app.run()
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File ".\object_detection\export_inference_graph.py", line 149, in main
write_inference_graph=FLAGS.write_inference_graph)
File "C:\Anaconda3\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 405, in export_inference_graph
write_inference_graph=write_inference_graph)
File "C:\Anaconda3\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 334, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "C:\Anaconda3\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 241, in write_graph_and_checkpoint
tf.import_graph_def(inference_graph_def, name='')
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func
return func(args, *kwargs)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\frameworkimporter.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\frameworkimporter.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\framework\ops.py", line 3289, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\framework\ops.py", line 3289, in
for c_op in c_api_util.new_tf_operations(self)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\framework\ops.py", line 3180, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Tensor name "BoxPredictor_0/BoxEncodingPredictor/biases" not found in checkpoint files .\object_de
tection\training\model.ckpt-649.index
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT
_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2
/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py", line 1737, in restore
checkpointable.OBJECT_GRAPH_PROTO_KEY)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 348, in get_tensor
status)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\framework\errors_impl.py", line 519, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File ".\object_detection\export_inference_graph.py", line 153, in
tf.app.run()
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File ".\object_detection\export_inference_graph.py", line 149, in main
write_inference_graph=FLAGS.write_inference_graph)
File "C:\Anaconda3\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 405, in export_inference_graph
write_inference_graph=write_inference_graph)
File "C:\Anaconda3\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 334, in _export_inference_graph
trained_checkpoint_prefix=checkpoint_to_use)
File "C:\Anaconda3\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 245, in write_graph_and_checkpoint
saver.restore(sess, trained_checkpoint_prefix)
File "C:\Anaconda3\Lib\site-packages\tensorflow\python\training\saver.py", line 1743, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or othe
r graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Origin
al error:

Tensor name "BoxPredictor_0/BoxEncodingPredictor/biases" not found in checkpoint files .\object_detection\training\model.ckpt-649.index
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT
_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2
/shape_and_slices)]]

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key th
at is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Source

pranali3215

Most helpful comment

Had the same issue, but realised I was using the wrong pipeline_config_path - make sure the config is for the correct model you're converting!

basedrhys on 16 Jan 2020

🎉1 👎1 👍1

All 12 comments

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

tensorflowbutler on 3 Sep 2018

I have updated the post.

pranali3215 on 3 Sep 2018

@pranali3215 i guess --trained_checkpoint_prefix .\object_detection\training\ and not \object_detection\training\model.ckpt-649.index .Where you able to solve it?

zishanahmed08 on 18 Sep 2018

No. Haven't solved it yet.

pranali3215 on 24 Sep 2018

I have come across the same problem yet.

ivorytowerkq on 26 Oct 2018

Somehow adding tf.reset_default_graph() before restoring solves my problem.

Navarro33 on 10 Jan 2019

Somehow adding tf.reset_default_graph() before restoring solves my problem.

Where did you add this line?

morgangallant on 11 Jan 2019

Somehow adding tf.reset_default_graph() before restoring solves my problem.

Where did you add this line?

Here is part of my code:

saver = tf.train.Saver()

with tf.Session() as sess:
sess.run(init)

save_path = saver.save(sess, "saved/save_1.ckpt")
print("Save to path: ", save_path)

tf.reset_default_graph()

Restore Variables

W = tf.Variable(np.arange(6).reshape(2,3), dtype= tf.float32, name= "weights")
b = tf.Variable(np.arange(3).reshape(1,3), dtype= tf.float32, name= "biases")

saver = tf.train.Saver()
with tf.Session() as sess:
saver.restore(sess, "saved/save_1.ckpt")
print("weights: ")
print(sess.run(W))
print("biases: ")
print(sess.run(b))

Navarro33 on 11 Jan 2019

I have got the same issue! Please resolve!