Facenet: KeyError occurs when importing the meta file

[simplysimleprob]

Hi David,

First of all, thank you for sharing your great work.
I have a problem importing the meta file.
(I use Tensorflow 0.10 in Ubuntu 14.04 LTS, with CUDA 7.5 and CUDNN 5. )

when running validate_on_lfw.py, the below error is thrown.

Model directory: ../models/20161030-023650
Metagraph file: model-20161030-023650.meta
Checkpoint file: model-20161030-023650.ckpt-80000
Traceback (most recent call last):
File "validate_on_lfw.py", line 89, in
main(parse_arguments(sys.argv[1:]))
File "validate_on_lfw.py", line 57, in main
facenet.load_model(args.model_dir, meta_file, ckpt_file)
File "/data/deep_learning/acl_work/facenet/src/facenet.py", line 469, in load_model
saver = tf.train.import_meta_graph(os.path.join(model_dir_exp, meta_file))
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1458, in import_meta_graph
return _import_meta_graph_def(read_meta_graph_file(meta_graph_or_file))
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1348, in _import_meta_graph_def
producer_op_list=producer_op_list)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 252, in import_graph_def
op_def = op_dict[node.op]
KeyError: u'RsqrtGrad'

when running validate_on_lfw_new.py w/ your exported pb file, it also threw the below error.

E tensorflow/core/common_runtime/executor.cc:334] Executor failed to create kernel. Invalid argument: NodeDef mentions attr 'out_type' not in Op output:int32; attr=T:type>; NodeDef: import/InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/moments/sufficient_statistics/Shape = ShapeT=DT_FLOAT, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"
[[Node: import/InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/moments/sufficient_statistics/Shape = ShapeT=DT_FLOAT, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]
Traceback (most recent call last):
File "validate_on_lfw_new.py", line 87, in
main(parse_arguments(sys.argv[1:]))
File "validate_on_lfw_new.py", line 61, in main
images_placeholder, phase_train_placeholder, embeddings, nrof_folds=args.lfw_nrof_folds)
File "/data/deep_learning/acl_work/facenet/src/lfw.py", line 51, in validate
emb_array[start_index:end_index,:] = sess.run(embeddings, feed_dict=feed_dict)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 710, in run
run_metadata_ptr)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 908, in run
feed_dict_string, options, run_metadata)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 958, in do_run
target_list, options, run_metadata)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: NodeDef mentions attr 'out_type' not in Op output:int32; attr=T:type>; NodeDef: import/InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/moments/sufficient_statistics/Shape = ShapeT=DT_FLOAT, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"
[[Node: import/InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/moments/sufficient_statistics/Shape = ShapeT=DT_FLOAT, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"]]
Caused by op u'import/InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/moments/sufficient_statistics/Shape', defined at:
File "validate_on_lfw_new.py", line 87, in
main(parse_arguments(sys.argv[1:]))
File "validate_on_lfw_new.py", line 50, in main
return_elements=return_elements, name='import')
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 279, in import_graph_def
op_def=op_def)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/misunpar/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1239, in __init
self._traceback = _extract_stack()

Can you please help me on this issue?
Thank you so much.

[xl94]

Check TF version

print (tf.version)
Use the latest 0.11 version

[simplysimleprob]

@xl94 Thank you. I will try with tensorflow 0.11 version.

Yes, after I updated it to (Tensorflow 0.11 in Ubuntu 14.04 LTS, with CUDA 8.0 and CUDNN 5.1), it can load the model. Thank you.

But, I notice that the loading takes quite a time. Is it normal? Or it depends on my hardware?
Please share your thoughts on this. Thank you.

[davidsandberg]

@simplysimleprob The ResNet-type models have a lot of layers so I think it's normal that it takes some time to load it.

[simplysimleprob]

@davidsandberg Thank you so much :) I close this issue.

[limtete]

@davidsandberg I met the same probelm with different value as " KeyError: u'VariableV2' " when i was trying to test lfw sets with pretrained Inception-ResNet-v1 model. My tensorflow's version is 0.12.1.

Does the version compatiblity cause the problem?
Thank you~