Facenet: How can i fine tuning the model?

[BKZero]

i am using the model model-20170511-185253.ckpt-80000.data-00000-of-00001 to fine tuning on my own dataset to improve the accuracy. but i found the code doesn't using the slim of tensorflow so it is not that convenient and i should write the code to exclude the Logits scope by hand.
i found the issue #70 (https://github.com/davidsandberg/facenet/issues/70)
and follow what the asker did:
if args.checkpoint_exclude_scopes:
exclusions = [scope.strip()
for scope in args.checkpoint_exclude_scopes.split(',')]
else:
exclusions = ["InceptionResnetV1/Logits"]
restore_vars = []
update_gradient_vars = []
if args.pretrained_model:
for var in tf.all_variables():
excluded = False
for exclusion in exclusions:
# skip this variable
if var.op.name.startswith(exclusion):
excluded = True
if not excluded:
print(var.op.name)
if not 'Embeddings/' in var.op.name:
restore_vars.append(var)
else:
update_gradient_vars.append(var)

    else:
        restore_vars = tf.all_variables()
        update_gradient_vars = tf.all_variables()

and modified the code to restore the ckpt file:
saver = tf.train.Saver(restore_vars, max_to_keep=3)
# saver = tf.train.Saver(tf.trainable_variables(), max_to_keep=3)

but it does not work. i only found many error log about Not found: Key centers not found in checkpoint something like:

W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/framework/op_kernel.cc:993] Not found: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"]]
Traceback (most recent call last):
File "src/train_softmax.py", line 483, in
main(parse_arguments(sys.argv[1:]))
File "src/train_softmax.py", line 243, in main
saver.restore(sess, pretrained_model)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1428, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_493/tensor_names, save/RestoreV2_493/shape_and_slices)]]
[[Node: save/Assign_492/_1148 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_4449_save/Assign_492", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_493', defined at:
File "src/train_softmax.py", line 483, in
main(parse_arguments(sys.argv[1:]))
File "src/train_softmax.py", line 224, in main
saver = tf.train.Saver(restore_vars, max_to_keep=3)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1040, in __init__
self.build()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1070, in build
restore_sequentially=self._restore_sequentially)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 675, in build
restore_sequentially, reshape)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 402, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 242, in restore_op
[spec.tensor.dtype])[0])
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 668, in restore_v2
dtypes=dtypes, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in __init__
self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key centers not found in checkpoint
[[Node: save/RestoreV2_493 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_493/tensor_names, save/RestoreV2_493/shape_and_slices)]]
[[Node: save/Assign_492/_1148 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_4449_save/Assign_492", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

what should i do? can anyone help?

[billtiger]

I have the same problem!Have you solved it? @BKZero

[pzw520125]

I have the same problem!Have you solved it?

[yanmenglu]

@davidsandberg ,I have the same problem ,too. could you tell me how to solve it .

[deeplearninggogogo]

,I have the same problem ,too. could you tell me how to solve it .

[LaviLiu]

I have the same problem, i change the saver.restore(sess, pretrained_model) to ckpt = tf.train.get_checkpoint_state(pretrained_model) if ckpt and ckpt.model_checkpoint_path: saver.restore(sess, ckpt.model_checkpoint_path)

[Mahsa1994]

Check your parameters which should be compatible with the given pre-trained model such as:
--model_def, --pretrained_model ./model/model-xxx.ckpt-xxxx, ...
Also, changing --gpu_memory_fraction can help to solve GPU related errors.