Facenet: can't train with pretrained_model

[cttsp]

I try to train with pretrained_model (tensorflow: v0.12, model: https://drive.google.com/file/d/0B5MzpY9kBtDVZEV3QmVmeGpSVWc/view)
And some error like this:
W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ~/models/20170117-215115: Failed precondition: ~/models/20170117-215115: perhaps your file is in a different file format and you need to use a different restore operator?

What should I do?

[cttsp]

When I change code like this:
if pretrained_model:
print('Restoring pretrained model: %s' % pretrained_model)
#saver.restore(sess, pretrained_model)
meta_file, ckpt_file = facenet.get_model_filenames(os.path.expanduser(pretrained_model))
facenet.load_model(pretrained_model, meta_file, ckpt_file)

I have new problem:
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [29868] rhs shape= [43331]
[[Node: save/Assign_264 = Assign[T=DT_FLOAT, _class=["loc:@Logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Logits/biases, save/RestoreV2_264/_115)]]

[davidsandberg]

Hi,
What did you set pretrained_model to?
With the new restore function you should remove the last part of the filename, so it should be something like
model-20170128-195958.ckpt-80000
For your second problem it's likely that your dataset that you are training on has another number of classes than the pretrained model. You can replace the final classification layer and finetune that one with your dataset, but that requires some python/tensorflow hacking.

[cttsp]

I changed to ~/models/20170117-215115/model-20170117-215115.ckpt-285000
and 1st problem changed to 2nd problem.

[davidsandberg]

Closing this. Reopen if needed.

[YuliyaLi]

I had the same problems, and solved by changing the code
" saver = tf.train.Saver(tf.trainable_variables(), max_to_keep=3) "
as follows:
all_vars = tf.trainable_variables()
var_to_restore = [v for v in all_vars if not v.name.startswith('Logits')]
saver = tf.train.Saver(var_to_restore)

which gets a list of all of the variables in the model and filter out the variables of the last layer.

[tengerye]

Hi,
I got the same problem here. Have you fixed the problem please? @davidsandberg @cttsp

I have tried to rename, but the problem stays the same.

[ronyuzhang]

@YuliyaLi hi，how to operate it when you need save finetuned model？

[YuliyaLi]

@ronyuzhang using the common method:
saver.save(sess, checkpoint_path, global_step=step, write_meta_graph=False)

[tjusxh]

@YuliyaLi
Thanks for your suggestion. But I still have error by changing the code according to you.
But I solved the question as following. I don't sure what's the reason? I guess the scope causes the question.
all_vars = tf.trainable_variables()
var_to_restore = [v for v in all_vars if not v.name.startswith('InceptionResnetV1/Bottleneck')]
saver = tf.train.Saver(var_to_restore)

[Victoria2333]

@YuliyaLi, I tried to finetune my dataset on david's pretrained model to get higher accuracy, but the resut turns out to be bad.
I want to, for exmaple, freeze the old weights of L-2 layer and restore the weights of remaining 2 layer and finetune the last 2 layers. I don't mean to drop some variables which could lower the accuracy, so how can i finetune??
But as far as i know, var_to_restore = [v for v in all_vars if not v.name.startswith('Logits')] ,this code will drop some varaibles, I want to save all variables and just to fintune some weights of these varibles, how should I do???

[YuliyaLi]

@tjusxh yeah,maybe you need to check the layer name

[YuliyaLi]

@Victoria2333 maybe this post can help you.

[Victoria2333]

@YuliyaLi, Many thanks!!! I'll try it.
And has you ever tried to train with the pretrained model on the original CASIA-WebFace dataset??? I think maybe there is something wrong with the code...
I run:
python src/train_softmax.py --data_dir /home/han/facenet/CASIA-WebFace_align --image_size 160 __embedding_size 512 /home/han/facenet/src/pretrained_models/20180408-102900/model-2018408-102900.ckpt-90
Theoretically, the accuracy of pretrained model trained 90 epoch should be nearly 0, but when I trained on the pretrained model on david's dataset-CASIA-WebFace, the loss is 1.198.

[Victoria2333]

@YuliyaLi, I've tried your method again which enhances the classifier accuracy, I still can't understand why filter out some variables rather than reserve them and fintune weights will do results better??
Could you explain it??

[phoebushe]

@Victoria2333 hello.I have the same problem. My accuracy is 0... Did you solve this problem? Thx~

[Victoria2333]

@phoebushe, my accuracy is fine, just confused about the loss problem...

[viethungtsdv]

hi @davidsandberg @YuliyaLi
I have just finetune with the asian face datasets use train_tripletloss.py and pretrained model '20180402-114759' (set --pretrained_model [...]/20180402-114759/model-20180402-114759.ckpt-275). However I meet an issue : "InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [128] rhs shape= [512]"

Can you please help me fix it?

Thanks you so much.

[phoebushe]

@viethungtsdv
you can try this:
'all_vars = tf.trainable_variables()
var_to_restore = [v for v in all_vars if not v.name.startswith('Logits')]
saver = tf.train.Saver(var_to_restore)'

[viethungtsdv]

hi @phoebushe
Thanks for your suggestion.
I try to do it but not solve the problem. I want to finetune with tripless loss method, not classification method.

This is error message:
"InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [128] rhs shape= [512]
[[Node: save/Assign_18 = Assign[T=DT_FLOAT, _class=["loc:@InceptionResnetV1/Bottleneck/BatchNorm/beta"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](InceptionResnetV1/Bottleneck/BatchNorm/beta, save/RestoreV2/_1633)]]"

[TianLei15822591582]

@viethungtsdv

759

[viethungtsdv]

Hi all,
I want to finetune model with triplet loss and embedding_size = 128D. Where can I download pre-trained model weight with 128-D size of embedding? (Currently, two pre-trained models public are 512D)