Facenet: The pretrained model is not restored properly to continue training (facenet_train.py)

[scotthong]

It seems that restoring the pretrained model is not working properly.
Here is how to reproduce the issue.

1. Use facenet_train.py to train for one epoch.
2. Use facenet_train.py and add "--pretrained_model" and point to the check point saved from the previous step.

Here are the observations that can support this assessment.

1. The training won't converge.
2. The training speed (step time) suddenly gets 3x performance boost.

I will try to compare the graph using tensorboard and update the information here.

Thanks!

[rajiv235]

@scotthong. How are you able to restore pretrained model? When I try to specify it with --pretrained_model I get
W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ../models/20170116-225008/model-20170116-225008.ckpt-160000.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
W tensorflow/core/framework/op_kernel.cc:993] Data loss: Unable to open table file ../models/20170116-225008/model-20170116-225008.ckpt-160000.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

I am using tensorflow0.12. Thanks any help would be appreciated

[ugtony]

hi @rajiv235 ,
Did you set the pretrained model path correctly?
In your case, you should set the path as "--pretrained_model ../models/20170116-225008/model-20170116-225008.ckpt-160000"

[davidsandberg]

Closing this due to inactivity. Reopen if needed.