Facenet: compare.py

[kaishijeng]

David

Can compare.py still work with inception_resnet_v1 model?

Thanks,

[davidsandberg]

The compare module has now been fixed to work with inception_resnet_v1, and also to use MTCNN for alignment.

[kaishijeng]

Thanks for the updated compare.py. It works, but results are not very good. I use your pre-trained model 20161030-023650 and did the following tests:
1)
Anthony_Hopkins_0001.jpg vs Anthony_Hopkins_0002.jpg : Distance between the embeddings: 0.003904
2)
clapton-1.jpg vs carell.jpg (images from openface): Distance between the embeddings: 0.005763
3)
clapton-1.jpg vs lennon-1.jpg (images from openface): Distance between the embeddings: 0.004337
4)
clapton-1.jpg vs clapton-2.jpg (images from openface): Distance between the embeddings: 0.004697 :

I will expect distance for different people should be much bigger than the same people. But I don't see much difference of the distance for different people vs same people. Is this what we expect?
How can compare.py be used for people identification?

Thanks

[xl94]

I am also testing compare.py and I guess i could post my question here too. I encounter some issues with it when i run with this command

python compare.py ~/models/facenet/20161030-023650 model-20161030-023650.meta model-20161030-023650.ckpt-80000 ../data/images/Anthony_Hopkins_0001.jpg ../data/images/Anthony_Hopkins_0002.jpg.

Return:

Creating networks and loading parameters
Loading model "model-20161030-023650.meta"
Traceback (most recent call last):
File "compare.py", line 112, in
main(parse_arguments(sys.argv[1:]))
File "compare.py", line 47, in main
facenet.load_model(args.model_dir, args.meta_file, args.ckpt_file)
File "/home/x/Desktop/facenet/src/facenet.py", line 469, in load_model
saver = tf.train.import_meta_graph(os.path.join(model_dir_exp, meta_file))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1693, in import_meta_graph
read_meta_graph_file(meta_graph_or_file), clear_devices)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1580, in _import_meta_graph_def
input_graph_def, name="", producer_op_list=producer_op_list)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 252, in import_graph_def
op_def = op_dict[node.op]
KeyError: u'RsqrtGrad'

Able to get help? Thanks!

[davidsandberg]

@kaishijeng I fixed one problem with compare.py. The caluclation of the distance should be

dist = np.sqrt(np.sum(np.square(np.subtract(emb[0,:], emb[1,:]))))

instead of

dist = np.sqrt(np.mean(np.square(np.subtract(emb[0,:], emb[1,:]))))

With this fix the distance values becomes similar to the distances when running validate_on_lfw.py, but not exactly the same. The reason for the difference is probably that batch normalization is working in training mode also during evaluation, so #56 needs to be solved to get results that are consistent with validate_on_lfw.
@xl94 This problem has to do with the restoring the model, and is hence not so much related to the compare tool. It has previously been reported in #58 and #64, but I'm not sure if it was ever solved. Looks like something related to tensorflow version or similar.

[kaishijeng]

David

After syncing up with the latest code from Github, now compare.py doesn't work anymore.
System has the following error:

Traceback (most recent call last):
File "./compare.py", line 112, in
main(parse_arguments(sys.argv[1:]))
File "./compare.py", line 51, in main
phase_train_placeholder = tf.get_default_graph().get_tensor_by_name("phase_train:0")
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2607, in get_tensor_by_name
return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2458, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2500, in _as_graph_element_locked
"graph." % (repr(name), repr(op_name)))
KeyError: "The name 'phase_train:0' refers to a Tensor which does not exist. The operation, 'phase_train', does not exist in the graph."

Any idea how to fix?

Thanks

[davidsandberg]

@kaishijeng The way training and evaluation is done has been changed, so there is a new model to accommodate for this. The new model (20161116-234200) can be downloaded from the wiki.

[dleonsanchez]

I have downloaded the new model and the problem was still there (The name 'phase_train:0' refers to a Tensor which does not exist), so I have commented the following line:

phase_train_placeholder = tf.get_default_graph().get_tensor_by_name("phase_train:0")

and changed this line:
feed_dict = { images_placeholder: images, phase_train_placeholder: False }

as:
feed_dict = { images_placeholder: images}

And it works

[simplysimleprob]

Hi David,

Squared L2 distance for comparing two image embeddings is used in OpenFace and mentioned in FaceNet. I think removing np.sqrt might be equal to squred L2 distance.

dist = np.sum(np.square(np.subtract(emb[0,:], emb[1,:])))

Thank you.

[xl94]

Able to load the model before running the session? Because compare.py takes really long to compute as a whole. Will computing the 2nd, 3rd, 4th pair of images be faster?

[simplysimleprob]

@xl94 compare.py takes a long time due to loading the pretrained FaceNet model (one time thing). After the model is loaded, computing many different images at once or interactively isn't bad at all.
Here is my sample code for doing both MTCNN alignment and FaceNet embedding from live webcam streams. (1) one-time loading both the models (2) find all BBs/landmarks per frame (3) draw each bb and associated 5-pts landmarks (4) compute embeddings of all BBs and calculate the distance between BBs.

with tf.Graph().as_default():

    with tf.Session() as sess:

        # Load MTCNN alignment model
        minsize = 20 # minimum size of face
        threshold = [ 0.6, 0.7, 0.7 ]  # three steps's threshold
        factor = 0.709 # scale factor
        pnet, rnet, onet = align.detect_face.create_mtcnn(sess, '../data/')

        # Load the FaceNet model
        print('Loading FaceNet model')
        model_dir = '../models/20161116-234200'
        meta_file = 'model-20161116-234200.meta'
        ckpt_file = 'model-20161116-234200.ckpt-80000'       
        facenet.load_model(model_dir, meta_file, ckpt_file)

        # Get input and output tensors
        images_placeholder = tf.get_default_graph().get_tensor_by_name("input:0")
        embeddings = tf.get_default_graph().get_tensor_by_name("embeddings:0")

        while(True):       
            ret, frame = video_capture.read()
            bbs, landmarks = align.detect_face.detect_face(frame, minsize, pnet, rnet, onet, threshold, factor)

            img_list = [None] * len(bbs)
            for i, bbox in enumerate(bbs):
                det = bbox[0:4]
                bb = np.zeros(4, dtype=np.int32)
                bb[0] = np.maximum(det[0]-margin/2, 0)
                bb[1] = np.maximum(det[1]-margin/2, 0)
                bb[2] = np.minimum(det[2]+margin/2, img_width)
                bb[3] = np.minimum(det[3]+margin/2, img_height)
                cropped = frame[bb[1]:bb[3],bb[0]:bb[2],:]
                aligned = misc.imresize(cropped, (image_size, image_size), interp='bilinear')
                prewhitened = facenet.prewhiten(aligned)
                in_image = np.expand_dims(prewhitened, axis=0)
                img_list[i] = prewhitened

                # draw bbx and landmarks 
                #color_cv = (0,255,0)
                color_cv = set_color(i)
                lt = (int(bbox[0]), int(bbox[1])) # (x1, y1)
                br = (int(bbox[2]), int(bbox[3])) # (x2, y2)
                cv2.rectangle(frame, lt, br, color=color_cv, thickness=3)

                marks = landmarks[:, i]
                numLandmarks = len(marks)/2
                for idx in range(numLandmarks):  
                    pos = (marks[idx], marks[idx+numLandmarks])  
                    cv2.circle(frame, pos, 2, color=color_cv)  

            # get embeddings
            images = np.stack(img_list)
            feed_dict = {images_placeholder: images}
            emb = sess.run(embeddings, feed_dict=feed_dict)

            for i in range(len(bbs)):
                for j in range(len(bbs)):
                    if not (i == j):
                        dist = np.sum(np.square(np.subtract(emb[i,:], emb[j,:])))
                        print ('(%d,%d): %1.4f ' % (i, j, dist))

[xl94]

@simplysimleprob wow. Great, that is what I been trying to work on. Thanks! Ill try it out now.

[kaishijeng]

Dave

i have 2 questions of my own trained model with inception_resnet_v1 using tensorflow 0.11.0rc2.
1) The saved ckpt has 2 files, data and index instead of one of your pretrained model
model-20161118-182010.ckpt-632000.data-00000-of-00001
model-20161118-182010.ckpt-632000.index
2) Size of model-20161118-182010.ckpt-632000.data-00000-of-00001 is 2.5GB which is much bigger than your trained model, model-20161116-234200.ckpt-80000 which 600MB.

Do you know why there are such differences?

Thanks,

[davidsandberg]

@kaishijeng I guess you are seeing the new checkpoint format.
This is from the release notes for r0.12:
New checkpoint format becomes the default in tf.train.Saver. Old V1 checkpoints continue to be readable; controlled by the write_version argument, tf.train.Saver now by default writes out in the new V2 format. It significantly reduces the peak memory required and latency incurred during restore.
Sounds like it's possible to save using the old format as well, so that is something you can try to see if the files gets smaller.

[kaishijeng]

David

Old V1 solves the format issue, but datasize of the saved weight is still bigger than 2GB which is much bigger than your pretrained weight

Thanks

[jhj7905]

@simplysimleprob can u provide whole source code related to webcam version?

[Zumbalamambo]

@simplysimleprob how do i use the protobuf file instead of the checkpoint file?

[JiaoPaner]

David
When I run the compare.py it spents too much time at this step：
Model directory: 20180402-114759
Metagraph file: model-20180402-114759.meta
Checkpoint file: model-20180402-114759.ckpt-275
...
The result will come out in a minute,it is too long
Is this time correct? What is the general running time?
Or is there a way to optimize it? my tensorflow is the CPU version.
thanks!

[modanesh]

@JiaoPaner That part is about loading the model which normally takes more time, even on a GPU. Also, since you're running on the CPU, it takes even more time and it's ok.

[test4fest]

David,

Thank you very much for this great work.
Does cosine distance in compare.py compatible with 20180402-114759 pre-trained model?
Based on metric loss, does it try to project all the identities into non-overlapping balls? if yes, what's the value of radius?

Thanks