Facenet: Is there only one variable named "centers" in the model?

[LiuzcEECS]

@davidsandberg

[davidsandberg]

In the center_loss function there is one tensor named centers that has the dimension nrof_features, i.e. one center per feature.

[LiuzcEECS]

I've a little confused. In "A Discriminative Feature Learning Approach for Deep Face Recognition", Wen et al. say "In each iteration, the centers are computed by averaging the features of the corresponding classes". So in my opinion, there should be one center for every class, instead of every feature. And in "3.3 Discussion", when they visualize the distribution of features on MNIST Dataset, they use 10 centers as 10 average values of 10 digits. I think there must be something wrong with my understanding. Would you please answer my question?

[melgor]

This is very strange for me too. I think that 'centers' should have dimmension (as in reference implementation).

The idea is that each class have his own centers and the CenterLoss make example of same class similar to other.
When we have only one 'center', it look like that we want that each class should we similar to each other.

But maybe this is because of not understanding the code of TensorFlow.

[davidsandberg]

@melgor yes, you're absolutely right. It can be seen as each class having one center, and each center having the dimension nrof_features.
I have started training using a new center loss based on some code found here.
Will see how it works, but it will probably require some parameter tuning.

[davidsandberg]

The new center loss looks like this:
def center_loss_new(logits, label, alfa, nrof_classes):
nrof_features = logits.get_shape()[1]
centers = tf.get_variable('centers', [nrof_classes, nrof_features], dtype=tf.float32,
initializer=tf.constant_initializer(0), trainable=False)
label = tf.reshape(label, [-1])
logits_batch = tf.gather(centers, label)
diff = (1 - alfa) * (logits_batch - logits)
centers = tf.scatter_sub(centers, label, diff)
centers_batch = tf.gather(centers, label)
loss = tf.nn.l2_loss(logits - centers_batch)
return loss, centers

[melgor]

Three more things about CenterLoss, based on Caffe implementation:

1. You should init the center using xavier or any other random initialization.
2. Paper based in CenterLoss learn the center as you implemented. But reference Caffe implementation do it differently. It is treated like a normal trainable variable (so using SGD) and it works. It could make code easier (link)
3. I saw that you are using very small weight for CenterLoss. Based on paper and implementation, it should be <0.001, 0.008> (and it may depend on nrof_features and number of classes). You used much smaller (1e-5) (but you still get very good results > 98%, which is great).

[LiuzcEECS]

@davidsandberg Hello, have you started training with the new center loss? Would you please share your parameter values such as learning_rate, decov_loss_factor and center_loss_factor with me?

[davidsandberg]

I did some training, but sadly it didn't improve the performance compared to using the old center loss. The parameter settings that gave the best performance was
weight_decay = 2e-4
center_loss = 2e-4
Will check the center loss implementation a bit more when I get some time to spare.

[davidsandberg]

The new implementation looks ok and there is a unit test (https://github.com/davidsandberg/facenet/blob/master/test/center_loss_test.py) to verify it. The new implementation has been committed to master.