Facenet: Why do centers get updated?

Created on 24 Jul 2017 · 9Comments · Source: davidsandberg/facenet

Hi,
I don't quite understand why the centers could get updated in David's implementation. Could anyone please explain the mechanism behind it?
My question is:

In facenet.center_loss(...) implementation, the centers are updated by a scatter_sub OP and the resulting tensor is returned:

centers = tf.scatter_sub(centers, label, diff)
loss = tf.nn.l2_loss(features - centers_batch)
return loss, centers

However, the returned tensor(centers) is not used later in train_softmax.py:

prelogits_center_loss, _ = facenet.center_loss(prelogits, ...)

So why the centers could get updated?
Thanks.

Source

ugtony

👍2

Most helpful comment

Hi @ugtony, @JianbangZ,
Thanks for spotting this! I guess the running of the center_op got dropped at some point. I added a control dependency in the center loss function on a local branch:

    with tf.control_dependencies([centers]):
        loss = tf.reduce_mean(tf.square(features - centers_batch))

I 'm currently training on casia with wd=2e-4 and cl=2e-2 and maybe the performance is slightly better after 37k iterations but it's too early to tell.
Please also note that the scaling of the center loss was changed some time ago, so the center loss factor should be multiplied by the batch size.

davidsandberg on 1 Aug 2017

👍3

All 9 comments

@ugtony
I have been (1) Epoch: [0][41/1000] Epoch: [0][51/1000] Epoch: [0][61/1000] Epoch: [0][71/1000] Epoch: [0][81/1000] Epoch: [0][91/1000] Epoch: [0][101/1000] Epoch: [0][111/1000] Epoch: [0][121/1000] Epoch: [0][131/1000] Epoch: [0][141/1000] Epoch: [0][151/1000] Epoch: [0][161/1000] Epoch: [0][171/1000] Epoch: [0][181/1000] Epoch: [0][191/1000] Epoch: [0][201/1000] Epoch: [0][211/1000] Epoch: [0][221/1000] Epoch: [0][231/1000] Epoch: [0][241/1000] Epoch: [0][251/1000] Epoch: [0][261/1000] Epoch: [0][271/1000] Epoch: [0][281/1000] Epoch: [0][291/1000] Epoch: [0][301/1000] Epoch: [0][311/1000] Epoch: [0][321/1000] Epoch: [0][331/1000] Epoch: [0][341/1000] Epoch: [0][351/1000] Epoch: [0][361/1000] Epoch: [0][371/1000] Epoch: [0][381/1000] Epoch: [0][391/1000] Epoch: [0][401/1000] Epoch: [0][411/1000] Epoch: [0][421/1000] Epoch: [0][431/1000] Epoch: [0][441/1000] Epoch: [0][451/1000] Epoch: [0][461/1000] Epoch: [0][471/1000] Epoch: [0][481/1000] Epoch: [0][491/1000] Epoch: [0][501/1000] Epoch: [0][511/1000] Epoch: [0][521/1000] Epoch: [0][531/1000] Epoch: [0][541/1000] Epoch: [0][551/1000] Epoch: [0][561/1000] Epoch: [0][571/1000] Epoch: [0][581/1000] Epoch: [0][591/1000] Epoch: [0][601/1000] Epoch: [0][611/1000] Epoch: [0][621/1000] Epoch: [0][631/1000] Epoch: [0][641/1000] Epoch: [0][651/1000] Epoch: [0][661/1000] Epoch: [0][671/1000] Epoch: [0][681/1000] Epoch: [0][691/1000] Epoch: [0][701/1000] Epoch: [0][711/1000] Epoch: [0][721/1000] Epoch: [0][731/1000] Epoch: [0][741/1000] Epoch: [0][751/1000] Epoch: [0][761/1000] Epoch: [0][771/1000] Epoch: [0][781/1000] Epoch: [0][791/1000] Epoch: [0][801/1000] Epoch: [0][811/1000] Epoch: [0][821/1000] Epoch: [0][831/1000] Epoch: [0][841/1000] Epoch: [0][851/1000] Epoch: [0][861/1000] Epoch: [0][871/1000] Epoch: [0][881/1000] Epoch: [0][891/1000] Epoch: [0][901/1000] Epoch: [0][911/1000] Epoch: [0][921/1000] Epoch: [0][931/1000] Epoch: [0][941/1000] Epoch: [0][951/1000] Epoch: [0][961/1000] Epoch: [0][971/1000] Epoch: [0][981/1000] Epoch: [0][991/1000] Saving variables
Variables saved in 0.42 seconds
Saving metagraph
Metagraph saved in 2.74 seconds
Runnning forward Accuracy: 0.722+-0.022
Validation suspecting the same thing. Finally I got some time to do some testing. To narrow down, I use weight_decay_factor=0.0, and center_loss_factor=1.0, in order to analyze how reg_loss is changing overtime. I have two setup (1) current implementation (2) my own: put center_op to the training loop to make sure centers gets recalculated every batch. I keep all the other parameters the same, and here are the 1st epoch result
current implementation
Time 0.552 Images/sec 181 Loss 10.657 RegLoss 1.002 lr 0.1000
Time 0.555 Images/sec 180 Loss 10.597 RegLoss 1.003 lr 0.1000
Time 0.570 Images/sec 175 Loss 10.728 RegLoss 1.004 lr 0.1000
Time 0.562 Images/sec 177 Loss 10.531 RegLoss 1.004 lr 0.1000
Time 0.562 Images/sec 177 Loss 10.497 RegLoss 1.004 lr 0.1000
Time 0.568 Images/sec 176 Loss 10.768 RegLoss 1.005 lr 0.1000
Time 0.566 Images/sec 176 Loss 10.667 RegLoss 1.006 lr 0.1000
Time 0.560 Images/sec 178 Loss 10.330 RegLoss 1.006 lr 0.1000
Time 0.571 Images/sec 175 Loss 10.424 RegLoss 1.007 lr 0.1000
Time 0.552 Images/sec 181 Loss 10.375 RegLoss 1.007 lr 0.1000
Time 0.558 Images/sec 179 Loss 10.317 RegLoss 1.009 lr 0.1000
Time 0.567 Images/sec 176 Loss 10.525 RegLoss 1.010 lr 0.1000
Time 0.557 Images/sec 179 Loss 10.316 RegLoss 1.010 lr 0.1000
Time 0.563 Images/sec 177 Loss 10.141 RegLoss 1.010 lr 0.1000
Time 0.579 Images/sec 172 Loss 10.642 RegLoss 1.010 lr 0.1000
Time 0.556 Images/sec 179 Loss 10.340 RegLoss 1.009 lr 0.1000
Time 0.567 Images/sec 176 Loss 10.398 RegLoss 1.009 lr 0.1000
Time 0.572 Images/sec 174 Loss 10.330 RegLoss 1.010 lr 0.1000
Time 0.565 Images/sec 176 Loss 10.366 RegLoss 1.010 lr 0.1000
Time 0.554 Images/sec 180 Loss 10.224 RegLoss 1.011 lr 0.1000
Time 0.573 Images/sec 174 Loss 10.359 RegLoss 1.012 lr 0.1000
Time 0.561 Images/sec 178 Loss 10.438 RegLoss 1.012 lr 0.1000
Time 0.571 Images/sec 174 Loss 10.136 RegLoss 1.012 lr 0.1000
Time 0.563 Images/sec 177 Loss 10.562 RegLoss 1.012 lr 0.1000
Time 0.565 Images/sec 176 Loss 10.069 RegLoss 1.013 lr 0.1000
Time 0.558 Images/sec 179 Loss 10.534 RegLoss 1.013 lr 0.1000
Time 0.557 Images/sec 179 Loss 10.297 RegLoss 1.014 lr 0.1000
Time 0.561 Images/sec 178 Loss 10.091 RegLoss 1.015 lr 0.1000
Time 0.567 Images/sec 176 Loss 10.128 RegLoss 1.014 lr 0.1000
Time 0.562 Images/sec 177 Loss 9.992 RegLoss 1.013 lr 0.1000
Time 0.565 Images/sec 176 Loss 10.378 RegLoss 1.013 lr 0.1000
Time 0.555 Images/sec 180 Loss 9.964 RegLoss 1.014 lr 0.1000
Time 0.554 Images/sec 180 Loss 10.224 RegLoss 1.014 lr 0.1000
Time 0.561 Images/sec 178 Loss 9.933 RegLoss 1.014 lr 0.1000
Time 0.554 Images/sec 180 Loss 9.910 RegLoss 1.014 lr 0.1000
Time 0.560 Images/sec 178 Loss 10.331 RegLoss 1.014 lr 0.1000
Time 0.559 Images/sec 178 Loss 10.353 RegLoss 1.014 lr 0.1000
Time 0.558 Images/sec 179 Loss 10.115 RegLoss 1.014 lr 0.1000
Time 0.556 Images/sec 179 Loss 9.897 RegLoss 1.014 lr 0.1000
Time 0.567 Images/sec 176 Loss 10.046 RegLoss 1.014 lr 0.1000
Time 0.558 Images/sec 179 Loss 9.966 RegLoss 1.013 lr 0.1000
Time 0.565 Images/sec 176 Loss 10.109 RegLoss 1.013 lr 0.1000
Time 0.570 Images/sec 175 Loss 10.013 RegLoss 1.014 lr 0.1000
Time 0.553 Images/sec 180 Loss 9.918 RegLoss 1.014 lr 0.1000
Time 0.563 Images/sec 177 Loss 10.132 RegLoss 1.013 lr 0.1000
Time 0.568 Images/sec 175 Loss 9.973 RegLoss 1.012 lr 0.1000
Time 0.561 Images/sec 178 Loss 10.077 RegLoss 1.011 lr 0.1000
Time 0.552 Images/sec 181 Loss 9.818 RegLoss 1.011 lr 0.1000
Time 0.560 Images/sec 178 Loss 10.184 RegLoss 1.011 lr 0.1000
Time 0.569 Images/sec 175 Loss 9.895 RegLoss 1.012 lr 0.1000
Time 0.562 Images/sec 177 Loss 10.085 RegLoss 1.013 lr 0.1000
Time 0.557 Images/sec 179 Loss 9.726 RegLoss 1.013 lr 0.1000
Time 0.565 Images/sec 176 Loss 9.970 RegLoss 1.013 lr 0.1000
Time 0.562 Images/sec 177 Loss 10.056 RegLoss 1.013 lr 0.1000
Time 0.556 Images/sec 179 Loss 9.665 RegLoss 1.013 lr 0.1000
Time 0.555 Images/sec 180 Loss 9.791 RegLoss 1.013 lr 0.1000
Time 0.572 Images/sec 174 Loss 9.686 RegLoss 1.012 lr 0.1000
Time 0.563 Images/sec 177 Loss 9.458 RegLoss 1.012 lr 0.1000
Time 0.555 Images/sec 180 Loss 9.602 RegLoss 1.013 lr 0.1000
Time 0.565 Images/sec 176 Loss 9.338 RegLoss 1.013 lr 0.1000
Time 0.557 Images/sec 179 Loss 9.611 RegLoss 1.012 lr 0.1000
Time 0.560 Images/sec 178 Loss 9.628 RegLoss 1.011 lr 0.1000
Time 0.562 Images/sec 177 Loss 9.622 RegLoss 1.011 lr 0.1000
Time 0.563 Images/sec 177 Loss 9.815 RegLoss 1.011 lr 0.1000
Time 0.561 Images/sec 178 Loss 9.767 RegLoss 1.011 lr 0.1000
Time 0.558 Images/sec 179 Loss 9.532 RegLoss 1.011 lr 0.1000
Time 0.567 Images/sec 176 Loss 9.688 RegLoss 1.011 lr 0.1000
Time 0.570 Images/sec 175 Loss 9.578 RegLoss 1.010 lr 0.1000
Time 0.559 Images/sec 178 Loss 9.607 RegLoss 1.009 lr 0.1000
Time 0.561 Images/sec 178 Loss 9.342 RegLoss 1.008 lr 0.1000
Time 0.563 Images/sec 177 Loss 9.457 RegLoss 1.008 lr 0.1000
Time 0.561 Images/sec 178 Loss 9.579 RegLoss 1.009 lr 0.1000
Time 0.560 Images/sec 178 Loss 9.622 RegLoss 1.009 lr 0.1000
Time 0.572 Images/sec 174 Loss 9.402 RegLoss 1.009 lr 0.1000
Time 0.558 Images/sec 179 Loss 9.442 RegLoss 1.009 lr 0.1000
Time 0.565 Images/sec 177 Loss 9.696 RegLoss 1.009 lr 0.1000
Time 0.562 Images/sec 178 Loss 9.414 RegLoss 1.008 lr 0.1000
Time 0.574 Images/sec 174 Loss 8.986 RegLoss 1.007 lr 0.1000
Time 0.566 Images/sec 176 Loss 9.262 RegLoss 1.006 lr 0.1000
Time 0.559 Images/sec 179 Loss 9.235 RegLoss 1.006 lr 0.1000
Time 0.562 Images/sec 177 Loss 9.349 RegLoss 1.006 lr 0.1000
Time 0.562 Images/sec 177 Loss 9.181 RegLoss 1.006 lr 0.1000
Time 0.568 Images/sec 176 Loss 9.514 RegLoss 1.005 lr 0.1000
Time 0.556 Images/sec 180 Loss 9.435 RegLoss 1.005 lr 0.1000
Time 0.564 Images/sec 177 Loss 9.248 RegLoss 1.005 lr 0.1000
Time 0.565 Images/sec 176 Loss 9.735 RegLoss 1.005 lr 0.1000
Time 0.597 Images/sec 167 Loss 9.447 RegLoss 1.005 lr 0.1000
Time 0.573 Images/sec 174 Loss 9.530 RegLoss 1.005 lr 0.1000
Time 0.581 Images/sec 172 Loss 9.581 RegLoss 1.004 lr 0.1000
Time 0.580 Images/sec 172 Loss 9.310 RegLoss 1.004 lr 0.1000
Time 0.572 Images/sec 174 Loss 9.538 RegLoss 1.004 lr 0.1000
Time 0.580 Images/sec 172 Loss 8.953 RegLoss 1.005 lr 0.1000
Time 0.577 Images/sec 173 Loss 9.209 RegLoss 1.005 lr 0.1000
Time 0.574 Images/sec 174 Loss 9.000 RegLoss 1.005 lr 0.1000
Time 0.567 Images/sec 176 Loss 9.425 RegLoss 1.004 lr 0.1000
Time 0.558 Images/sec 179 Loss 9.542 RegLoss 1.004 lr 0.1000
pass on LFW images
rate: 0.02767+-0.01239 @ FAR=0.00133

(2) my own implementation
Epoch: [0][41/1000] Time 0.554 Images/sec 180 Loss 10.757 RegLoss 1.006 lr 0.1000
Epoch: [0][51/1000] Time 0.550 Images/sec 181 Loss 10.665 RegLoss 1.016 lr 0.1000
Epoch: [0][61/1000] Time 0.560 Images/sec 178 Loss 10.685 RegLoss 1.010 lr 0.1000
Epoch: [0][71/1000] Time 0.560 Images/sec 178 Loss 10.592 RegLoss 0.995 lr 0.1000
Epoch: [0][81/1000] Time 0.566 Images/sec 176 Loss 10.435 RegLoss 1.000 lr 0.1000
Epoch: [0][91/1000] Time 0.553 Images/sec 180 Loss 10.772 RegLoss 0.998 lr 0.1000
Epoch: [0][101/1000] Time 0.552 Images/sec 181 Loss 10.610 RegLoss 1.009 lr 0.1000
Epoch: [0][111/1000] Time 0.570 Images/sec 175 Loss 10.264 RegLoss 0.981 lr 0.1000
Epoch: [0][121/1000] Time 0.551 Images/sec 181 Loss 10.624 RegLoss 1.007 lr 0.1000
Epoch: [0][131/1000] Time 0.576 Images/sec 173 Loss 10.317 RegLoss 0.958 lr 0.1000
Epoch: [0][141/1000] Time 0.572 Images/sec 174 Loss 10.439 RegLoss 0.998 lr 0.1000
Epoch: [0][151/1000] Time 0.559 Images/sec 179 Loss 10.416 RegLoss 1.017 lr 0.1000
Epoch: [0][161/1000] Time 0.560 Images/sec 178 Loss 10.441 RegLoss 0.965 lr 0.1000
Epoch: [0][171/1000] Time 0.558 Images/sec 179 Loss 10.226 RegLoss 1.002 lr 0.1000
Epoch: [0][181/1000] Time 0.565 Images/sec 177 Loss 10.614 RegLoss 0.998 lr 0.1000
Epoch: [0][191/1000] Time 0.558 Images/sec 179 Loss 10.331 RegLoss 0.995 lr 0.1000
Epoch: [0][201/1000] Time 0.566 Images/sec 176 Loss 10.212 RegLoss 0.961 lr 0.1000
Epoch: [0][211/1000] Time 0.554 Images/sec 180 Loss 10.273 RegLoss 0.985 lr 0.1000
Epoch: [0][221/1000] Time 0.563 Images/sec 177 Loss 10.251 RegLoss 0.972 lr 0.1000
Epoch: [0][231/1000] Time 0.557 Images/sec 179 Loss 10.140 RegLoss 0.906 lr 0.1000
Epoch: [0][241/1000] Time 0.562 Images/sec 177 Loss 10.295 RegLoss 0.996 lr 0.1000
Epoch: [0][251/1000] Time 0.550 Images/sec 181 Loss 10.411 RegLoss 1.005 lr 0.1000
Epoch: [0][261/1000] Time 0.554 Images/sec 180 Loss 10.324 RegLoss 0.985 lr 0.1000
Epoch: [0][271/1000] Time 0.556 Images/sec 179 Loss 10.667 RegLoss 1.058 lr 0.1000
Epoch: [0][281/1000] Time 0.557 Images/sec 179 Loss 10.039 RegLoss 0.957 lr 0.1000
Epoch: [0][291/1000] Time 0.559 Images/sec 179 Loss 10.512 RegLoss 1.001 lr 0.1000
Epoch: [0][301/1000] Time 0.557 Images/sec 179 Loss 10.205 RegLoss 0.961 lr 0.1000
Epoch: [0][311/1000] Time 0.563 Images/sec 177 Loss 10.026 RegLoss 0.903 lr 0.1000
Epoch: [0][321/1000] Time 0.550 Images/sec 181 Loss 10.129 RegLoss 0.969 lr 0.1000
Epoch: [0][331/1000] Time 0.546 Images/sec 183 Loss 9.954 RegLoss 0.950 lr 0.1000
Epoch: [0][341/1000] Time 0.559 Images/sec 178 Loss 10.225 RegLoss 0.960 lr 0.1000
Epoch: [0][351/1000] Time 0.554 Images/sec 180 Loss 9.840 RegLoss 0.888 lr 0.1000
Epoch: [0][361/1000] Time 0.555 Images/sec 180 Loss 10.147 RegLoss 0.943 lr 0.1000
Epoch: [0][371/1000] Time 0.560 Images/sec 178 Loss 10.034 RegLoss 0.924 lr 0.1000
Epoch: [0][381/1000] Time 0.553 Images/sec 180 Loss 9.815 RegLoss 0.937 lr 0.1000
Epoch: [0][391/1000] Time 0.551 Images/sec 181 Loss 10.288 RegLoss 0.946 lr 0.1000
Epoch: [0][401/1000] Time 0.552 Images/sec 181 Loss 10.229 RegLoss 0.930 lr 0.1000
Epoch: [0][411/1000] Time 0.543 Images/sec 184 Loss 10.026 RegLoss 0.932 lr 0.1000
Epoch: [0][421/1000] Time 0.561 Images/sec 178 Loss 9.696 RegLoss 0.837 lr 0.1000
Epoch: [0][431/1000] Time 0.561 Images/sec 178 Loss 10.041 RegLoss 0.938 lr 0.1000
Epoch: [0][441/1000] Time 0.557 Images/sec 179 Loss 9.859 RegLoss 0.878 lr 0.1000
Epoch: [0][451/1000] Time 0.556 Images/sec 179 Loss 10.098 RegLoss 0.944 lr 0.1000
Epoch: [0][461/1000] Time 0.560 Images/sec 178 Loss 9.833 RegLoss 0.870 lr 0.1000
Epoch: [0][471/1000] Time 0.556 Images/sec 179 Loss 9.745 RegLoss 0.868 lr 0.1000
Epoch: [0][481/1000] Time 0.552 Images/sec 181 Loss 9.983 RegLoss 0.897 lr 0.1000
Epoch: [0][491/1000] Time 0.550 Images/sec 181 Loss 9.851 RegLoss 0.874 lr 0.1000
Epoch: [0][501/1000] Time 0.551 Images/sec 181 Loss 9.934 RegLoss 0.890 lr 0.1000
Epoch: [0][511/1000] Time 0.554 Images/sec 180 Loss 9.787 RegLoss 0.862 lr 0.1000
Epoch: [0][521/1000] Time 0.554 Images/sec 180 Loss 10.010 RegLoss 0.890 lr 0.1000
Epoch: [0][531/1000] Time 0.556 Images/sec 179 Loss 9.805 RegLoss 0.888 lr 0.1000
Epoch: [0][541/1000] Time 0.571 Images/sec 175 Loss 9.826 RegLoss 0.913 lr 0.1000
Epoch: [0][551/1000] Time 0.563 Images/sec 177 Loss 9.457 RegLoss 0.780 lr 0.1000
Epoch: [0][561/1000] Time 0.544 Images/sec 183 Loss 9.791 RegLoss 0.863 lr 0.1000
Epoch: [0][571/1000] Time 0.559 Images/sec 178 Loss 9.658 RegLoss 0.805 lr 0.1000
Epoch: [0][581/1000] Time 0.553 Images/sec 180 Loss 9.604 RegLoss 0.824 lr 0.1000
Epoch: [0][591/1000] Time 0.560 Images/sec 178 Loss 9.561 RegLoss 0.820 lr 0.1000
Epoch: [0][601/1000] Time 0.550 Images/sec 181 Loss 9.420 RegLoss 0.766 lr 0.1000
Epoch: [0][611/1000] Time 0.549 Images/sec 182 Loss 9.463 RegLoss 0.794 lr 0.1000
Epoch: [0][621/1000] Time 0.557 Images/sec 179 Loss 9.324 RegLoss 0.771 lr 0.1000
Epoch: [0][631/1000] Time 0.553 Images/sec 180 Loss 9.016 RegLoss 0.713 lr 0.1000
Epoch: [0][641/1000] Time 0.555 Images/sec 180 Loss 9.398 RegLoss 0.769 lr 0.1000
Epoch: [0][651/1000] Time 0.558 Images/sec 179 Loss 9.361 RegLoss 0.752 lr 0.1000
Epoch: [0][661/1000] Time 0.554 Images/sec 180 Loss 9.360 RegLoss 0.768 lr 0.1000
Epoch: [0][671/1000] Time 0.556 Images/sec 179 Loss 9.643 RegLoss 0.784 lr 0.1000
Epoch: [0][681/1000] Time 0.560 Images/sec 178 Loss 9.605 RegLoss 0.815 lr 0.1000
Epoch: [0][691/1000] Time 0.557 Images/sec 179 Loss 9.324 RegLoss 0.748 lr 0.1000
Epoch: [0][701/1000] Time 0.544 Images/sec 183 Loss 9.391 RegLoss 0.746 lr 0.1000
Epoch: [0][711/1000] Time 0.555 Images/sec 180 Loss 9.322 RegLoss 0.715 lr 0.1000
Epoch: [0][721/1000] Time 0.555 Images/sec 180 Loss 9.577 RegLoss 0.797 lr 0.1000
Epoch: [0][731/1000] Time 0.556 Images/sec 179 Loss 9.173 RegLoss 0.733 lr 0.1000
Epoch: [0][741/1000] Time 0.559 Images/sec 178 Loss 9.152 RegLoss 0.691 lr 0.1000
Epoch: [0][751/1000] Time 0.552 Images/sec 181 Loss 9.433 RegLoss 0.769 lr 0.1000
Epoch: [0][761/1000] Time 0.557 Images/sec 179 Loss 9.313 RegLoss 0.731 lr 0.1000
Epoch: [0][771/1000] Time 0.552 Images/sec 181 Loss 9.259 RegLoss 0.726 lr 0.1000
Epoch: [0][781/1000] Time 0.561 Images/sec 178 Loss 9.268 RegLoss 0.704 lr 0.1000
Epoch: [0][791/1000] Time 0.557 Images/sec 179 Loss 9.373 RegLoss 0.765 lr 0.1000
Epoch: [0][801/1000] Time 0.544 Images/sec 183 Loss 9.119 RegLoss 0.697 lr 0.1000
Epoch: [0][811/1000] Time 0.562 Images/sec 178 Loss 8.764 RegLoss 0.625 lr 0.1000
Epoch: [0][821/1000] Time 0.562 Images/sec 177 Loss 9.047 RegLoss 0.697 lr 0.1000
Epoch: [0][831/1000] Time 0.556 Images/sec 179 Loss 8.937 RegLoss 0.643 lr 0.1000
Epoch: [0][841/1000] Time 0.556 Images/sec 179 Loss 9.056 RegLoss 0.637 lr 0.1000
Epoch: [0][851/1000] Time 0.563 Images/sec 177 Loss 8.875 RegLoss 0.649 lr 0.1000
Epoch: [0][861/1000] Time 0.552 Images/sec 181 Loss 9.159 RegLoss 0.682 lr 0.1000
Epoch: [0][871/1000] Time 0.563 Images/sec 177 Loss 9.168 RegLoss 0.665 lr 0.1000
Epoch: [0][881/1000] Time 0.563 Images/sec 177 Loss 9.087 RegLoss 0.651 lr 0.1000
Epoch: [0][891/1000] Time 0.559 Images/sec 178 Loss 9.463 RegLoss 0.703 lr 0.1000
Epoch: [0][901/1000] Time 0.566 Images/sec 176 Loss 9.085 RegLoss 0.625 lr 0.1000
Epoch: [0][911/1000] Time 0.562 Images/sec 177 Loss 9.378 RegLoss 0.714 lr 0.1000
Epoch: [0][921/1000] Time 0.558 Images/sec 179 Loss 9.199 RegLoss 0.652 lr 0.1000
Epoch: [0][931/1000] Time 0.569 Images/sec 175 Loss 8.975 RegLoss 0.628 lr 0.1000
Epoch: [0][941/1000] Time 0.572 Images/sec 174 Loss 9.119 RegLoss 0.652 lr 0.1000
Epoch: [0][951/1000] Time 0.566 Images/sec 176 Loss 8.669 RegLoss 0.580 lr 0.1000
Epoch: [0][961/1000] Time 0.564 Images/sec 177 Loss 8.897 RegLoss 0.632 lr 0.1000
Epoch: [0][971/1000] Time 0.563 Images/sec 177 Loss 8.614 RegLoss 0.560 lr 0.1000
Epoch: [0][981/1000] Time 0.545 Images/sec 183 Loss 9.130 RegLoss 0.680 lr 0.1000
Epoch: [0][991/1000] Time 0.564 Images/sec 177 Loss 9.329 RegLoss 0.688 lr 0.1000
Saving variables
Variables saved in 0.38 seconds
Saving metagraph
Metagraph saved in 2.64 seconds
Runnning forward pass on LFW images
Accuracy: 0.735+-0.015
Validation rate: 0.05000+-0.01282 @ FAR=0.00100

This probably also explains #390

@davidsandberg
@apollo-time

JianbangZ on 1 Aug 2017

@JianbangZ
I'm glad to see your response. RegLoss seemed to decrease much faster. Can't wait to see the results after more epochs. I hope the change could make the performance even better.
I had printed the centers before, the centers did change over time but kept very small. So I wasn't sure whether it is updated or not. Maybe the centers were shifted just because of the numerical error?

ugtony on 1 Aug 2017

Hi @ugtony, @JianbangZ,
Thanks for spotting this! I guess the running of the center_op got dropped at some point. I added a control dependency in the center loss function on a local branch:

    with tf.control_dependencies([centers]):
        loss = tf.reduce_mean(tf.square(features - centers_batch))

davidsandberg on 1 Aug 2017

👍3

@davidsandberg I don't think the center loss factor should be multiplied by the batch size. Each iteration the softmax loss is also mini-batch based, same as center loss. So if you multiply by batch size, the other loss should be multiplied by batch size as well. Correct me if I am wrong.

JianbangZ on 2 Aug 2017

@JianbangZ I think you're right, softmax and center loss are both normalized by batch size, so center loss factor could be kept unchanged while batch size changes.

But I guess what davidsandberg try to remind us is that the center loss factor used in his earlier version (implemented by l2_loss, which is sum of square difference) should be multiplied by batch size before used in the current version(mean of square difference).

BTW, I think the implementation of center_loss_alpha is different to the definition in the paper.
alpha_in_paper = 1 - center_loss_alpha

ugtony on 2 Aug 2017

@ugtony @davidsandberg Share my results here. I train with casia. I use center_loss_factor=5e-3, and when computing the loss, I multiplied it by batch_size=100. Eventually I got accuracy around 98.4% (pretty much the same as before), but the validation rate @ 0.001 is 0.94 (compared to 0.92 before).
Thoughts?

JianbangZ on 3 Aug 2017

What value do you set for center_loss_alpha?
You set center_loss_actor to 5e-3 and mutually the loss by 100, so the effective factor is 5e-1? I think it might be too large according to Fig.5 in the paper.

ugtony on 3 Aug 2017

my center_loss_alpha=0.6 (I try to set it around the middle of [0, 1] according to the paper)
I tried center_loss_factor = 5e-5, after multiplying by batch_size it equals to 5e-3 (which is OK according to the paper). However, the results was worse, worse than the center op fix. (accuracy 98.2%, validation rate ~ 90%)

JianbangZ on 3 Aug 2017

I just finished my training with center_loss_factor=2e-2 and center_loss_alpha=0.95, the performance is pretty similar to before.
Another training with center_loss_alpha=0.5 is still running, but the performance isn't better so far.
Still cannot reproduce the results of the paper. :(

ugtony on 4 Aug 2017

Was this page helpful?

0 / 5 - 0 ratings