Caffe: how to get test loss?

[spotofleopard]

Thanks to Wei for the great work, and sharing it with everybody.

I need to get TEST loss during training, so that I can get some idea about how well the network is doing in generalization. In the python script I tried to add MultiBoxLoss layer to the test net the same way train net did, however it give me error at test time:
solver.cpp:464 check failed: result[j]->width() ==5 (1 vs 5)

I am not familiar with the internals of Caffe, could you give me a pointer on proper way of enabling test loss in the Python script?
Your help is greatly appreciated.

[weiliu89]

I think you have to add a silence layer after the loss layer.

[spotofleopard]

That worked! Now I can see my network is overfitting a lot:)
Thanks again!

[song6299]

Please, can you show the code to add a multibox loss layer? I don't know how to add a silence layer

[spotofleopard]

in the python script,when you create the test net, after CreateMultiBoxHead, add these lines

name = "test_loss"
mbox_layers.append(net.label)
net[name] = L.MultiBoxLoss(*mbox_layers, multibox_loss_param=multibox_loss_param,
        loss_param=loss_param, include=dict(phase=caffe_pb2.Phase.Value('TEST')),
        propagate_down=[True, True, False, False])
 name="silence"
 net[name] = L.Silence(net['test_loss'],ntop=0)

also, add this to the solver_param:
'test_compute_loss':True,

@weiliu89
I am not 100% the output is correct though, because when my test accuracy steadily increases and train loss drops, the test loss is stuck mostly flat. If it's overfitting, the test loss should increase somewhat.

[song6299]

You are so nice, thanks a lot @spotofleopard

[song6299]

Hi! @spotofleopard Are you sure that is all you have added to your code? I add it to the python script, but it doesn't work. It gives me error at test time :
Check failed: match_index[p] Can you help me to solve it?

[abdelrahman-gaber]

@song6299
Did you manage to get the solution for this error: Check failed: match_index[p]

@spotofleopard
Can you please tell us if you did any more modifications in the code before adding these lines. or if possible upload your whole code as a reference, and tell us which dataset you were using.

[song6299]

@abdelrahman-gabero No, I didn' t. At last, I replaced train dataset with test dataset, and trained the test dataset by fintuing on pre-trained model to get test loss.

[abdelrahman-gaber]

@song6299 Thank you for replying.
You tried to solve the problem and you got some numbers and curves, but I think it is not the right way to get validation loss. The idea is to get the validation loss during training and compare it with training loss, so that we measure how much our model is over-fitting. However, with your solution you already have new weights when you train on test set, and mainly training on test set is not a good approach !

[louis0802]

@abdelrahman-gaber @spotofleopard I have a problem as same as @song6299 .I try to google that the keyword "Check failed: match_index[p] < gt_bboxes.size() (1 vs. 1)".But still no answer.Maybe some answers are right but I can't get it.Can you help me?Below link is partly code.https://drive.google.com/open?id=1bpYhbjAFmg_-s0pH-3i0wMu35Zswah8P

[wantai2017]

@song6299 Hello! Could you please tell me if you have already solved the problem "Check failed: match_index[p] < gt_bboxes.size() " ? If you have fixed it, could you share the method with me? Thank you so much!

[angiend]

@wantai2017 @song6299 @louis0802
Did the problem have been solved already?Would you share the solution with me?thanks!

[fraukej]

@song6299 @abdelrahman-gaber @louis0802 @wantai2017 @angiend
I think I found the solution. I had the same error after adding @spotofleopard 's code to the python script. Meaning, I added the test_loss layer, the silence layer and the test_compute_loss: True solver parameter and got the error "Check failed: match_index[p] < gt_bboxes.size() (1 vs. 1)" or similar.

I then digged deeper into the code and found that these match indices apparently have indices of ground truth boxes of previous samples. E.g. if there were 6 ground truth bounding boxes in one sample, the values in match_index go from 0..5. If in the next sample we have less ground truth boxes, e.g. 3, the indices go from 0..2. For whatever reasons, the indices of previous samples don't get removed from the match_index vector and when the indices are called, we encounter the problem that there is no ground truth box for the matched index, resulting in the above error.

I saw that the clearing of the affected vectors is done in bbox_util.cpp, but it is commented.
So, I removed the following //'s:

https://github.com/weiliu89/caffe/blob/4817bf8b4200b35ada8ed0dc378dceaf38c539e4/src/caffe/util/bbox_util.cpp#L729-L730
in function FindMatches and
https://github.com/weiliu89/caffe/blob/4817bf8b4200b35ada8ed0dc378dceaf38c539e4/src/caffe/util/bbox_util.cpp#L863
in function MineHardExamples.

Afterwards it worked for me. However, I don't understand why the error just now happens, after only adding extra layers. Any more insight and debugging on this subject is greatly appreciated!

[ChrystleMyrnaLobo]

Thanks, @fraukej. It worked.
Just want to add, after making changes to the file, we need to run make

[PiseyYou]

@fraukej after add the layer, is it necessary to make clean, then make all?
@ChrystleMyrnaLobo follow your instrustion, after I change these, when I make clean, and make all, the runtest failed, it show as below:
[ FAILED ] 4 tests, listed below:
[ FAILED ] MultiBoxLossLayerTest/0.TestConfGradient, where TypeParam = caffe::CPUDevice
[ FAILED ] MultiBoxLossLayerTest/1.TestConfGradient, where TypeParam = caffe::CPUDevice
[ FAILED ] MultiBoxLossLayerTest/2.TestConfGradient, where TypeParam = caffe::GPUDevice
[ FAILED ] MultiBoxLossLayerTest/3.TestConfGradient, where TypeParam = caffe::GPUDevice
Is there some suggestion to deal with such error?