Caffe: Error on test_multibox_loss_layer

[shaibagon]

I am trying to run tests on ssd branch.
I see something weird on test_multibox_loss_layer.cpp:

On the one hand, GT bottom is allocated 7 elements per item:

blob_bottom_gt_(new Blob<Dtype>(1, 1, 4, 7))

But on the other hand, GetGroundTruth assumes there are 8 elements per item:

int start_idx = i * 8;

Running this test:

~$ ./build/test/test_multibox_loss_layer.testbin

Results in all sorts of errors: sometimes

F0209 08:47:37.873329 8777 bbox_util.cpp:1066] Check failed: background_label_id != label (0 vs. 0) Found background label in the dataset.
And occasionally I get a segmentation fault.

Can you please look closely at this test, specifically at the allocation of its "bottom"s?

Thanks!
-Shai

[weiliu89]

Good catch. It is an oversight.

However, it shouldn't cause any real problem. Because here and here will reshape it to 8 in width direction.

Not sure why you get this error. Travis didn't complain. I have fixed it now.

[shaibagon]

@weiliu89 Thank you for your prompt response!
I am trying to merge ssd into master caffe (with some extra "goodies" I have) therefore there are some slight differences between your code (that passes all tests) and mine. I am trying to debug it.

I was wondering why you took a very non-trivial approach in the method Fill of the multibox loss layer test: why having fake input, fake pooling, fake convolution when you only want to test the loss layer? shouldn't there be a simpler way to Fill the "bottom"s of the loss layer directly?
This way the test would be simpler, easier to understand and debug and have better readability.

Thanks!

[weiliu89]

The intention was to fake a mini SSD network to make sure the whole thing work end to end (from LMDB input to multibox loss).

I think your error happens because you have some conflicts in the annotated_data_layer.

[shaibagon]

@weiliu89 you might be right, though the test_annotated_data_layer passes.

Anyhow, there is a gradient check util for testing entire net end-to-end. The purpose of unit-tests is to test the units themselves, the more focused these test the more effective and useful they are.

Thank you very much!

[geeheim]

I run into the same problem, when merging with the current caffe branch (rc5).

The error is because of a change in BasePrefetchingDataLayer::Forward_cpu. The old version copies the data, while the new version just sets the pointer.

Thus in the test TestLocGradient the member variables blob_bottom_vec_ and blob_top_vec_ are referencing data from the AnnotatedDataLayer, which is already destroyed.

I fixed the test by just making AnnotatedDataLayer also a member of MultiBoxLossLayerTest .

Regards.
Geert

[myme5261314]

@geeheim Thanks for your hint. I'm doing the same thing with you which is merging ssd into rc5.
For the specific thing, the AnnotatedDataLayer anno_data_layer should be a pointer member of MultiBoxLossLayerTest
And then delete if already allocated and new as the old one.

I've tested that use a variable rather than pointer of AnnotatedDataLayer will cause another LMDB lock issue which will fail the test.

[jmarwane]

Hi !
I'm trying to compute loss for my test net, all i did is copy/paste the loss layer definition in the train net proto and change phase to TEST.
When i try to forward the test net i get this error :
... bbox_util.cu:595] Check failed: match_index[p] < gt_bboxes.size() (1 vs. 1)
@shaibagon how did you define your loss layer in the test net ?

[liyuanyaun]

@jmarwane i encounter the same problem as you, did you find how to fix this and could you share it? thank you very much

[wantai2017]

@liyuanyaun I met the same problem, too. Mine is Check failed: match_index[p] < gt_bboxes.size() (4 vs. 1) . Did you fix this problem? If so, could you please tell me the way you fix it? Thanks a lot!