Caffe: Fine tuning issue with SSD500

[siddharthm83]

I am using SSD500 to fine tune on some(16) imagenet synsets (between 1k to 4k images per synset). I am using same hyper parameters as Pascal VOC. Even after ~25k iterations, loss is stagnant at 6.5 to 7.5 (learning rate is still 0.001). I noticed that some of the imagenet synsets only have 1 bbox per image (~30% of images)and usually bbox may occupy >80% of the image. I am able to train with faster r-cnn. I am wondering why SSD is not converging? (is it perhaps due to the batch_sampler randomly cropping off images?). Should I try with no data augmentation (deleting batch_sampler)? This may be a problem with my specific dataset but I was wondering if you have any tips for me to try to get the loss to converge.

[weiliu89]

Maybe you can try to adjust your learning rate? What is your normalization_mode? How many GPUs you are using? What is the batch_size and accum_batch_size? I have some estimation of the learning rate for each different setting in the ssd_pascal.py, but it is by no means the optimal for every dataset. For example, since ImageNet has more sparse annotation than VOC, then the "effective" batch_size is smaller than VOC (there are less matched bboxes), which may require lower learning rate? (I am not entirely sure. You could try increase lr as well.) Besides, I would also suggest evaluating on some validation set on the fly so you know roughly how the model really performs.

[siddharthm83]

@weiliu89 . I tried with learning rate =0.001 and 0.0001 (~10k iterations, loss is between 6.5 and 7.5). Most parameters are similar to your pascal voc script.

1. What is your normalization_mode?
   normalization_mode = P.Loss.VALID
2. How many GPUs you are using? What is the batch_size and accum_batch_size?
   I have 1 Titan X. I am using batch_size = 16 accum_batch_size = 32
   What batch size would you recommend?

[siddharthm83]

Here is a distribution of num bbox/image for the dataset (most images have only 1 bbox)

I also went through your juypter notebook to verify if the dataset was created properly.

[siddharthm83]

@weiliu89 : I increased accum_batch_size as per your suggestion and decreased initial learning rate (as per suggestions in your paper for COCO/ISLVRC). The loss is converging nicely now.