Ssd.pytorch: Loss is nan when using default configs

[DHZS]

I am using pytorch 1.0. When I was training the model, I got these:

iter 0 || Loss: 3.2635 || timer: 0.0842 sec.
iter 10 || Loss: 14.6995 || timer: 0.0851 sec.
iter 20 || Loss: 12.2860 || timer: 0.0846 sec.
iter 30 || Loss: 14.0080 || timer: 0.0854 sec.
iter 40 || Loss: 20.0848 || timer: 0.0903 sec.
iter 50 || Loss: 29821698048.0000 || timer: 0.0833 sec.
iter 60 || Loss: nan || timer: 0.0837 sec.
iter 70 || Loss: nan || timer: 0.0833 sec.

I only modified the code to make the code run in pytorch 1.0. The configs I changed:

batch_size = 1
basenet = '.../vgg16_reducedfc.pth'

# using trained model also got loss = nan
# resume = '.../ssd300_mAP_77.43_v2.pth'

[DHZS]

Solved this problem by using a small learning rate.

The following post may help to explain this problem:
https://www.reddit.com/r/MachineLearning/comments/84waz4/d_relation_between_learning_rate_batch_size_and/

[krxat]

@DHZS What learning rate did you use?
Did the model predict correctly after training??
Thanks

[DHZS]

@cypherix Sorry, I forgot the result. But small batch size(such as 1) in one-stage detector is not a good choice. Setting batch size >= 16 can get good results.
Anyway, if you want to try a small batch size, you can change the learning rate according to this paper:
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
The author said:

Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.

[krxat]

@DHZS Hi again. How many iterations did you train for?

[ElegantLin]

I set the lr 1e-7 and the batch size 16 to solve the problem finally TAT...