Mmdetection: Cannot train a (faster + res50 + fpn) model with similar mAP using one GPU

Created on 21 Oct 2018 · 7Comments · Source: open-mmlab/mmdetection

Hello, Thanks for your excellent work.
I firstly use the pre-trained model of "faster_rcnn_r50_fpn_1x_20181010-3d1b3351.pth",
and test using one Titan xp on server.
the mAP show a similar result:

Then I use the following cmd to train: python ./tools/train.py ./configs/faster_rcnn_r50_fpn_1x.py --gpus 1 --work_dir ./experiments/faster_rcnn_r50_fpn_1x
On the only one Titan xp, it costs 3 days...
When I test it use the same scripts, the mAP shows an inferior result:

I guess this might because I only use one GPU to train. But the total number of mini-batch is fixed, and there exists no BN option for res50 on single GPU...I cannot figure why the mAP drops nearly 10 points...
Could you please give some advice

Source

hedes1992

Most helpful comment

I finished training voc0712 by using one 1080TI and the results is follow:

+-------------+------+-------+--------+-----------+-------+
| class | gts | dets | recall | precision | ap |
+-------------+------+-------+--------+-----------+-------+
| aeroplane | 285 | 968 | 0.951 | 0.284 | 0.865 |
| bicycle | 337 | 1223 | 0.955 | 0.269 | 0.864 |
| bird | 459 | 1113 | 0.915 | 0.391 | 0.847 |
| boat | 263 | 1573 | 0.878 | 0.153 | 0.693 |
| bottle | 469 | 2073 | 0.855 | 0.199 | 0.722 |
| bus | 213 | 813 | 0.944 | 0.253 | 0.848 |
| car | 1201 | 3966 | 0.960 | 0.308 | 0.881 |
| cat | 358 | 964 | 0.975 | 0.365 | 0.893 |
| chair | 756 | 5048 | 0.890 | 0.144 | 0.662 |
| cow | 244 | 918 | 0.967 | 0.274 | 0.860 |
| diningtable | 206 | 2375 | 0.917 | 0.082 | 0.724 |
| dog | 489 | 1396 | 0.980 | 0.351 | 0.880 |
| horse | 348 | 1168 | 0.951 | 0.291 | 0.866 |
| motorbike | 325 | 1340 | 0.945 | 0.235 | 0.844 |
| person | 4528 | 15305 | 0.951 | 0.290 | 0.858 |
| pottedplant | 480 | 2951 | 0.825 | 0.137 | 0.582 |
| sheep | 242 | 775 | 0.938 | 0.304 | 0.839 |
| sofa | 239 | 1619 | 0.954 | 0.154 | 0.775 |
| train | 282 | 1184 | 0.954 | 0.229 | 0.856 |
| tvmonitor | 308 | 1275 | 0.925 | 0.229 | 0.810 |
+-------------+------+-------+--------+-----------+-------+
| mAP | | | | | 0.808 |
+-------------+------+-------+--------+-----------+-------+

xtanitfy on 22 Dec 2018

👍4

All 7 comments

The following is the end of the train log:

Is this loss and acc similar to yours?

hedes1992 on 21 Oct 2018

As stated in README, You need to modify the learning rate according to the number of GPUs you use. If you use only one GPU, then the initial learning rate should be 0.02/8.

hellock on 21 Oct 2018

@hellock oh, I get it. it's linear scale strategy in kaiming's paper. Let me do experiment to validate this conclusion

hedes1992 on 21 Oct 2018

@hedes1992 I want to train the model as yours by using only one 1080TI.
Have you trained again after changing learning rate.
Looking for your reply!Thanks!

xtanitfy on 21 Dec 2018

@hedes1992 I want to train the model as yours by using only one 1080TI.
Have you trained again after changing learning rate.
Looking for your reply!Thanks!

Yes, I use one gpu after changing learning rate. the result is 35.7 (while the official result is 36.4)

After that, I use 8 gpus to do training, the result is 36.5, very near to 36.4

So the mmdetection is very robust!