Ssd.pytorch: how about the current performance

[gaopeng-eugene]

[christopher5106]

mAP score ?

[amdegroot]

Hey sorry just getting back to these, was on vacation this past weekend away from my computer. We should have an official mAP score as well as timing metrics by Thursday of this week, hopefully sooner... We've just recently been caught up on a lot of other work, but stay tuned.

[oarriaga]

Hello @amdegroot do you have any info regarding the current mAP score?

[amdegroot]

Hey @oarriaga,
Thanks for checking out the repo. Unfortunately, I've been distracted by a lot of school work recently, but I'm still in the process of getting an official 'training from scratch' mAP score and should have it within the next week. Currently I can confirm that the model with pre-trained weights reproduces the 77.26% mAP of the original caffe version, and training from scratch is close. We just need to finish data augmentation, and combining VOC2007 and VOC2012 for training (currently the repo is set up to only train on one or the other). Any and all contributions are welcome, so feel free to submit a PR, and the repo is updated fairly frequently so make sure to stay tuned.

[gaopeng-eugene]

How about the FPS of current model? This is another very important factor.

[amdegroot]

@gaopeng-eugene You're right, I will post this as soon as I can. I am currently maxing out my GTX 1060 for training right now, but to give you an idea, it was taking ~0.022s per image during evaluation (on GTX 1060), so it should be above 50 fps for just detection. It should also be easy to benchmark yourself. I would love to hear results from someone with a TitanX.

[amdegroot]

Actually I was off by a bit, after benchmarking the current FPS on a GTX 1060, it looks like we're at 45.45 FPS for detection on a single image.

[gaopeng-eugene]

Nice to hear the results. Thank you for your contribution.

[gaopeng-eugene]

The mAP is 50.8 when training from scratch? Right ?

[amdegroot]

For now, without the data augmentation, correct. However, we haven't had a chance to adjust the learning rate, so I think it should actually be able to achieve a higher result than that.

[amdegroot]

Actually, I'm training right now and with a different learning rate, at 20k iterations I'm already at 56.2% mAP (still w/o data augmentation), so hopefully once this finishes training it'll be a lot better than before.

[xwjabc]

What is the mAP for pretrained models? I am trying to migrate mAP calculation module from longcw's repo of Faster RCNN. Is there any possibility to offer the mAP calculation part?

[amdegroot]

Just posted the link to Pytorch trained from scratch weights that achieve mAP of 77.43%. Also added the mAP eval code used to calculate that result.

[acrosson]

Just downloaded and trained for 10k iterations. The evaluation script is taking ~0.30 seconds / image with GTX 1080 and ~0.10 on GTX 1080 ti. Are you still seeing 45 FPS?

[liuchang8am]

Same with acrosson here, inference time is about 300ms per image on GTX 1070, which is far slower than the 45FPS in the description.

[SystemErrorWang]

I met the same problem. When using original vgg backbone the speed is ok. I then changed the backbone to resnet101, the training speed did not slow down much but testing for each image took 2 seconds now.

[d-li14]

Same issue about speed much lower than 45fps. And @SystemErrorWang, how long did you take when trying the original vgg16-based model, up to 45fps? My evaluation process will take about 0.1s even with the provided pretrained vgg model.

[CasonTsai]

@acrosson i met the same situation.Have you solved this problem?