Models: Only 0.5 MAP on ssd_mobilnet_v1 object detection for pascal dataset.

[blake-varden]

System information

• What is the top-level directory of the model you are using:
  object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  False
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Ubuntu 14.04
• TensorFlow installed from (source or binary):
  binary
• TensorFlow version (use command below):
  1.2
• Bazel version (if compiling from source):
• CUDA/cuDNN version:
  8.0/5.1
• GPU model and memory:
  4 x Titan X Pascal
• Exact command to reproduce:

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

I successfully trained an ssd_mobilenet_v1 detection model locally using the oxford pet dataset and the file samples/configs/ssd_mobilenet_v1_pets.config and the initialized weights /ssd_mobilenet_v1_coco_11_06_2017/model.ckpt
I achieved ~.9 test MAP score.

I attempted to replicate this process with the pascal voc12 dataset. To do this, I followed the instructions for downloading the dataset and converting it to a tensorflow train/val record.

I copied the file samples/configs/ssd_mobilenet_v1_pets.config and pointed the datasets and labels file to the train/val tensorflow recrods. I have attached this config.

I then ran the training and testing configuration locally. Even after 120k iterations, the test MAP seemed to plateau around 0.5-0.6. Are there different recommended parameters for the pascal dataset?

I understand this may be more a stack overflow question, however, because the dataset exists in repo documentation and is mentioned in the paper, I would hope the code in the repository is set up to replicate the results of the paper.

I have attached photos of the tensorboard. Orange is train and teal is test.

ssd_mobilenet_v1_voc12.config.txt

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

[jch1]

Hi @blake-varden -

It may be the case that you will have to tune the hyperparameters to achieve good performance on the Pascal dataset. But some other things to consider... which training set are you using? Are you combining 07+12 for training? Are you including the validation set as part of the training set? And how are you treating the "difficult" instances?

[huanyuxinchen]

Hi， @blake-varden
I have met the same problem as you. I trained ssd_mobilnet_v1 on pascal voc07+12 datasets and followed all the instructions well. But after 460k iterations, the mAP is only 0.59. Have you solved this problem? Thank you!

[abearman]

I am also having the exact same problem as the above commenters. I trained ssd_mobilenet_v1 on PASCAL VOC 07+12 trainval and followed all the instructions. My mAP on the VOC 2007 test set is flattening out at 0.55

[PythonImageDeveloper]

@blake-varden @abearman, Hello dears, i trained ssd_mobile_coco on my custom dataset , and that's good , but i don't know how i get my mAP , recall , .... How do you get mAP for each category ? in my Tensorboard , the Performance Tab don't exist ? what do i do ?
Q2 : is there a way to see both my losses (train and evaluation) when i doing a training ? i want to know when i faced with over-fitting ? and how long to train my problem .

[sooonwoo]

@zeynali
Do you solve your questions? I have same quetions too.

[LXWDL]

@blake-varden @huanyuxinchen
Do you solve this problem? I encountered the same problem. I trained on the trainval in VOC0712, tested on the VOC2007 test data, and only reached 65% of the map using MobileNet V2. Does anyone know what parameters can be adjusted to achieve the accuracy of the paper? thank you very much.

[cr-hxfan]

@LXWDL Did you train from scratch? Did you find any solution now?

[Harshini-Gadige]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

[a56256324]

@blake-varden @jch1 @huanyuxinchen @abearman @PythonImageDeveloper hi,
Has this problem been resolved? I also encountered a similar problem. In the pascal voc competition, I directly downloaded the code and submitted the test results. I found that the difference from the official value was 5-6%. What is the reason?