Models: cannot get the same mAP for SSD MobileNet as provided in official table

Created on 5 Apr 2018 · 11Comments · Source: tensorflow/models

System information

What is the top-level directory of the model you are using: models/research/object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.6
Bazel version (if compiling from source): /
CUDA/cuDNN version: 9.0 / 7.0.5
GPU model and memory: GTX 1080 Ti, 11GB
Exact command to reproduce:

python object_detection/eval.py \
--logtostderr \
--checkpoint_dir=ssd_mobilenet_v1_coco_2017_11_17 \
--eval_dir=$eval_dir \
--pipeline_config_path=object_detection/samples/configs/ssd_mobilenet_v1_coco.config

Describe the problem

First bug, I cannot even evaluate model ssd_mobilenet_v1_coco_2017_11_17 without adding "metrics_set: coco_detection_metrics" in eval_config{} in object_detection/samples/configs/ssd_mobilenet_v1_coco.config

More important, I got mAP: 26, not 21 as in the offical table. Also for ssd_mobilenet_v2_coco_2018_03_29 I got 25, not 22 as in the official table.
Evaluation was on: COCO val_2017 (tfRecords are created by provided script ./object_detection/dataset_tools/download_and_preprocess_mscoco.sh )

Link on the official table: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

Source code / logs

here is my output:

INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Restoring parameters from tiris/ssd_mobilenet_v1_coco_2017_11_17/model.ckpt
INFO:tensorflow:Restoring parameters from tiris/ssd_mobilenet_v1_coco_2017_11_17/model.ckpt
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
INFO:tensorflow:Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.39s)
INFO:tensorflow:DONE (t=0.39s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=71.17s).
Accumulating evaluation results...
DONE (t=11.80s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.263
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.419
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.279
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.016
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.132
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.508
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.238
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.342
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.362
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.042
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.253
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.643

awaiting model gardener

Source

dextroza

Most helpful comment

@dextroza similar with your result:

creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
INFO:tensorflow:Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.27s)
INFO:tensorflow:DONE (t=0.27s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=43.80s).
Accumulating evaluation results...
DONE (t=7.45s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.263
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.421
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.278
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.017
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.131
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.510
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.240
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.343
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.362
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.253
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644

ShuaiZ1037 on 9 May 2018

👍3

All 11 comments

@dextroza would you mind posting your actual command to reproduce this?

ahundt on 10 Apr 2018

@ahundt check my post above, the actual command is added.

dextroza on 10 Apr 2018

@dextroza , Hi Dear,i've trainined ssd_mobile_coco_v1 on my own dataset successful , but i have problem with running in eval.py , i saw you get result with eval.py , please give your experiences .
and my other problem evaluation depend on map coco , i get some error about this protocol.

PythonImageDeveloper on 12 Apr 2018

@zeynali What problem do you have? Could you be more specific? I've just run their eval.py with paths to ckpt, config and eval_dir and it did evaluation successfully.

dextroza on 13 Apr 2018

@dextroza , Hi , i've trained the ssd_mobilev1_coco on my own dataset that have only one class the total dataset for training is 200k samples and for testing 35k , and i go through 450k step with size 608*608 and batch_size 20 , and my train mAP is 85 but test is 35 why ??????

PythonImageDeveloper on 18 Apr 2018

👍1

@zeynali You should check your dataset, data augmentation or try with L2 or dropout regularization if is possible. Let me know if you have any news. Good luck!

dextroza on 18 Apr 2018

@dextroza similar with your result:

ShuaiZ1037 on 9 May 2018

👍3

@jch1 do you have time to check this, please?

dextroza on 10 May 2018

I heard that 21 is from coco2014, while with coco2017 your number is correct. Can you test with coco2014?

tensorbuffer on 27 Aug 2018

can someone explain why do we get 6 precision and recall values each time?

sujeet-gandhi on 30 Aug 2018

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.