Centernet: Possible BUG when evaluating PASCAL format mAP?

[liuchang8am]

Following experiments script, test.py for pretrained model pascal_resdcn18_384, the evaluation result is as shown in the figure:

python3 test.py ctdet --exp_id pascal_resdcn18_384 --arch resdcn_18 --dataset pascal --load_model ~/data/models/pytorch/CenterNet/ctdet_pascal_resdcn18_384.pth

where "ctdet_pascal_resdcn18_384.pth" is provided in ModelZoo.

Anyone else has the same issue? Is this a bug or my incorrect configuration (following DATA.md PASCAL VOC)

[xingyizhou]

It works fine in my machine. Do you see some abnormal logs when loading the model? Can you add --debug 2 to your command to visualize the heatmap outputs?

[liuchang8am]

It works fine in my machine. Do you see some abnormal logs when loading the model? Can you add --debug 2 to your command to visualize the heatmap outputs?

No abnormal logs are observed. with --debug 2 the visualizations are correct. Output log is attached, bar processing is not shown in the log.

resdcn_18_test.log

[xingyizhou]

I have fixed a pickle version bug. Can you remove data/voc/VOCdevkit/VOC2007/ImageSets/Main/test.txt_annots.pkl and try the current version?

[liuchang8am]

I have fixed a pickle version bug. Can you remove data/voc/VOCdevkit/VOC2007/ImageSets/Main/test.txt_annots.pkl and try the current version?

I've updated the code as https://github.com/xingyizhou/CenterNet/blob/d56bfd94cfe8c4697ebca8829a2c18a637480a7d/src/tools/voc_eval_lib/datasets/voc_eval.py#L122
and remove the cachefiletest.txt_annots.pkl, but the eval results seems unchanged. I've uploaded the generated test.txt_annots.pkl and bboxes results for reference:
https://drive.google.com/file/d/1mxQsu0vUsoVUw2H5FokQwynsw_ndMikr/view?usp=sharing
https://drive.google.com/file/d/1XO14VNnqQzjjouhRSbiMbClCrohJ3QN6/view?usp=sharing

[moothes]

I think maybe it's cause by the image orders in Dataset.
The results I get with source code:

I modify the test.py :
```
class PrefetchDataset(torch.utils.data.Dataset):
def __init__(self, opt, dataset, pre_process_func):
self.images = dataset.images
print(self.images[:20])
self.load_image_func = dataset.coco.loadImgs
self.img_dir = dataset.img_dir
self.pre_proces
...

It print:
![捕获](https://user-images.githubusercontent.com/17693547/56574940-124eea80-65f7-11e9-974f-026fd058bb18.JPG)

And I modify src/lib/datasets/dataset/pascal.py：

def run_eval(self, results, save_dir):
print(self.image_index[:20])
...

It print:
![捕获](https://user-images.githubusercontent.com/17693547/56575230-d10b0a80-65f7-11e9-91da-d146d876f08c.JPG)

So I add `sort()` in test.py :
  ```
class PrefetchDataset(torch.utils.data.Dataset):
    def __init__(self, opt, dataset, pre_process_func):
    self.images = dataset.images
    self.images.sort()
    self.load_image_func = dataset.coco.loadImgs
    self.img_dir = dataset.img_dir
    self.pre_proces
    ...

I get an acceptable results with flip test:

Hope this can help you fix this bug.
@xingyizhou

[moothes]

Since adds sort() in test.py maybe not suitable for coco dataset, I move the sort() function to src/lib/datasets/dataset/pascal.py:

def __init__(self, opt, split):
    ...
    self.images = self.coco.getImgIds()       \\ line 49
    self.images.sort()
    ...

The results are the same.

[xingyizhou]

Hi,
Thanks for pointing it out! There is indeed a bug in converting the evaluation format. However in my machine, the image ids are sorted itself. I have fixed this by fixing the image id when saving them to the disk.

[liuchang8am]

Since adds sort() in test.py maybe not suitable for coco dataset, I move the sort() function to src/lib/datasets/dataset/pascal.py:

def __init__(self, opt, split):
    ...
    self.images = self.coco.getImgIds()       \\ line 49
    self.images.sort()
    ...

The results are the same.

This is indeed the cause. I can get proper mAP after adding sort(). Thanks a lot! Closing the issue.