Vision: COCO evaluation can fail for custom datasets which inherit from `torchvision.datasets.CocoDetection`

[ndrplz]

🐛 Bug

The COCO evaluation pipeline in references/detection (also used in the object detection finetuning tutorial here) can fail if the custom dataset provided inherits from torchvision.datasets.CocoDetection.

To Reproduce

In the evaluation function the get_coco_api_from_dataset is called to guarantee that the provided dataset has the right COCO interface.

Internally, this runs convert_to_coco_api. However, this is not called if the dataset at hand is an instance of CocoDetection already (see here).

However, the dataset can be a custom dataset inheriting from CocoDetection but featuring some different details.

Let's say I have a CustomCocoDetection dataset where in the getitem images are downscaled to a fixed resolution and bounding boxes are rescaled accordingly (my case). This will not enter the convert_to_coco_api function causing bounding boxes to have different sizes with respect to what the evaluator expects. So the evaluation will run but will meaningless results.

Expected behavior

I would expect the check here to check the actual type. That is

if type(dataset) == torchvision.datasets.CocoDetection:
    return dataset.coco
return convert_to_coco_api(dataset)

this will avoid fishy bugs if someone subclasses CocoDetection modifying some detail in the child.

Environment

Collecting environment information...
PyTorch version: 1.7.0+cu101
Is debug build: True
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 450.80.02
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.0
[pip3] pytorch-lightning==1.0.5
[pip3] pytorch-lightning-bolts==0.2.2
[pip3] torch==1.7.0+cu101
[pip3] torchaudio==0.7.0
[pip3] torchvision==0.8.1+cu101
[conda] Could not collect

Additional context

Thanks for the great work :)

cc @pmeier

[datumbox]

Hi @ndrplz, thanks for reporting the issue.

I understand that this can be limiting for your use case. Unfortunately changing

isinstance(dataset, torchvision.datasets.CocoDetection)

to

type(dataset) == torchvision.datasets.CocoDetection

is going to break things because the torchvision.datasets.CocoDetection class is inherited by other classes within the repo (for example here). I wonder if your problem can be resolved by overwriting the default self.coco parameter in the constructor of your class with self.coco = convert_to_coco_api(self) (make sure this is the last statement in your constructor). This can also happen outside of your constructor if you prefer.

Let me know what you think.

[ndrplz]

Hi @datumbox thanks you for your answer.

On my side I fixed it exactly in the way you mentioned.

It just took me a while to understand what was going wrong at the beginning, because there were no errors and all results were simply zero. If that is something that is not easily fixable, fine. At least there will be this discussion for posterity. I see that is kind of an edge case

[datumbox]

Awesome, thanks for confirming, glad it worked!

@pmeier are you happy to close this or you want to leave open and have a look on the future?

[pmeier]

IMO, we can close it. The use-case described by @ndrplz while valid is uncommon. Given that this has an easy fix, I don't think we need to act on it. We can always re-open this, if this comes up again.