Icevision: Question on null annotations in COCO

[shimsan]

Hi,

My custom dataset has null annotations, images without bboxes are represented in the "images" tag, and all the bboxes are in the "annotations" tag.

Now, when I load using the coco parser, it is removing all of these null annotations.

This might be more of a philosophical question, but what do I do if I want to keep my "negative samples" while training. Because, in my case, the background also can change without any "objects" to detect, and I want my model to know that.

Cheers!

[lgvaz]

That is a hard question indeed hahaha

At least the models we currently support mask/faster rcnn and efficientdet cannot handle null annotations. I would actually be interested in a model that could do so (if there are any).

It's hard because there is really nothing to learn on those images, what you can do is distribute some random bboxes onto your image and mark those as background (label 0), but I'm not sure that is going to help much

@oke-aditya any thoughts?

[oke-aditya]

@shimsan Great Question. Let me answer it in detail as this doubt can occur to most people who try object detection with any package or repo for that matter.

Problem: - You have no objects to detect, it is an example of having no classes that you have to detect. E.g. you are training a face detector and you pass an image of say tiger/lion. You do not want to draw any bounding box in this image.

Object-detection-theory: -
We never calculate metrics on false negatives. As there is no notion of True negatives in object detection. E.g. You draw a bounding box where there is nothing and say it as negative i.e. there is no object. Now there can such multiple boxes possible in every image. We cannot calculate such boxes. These are simple True negatives. You cannot evaluate your model on these objects or try to train on these bounding boxes as they make no sense.

Refer here and here

Problem of Null annotations : -

But you want to still try and train a face detector by showing it images of lion/tiger and saying it is NULL. This is called Negative sampling. Yes you can do that. i.e. you do not contain any positive samples but want the model to ensure that it does not confuse these type of samples. Torchvision started supporting these from release 0.6. It is fairly simple to support too.
Have a look here
Just reposting the highlight of Faster RCNN

It is now possible to feed training images to Faster / Mask / Keypoint R-CNN that do not contain any positive annotations.
This enables increasing the number of negative samples during training. For those images, the annotations expect a tensor with 0 in the number of objects dimension, as follows:

target = {"boxes": torch.zeros((0, 4), dtype=torch.float32),
          "labels": torch.zeros(0, dtype=torch.int64),
          "image_id": 4,
          "area": torch.zeros(0, dtype=torch.float32),
          "masks": torch.zeros((0, image_height, image_width), dtype=torch.uint8),
          "keypoints": torch.zeros((17, 0, 3), dtype=torch.float32),
          "iscrowd": torch.zeros((0,), dtype=torch.int64)}

How does this solve the Problem: -

What it does is simply makes Bounding box and mask to (0,0,0,0) that is nowhere in the image.
The label is also 0. Which is background class for Faster RCNN / Mask RCNN. And makes iscrowd to 0 too.

This effectively tells the detector. "Look there is a bounding box at 0,0,0,0 with the label as background".
Since we ignore the background class while you are deploying/detecting in test time. (Why would you detect background in test time?) This makes such negative samples too become background.

Solution: -
Simply pass these negative sampled images as background class (it is 0 for F-RCNN may be different for different models), and make the bounding box, areas, masks, crowds as 0,0,0,0. This would work for most models, e.g. FRCNN (any backbone or FPN) Efficient-Det, Retinanet (soon). Just take care of this preprocessing while parsing and you are done. Background class may wary from model to model so just be careful for that (we can't take care of this as it is left to the researcher (person who implemented in research paper) to handle background class, we will simply throw an error)

Mantisshrimp support: -
For mantisshrimp since we support F-RCNN from torchvision. Yes, we support this feature too natively. Just preprocess data as mentioned above.
For efficient-det @lgvaz can you have a check from dataset code here It does make bboxes 0 from line 98-102. Please have a look and do a post here.

[shimsan]

Thanks for the awesome answer, @oke-aditya!