Vision: How to feed negative samples during Faster R-CNN training

[kkirtac]

Hi all,
I have lots of non-annotated images in my training set, where there is no object of interest but there are couple other objects that should be interpreted as part of background. Is there any way I can provide background (negative) samples explicitly in my dataloder?
I tried to set a single fake bounding box with label zero for those non-annotated images, and set my num_classes as 3, i.e., I have 2 objects and background, and then performed transfer learning,

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, pretrained_backbone=False)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

But I received a crash at /torchvision/models/detection/roi_heads.py", line 34, in fastrcnn_loss
sampled_pos_inds_subset = torch.nonzero(labels > 0).squeeze(1)

I think this is happening because I have fed some images with only label zero, i.e., with no positive bbox.
Is there any workaround for that purpose?

[fmassa]

We currently do not support feeding images without annotations during training.
A workaround for that is to filter beforehand all the training images that do not have annotations, like we do in
https://github.com/pytorch/vision/blob/537f0df79c464983dc1f32b8697dbc42d3f4872b/references/detection/coco_utils.py#L106-L143

[rronan]

Hi,
Do you plan to support these negative samples during training? By filtering the images beforehand, I get a lot of false positive detections during inference, when the input image in empty.
Without this feature I need a second model to classify empty images, sharing the same backbone (or do you think of another workaround?).
Thanks

[ferna11i]

Hi @fmassa ,

Thanks for pointing out the code which we can use to filter the images that do not contain the object of interest.
Out of curiosity does this mean, the images in the test/validation sets that have only negative samples i.e. no objects will also be elimiated from the test/validation set?

[iantimmis]

+1 Would love a solution for this as well.

[fmassa]

This will be fixed once https://github.com/pytorch/vision/pull/1911 gets merged

[FiReTiTi]

Hi,
I also have the same issue. I've filtered the empty (no object to detect) images, but it's highly problematic as it create an over-segmentation.
Indeed, I have tricky images containing object like patterns that I cannot use for training, and as a result these false object are all detected during testing.
Is there any work-around while waiting for the fix?

[fmassa]

@FiReTiTi the fix (for Faster R-CNN for now, not yet for Mask R-CNN, Keypoint R-CNN) has been merged in master

[FiReTiTi]

@fmassa good to know, thanks.

[fmassa]

This is fixed now for Mask R-CNN and Keypoint R-CNN, thanks to https://github.com/pytorch/vision/pull/1911 and https://github.com/pytorch/vision/pull/2069, and will be present in the next release of torchvision next week

[FiReTiTi]

Thank you very much!

[ferna11i]

Thank you!
Great work!

[FiReTiTi]

@fmassa: I haven't seen a new torchvision release. Pip still propose version 0.5.0

[fmassa]

@FiReTiTi the release will be today