Models: Object detection : hard negatives.

[OverFlow7]

Hello!

I am currently using object detection on my own dataset, for some of my classes, i have a lot of false positives with high scores (>0.99, so having a higher score threshold won't help).

I know there is already some hard negative mining implemented ,but would it be possible to have a feature where one could add hard negatives examples to the training ?

Let's say we want to detect object A, and we know that object A look a lot like object B, but we are not interested in detecting, object B , in that case we could add images of object B to training (without any bounding boxes) in order for the network to distinguish between A and B.

[cipri-tom]

Depends on what network you are using, I guess.

For example, Faster R-CNN considers as negative boxes those which have IoU under a certain threshold (0.3 by default, I think). And then it takes as many negative examples as positive ones (this is configurable, I think).
So basically, if you have your object B in images with A as well, and it is not labeled, it is some chance that it will be considered as a negative.

To be more certain, I guess you could use object _classification_ (detecting both A and B) and then only keep the objects A.
This should work a bit better, because it tries to regress the class difference to be as big as possible.

[skye]

@jch1 @tombstone @derekjchow @jesu9 @dreamdragon

[tombstone]

@OverFlow7 You can already do this as @cipri-tom mentioned. Can you specify the model you are working with ?

[OverFlow7]

Hello thank you for your answers.

I am using Faster RCNN with inception v2. What cipri-tom mentioned is indeed a good idea that i will be trying, but it still require either for the hard negatives to be in the same picture as the object i'am trying to detect, or hand labelling the hard negatives .

[niyazpk]

Hello, I guess I have a similar/same problem. Using the InceptionV2-SSD model, and the model gives me great precision and recall on test set that contains at least one of the classes that I am interested in. But as soon as I add some images which do not contain any of my classes, I start getting false positives. Tried training with these images and provided an empty array in place of the truth data boxes. This did not seem to help.

It would be nice to get answers to two questions (1) Is this an existing problem in the code, or am I doing something wrong here. (2) Is there a recommended solution other than adding another classifier? Would something like modifying the loss functions to take these examples into account work?

Thanks!

[tombstone]

@niyazpk Sorry, I should have been more specific. You need to set min_negatives_per_image to a non zero number in the config for the model to sample boxes from purely negative images: https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/ssd_inception_v2_coco.config#L118 . Few 10s might be a good number to choose.

@OverFlow7 , You can have purely negative images and faster_rcnn models will sample from anchors from them. We use sampling in both stages with a certain ratio of positives to negatives. If the sampler can't achieve that ratio ( in purely negative images), it fills the batch with negatives. See https://github.com/tensorflow/models/blob/master/research/object_detection/core/balanced_positive_negative_sampler.py#L18

[tombstone]

I feel this discussion might benefit others if do it on stackoverflow. Can you please move the discussion there if my response does not sufficiently answer your question alreadt. I will close this issue for now.

[niyazpk]

@tombstone Thank you! This was helpful.

[OverFlow7]

Thank you @tombstone .

But how do you create tfrecords with purely negative images? what do you put in the .xml?

[rickragv]

@tombstone
how do you create tfrecords with purely negative images? what do you put in the .xml?

[liangxiao05]

@tombstone
Hi,can you please give more details about the trouble @OverFlow7 has met
how to create tfrecords with purely negative images ,are there no xmls for these images?
I think many people are puzzled by this. Looking forward to your reply~

[Kuldeep-Attri]

Hello Everyone, I am new with Faster-RCNN program and I am trying to extract negative example from my training dataset (like getting the bounding box of the regions with label == 0 i.e. no object), I am reading the code but I am still confused about how to do this. Is it possible to extract negative example (a part of an image) from the training dataset? Any help would be appreciated. Thank you!!!

[cipri-tom]

@Kuldeep-Attri what do you mean extract ? Please use StackOverflow to ask questions (and give good examples if you want answers 😃 )!

[Kuldeep-Attri]

@cipri-tom I am sorry if I was not clear, what I mean by extract is that can we save the negative examples as an image(like .jpg file) after knowing the bbox of the negative examples and the original image? I want to store all background images in a folder. Thank you!!!

[in-skyadav]

Hello, I am also getting the similar error while Using the "ssd_mobilenet_v1_pets" model, and the model gives me good accuracy on positive example, but as soon as I add some images which do not contain any of my classes, I start getting false positives( it gives face as thumb with .99 accuracy).
Please explain me the role of : max_negatives_per_positive and min_negatives_per_image also how to modify the default value of num_hard_examples: 3000. currently I am training for single class and all the input data i gave contains my target class. how to input negative data ?

[guruvishnuvardan]

@cipri-tom @tombstone @overflow7 - Please help us resolve this issue, as we have made several attempts for the past 10 days, pls let us know, where we are failing, pls find the link below:

https://stackoverflow.com/questions/52696408/tensorflow-object-detection-api-detecting-the-humans-not-wearing-helmet/52698252?noredirect=1#comment92355684_52698252

We are attempting to detect people with out helmet in motor bikes.

[KKatya]

I was able to train the ssd_mobilenet_v1_coco model with quite good accuracy. It successfully detects and classifies 5 different classes and does not have any issues with negative examples (no false positives with threshold value 0.5).

I used VoTT tool to prepare Pascal VOC dataset for training.
Beside images with classes i added a bunch of empty images (without any bounding boxes).
This is the xml for empty image generates by this tool:

<?xml version="1.0"?>
<annotation verified="yes">
  <folder>Annotation</folder>
  <filename>20171203_211505_10730</filename>
  <path>C:\Tensorflow\models\research\object_detection\VOCdevkit\VOC2012\JPEGImages\20171203_211505_10730.jpg</path>
  <source>
    <database>Unknown</database>
  </source>
  <size>
    <width>768</width>
    <height>432</height>
    <depth>3</depth>
  </size>
  <segmented>0</segmented>
</annotation>

Hope this helps!

[joskaaaa]

@KKatya Did you get that XML & image data into the TFRecord format for training?

[KKatya]

@joskaaaa Yes, i used create_pascal_tf_record.py to create TFRecord files for training and evaluation.

[dkashkin]

@KKatya can you please share your hard_example_miner section of pipeline.config? I am a bit confused about using negative images in training because Zhichao Lu from Google recently said the following:

"For your negative images, is it true that you have zero groundtruth boxes? We currently don't support explicit negative image in training so adding those images don't help at all."

This comment relates to training an SSD object detector: https://stackoverflow.com/questions/52696408/tensorflow-object-detection-api-detecting-the-humans-not-wearing-helmet/52698252

Anyways if you got good results by adding the negative images to your training set that sounds awesome... I'd love to know how many negative vs positive images you are using?

[CasiaFan]

@in-skyadav According to explanation in HardExampleMiner during loss calculation:

max_negatives_per_positive: maximum number of negatives to retain for each positive anchor. By default, num_negatives_per_positive is None, which means that we do not enforce a prespecified negative:positive ratio. Note also that num_negatives_per_positives can be a float (and will be converted to be a float even if it is passed in otherwise).

min_negatives_per_image: minimum number of negative anchors to sample for a given image. Setting this to a positive number allows sampling negatives in an image without any positive anchors and thus not biased towards at least one detection per image.

In your case when all input samples contain target class, just keep min_negatives_per_image 0, unless you want to add some pure background samples without any foreground target. If in that case, set a 10s number is enough.

[AbdullahBahi]

@joskaaaa Yes, i used create_pascal_tf_record.py to create TFRecord files for training and evaluation.

Are you sure that this script created TFRecords that contain the negative images? i used a similar script that creates TFRecords from .CSV files that is created by extracting data from XML files -I used verify feature in LabelIMG to create XMLs for negative images- and it created the TFRecord successfully but it had only the data of the images with labels, it didn't even see the empty images.

[sampathkumaran90]

Hi, I am facing similar issue. When i set min_negatives_per_image to 0 and not set any negative images, i get prediction correct for positive images but getting a lot of false positives. When i add a bunch of negative images say 50% positive and 50% negative and set min_negatives_per_image to 10, the results are not good. I am getting more false negatives as well as false positives. Anyone could help me on this issue... I am using ssd mobilenet v2 or finetuning

[sainisanjay]

@niyazpk Sorry, I should have been more specific. You need to set min_negatives_per_image to a non zero number in the config for the model to sample boxes from purely negative images: https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/ssd_inception_v2_coco.config#L118 . Few 10s might be a good number to choose.

@OverFlow7 , You can have purely negative images and faster_rcnn models will sample from anchors from them. We use sampling in both stages with a certain ratio of positives to negatives. If the sampler can't achieve that ratio ( in purely negative images), it fills the batch with negatives. See https://github.com/tensorflow/models/blob/master/research/object_detection/core/balanced_positive_negative_sampler.py#L18

What ratio we should follow to set min_negatives_per_image and max_negatives_per_positive from purely negative samples. For example, if i have a total of 41K training images (5 classes) and 1724 images are purely negative samples, then what should i suppose to set min_negatives_per_image and max_negatives_per_positive in the config file.