Turicreate: Object Detection reduce false positives.

[tagy]

HI, I'm using an object detector model and it works well and correctly identifies the desired object in most cases.

However it also wrongly identifies similar but different objects often with a high confidence.

I'm wondering what is the best strategy to improve this? Can I include images with no tags that don't have any of the desired object in them? A negative sample? Or should I start to tag similar objects so hopefully they will be identified as that and not the desired object?

Thanks.

[TobyRoseman]

@tagy - I think both solutions you mention should improve things. Just adding images of the similar objects without any annotation is probably easier. I would probably start with that approach.

Let me know how it goes. Also let me know if you have any more questions

[tagy]

Thanks for your reply Toby, I will try as you say and let you know.

Can I ask about the confidence of an observation, when I get a VNClassificationObservation and the label I can inspect the VNConfidence which is normally around 0.9 - 1.0

I'm confused how this confidence score is related to the confidence threshold you can set in the model with a MLFeatureProvider eg:

open var values = [ "iouThreshold": MLFeatureValue(double: 0.45), "confidenceThreshold": MLFeatureValue(double: 0.5) ]

Increasing the confidenceThreshold here does make the model stricter, but the VNConfidence is always close to 1.0

How do these number relate? is the confidenceThreshold something that happens before Non-maximum suppression?

Presumably I can set the confidenceThreshold low and apply a filter once I have the VNConfidence? Or I can set a higher confidenceThreshold and not filter using the VNConfidence? Would both give similar results? Thanks.

[shreyajain17]

@tagy - A neural network returns fixed number of predictions which are first filtered by the confidence threshold and then only the boxes left after this filtering with confidence threshold are subjected to Non Maximum Suppression (NMS). Essentially setting higher threshold would affect what the input to Non maximum suppression. So depending on how different your input to NMS is based on your threshold, the output of the NMS could vary.

I think adding more data for the classes which are similar looking as well as more negative data would be great.

[tagy]

Just to confirm I have tried adding images without any annotation and it does help in reducing false positives thanks.

But also I wasn't understanding the confidence levels, the sample Apple live preview app shows a confidence but it is the confidence of the top label (and nearly always 0.9+). Not the confidence of the observation. Getting the confidence of the actual VNRecognizedObjectObservation now matches what I'm getting in the predict function, and so I can more easily set a threshold. Someone else mentioned it here:

https://github.com/apple/turicreate/issues/1314#issuecomment-536052657

[TobyRoseman]

@tagy - I'm glad both those things worked out for you. I'm going to close this issue, but let us know if you have any more questions.