Detr: NMS vs. bipartite matching

[eslambakr]

Thanks for your fantastic work.

I have a question regarding the NMS and bipartite matching loss.
You have studied in the paper the effect of applying the NMS on the output of the transformers which means for me you are applying NMS after bipartite matching, but you didn't study the impact of each one alone (I don't know is this a valid study? or not as you will need to change significant changes in DETR architecture)

But at least could we compare it from an inference time perspective?
I mean, is using bipartite matching is faster than using NMS as regular object detectors have done?

What are the benefits of replacing normal NMS by bipartite matching, as from accuracy wise it seems there is no significant impact?

[alcinos]

Hi @eslambakr
Thank you for your interest in DETR.

I think you may be confusing things here. Bipartite matching and NMS are not interchangeable part of the architecture, they serve completely different purposes:

• The NMS is useful mostly at inference time, to avoid double predictions. It is especially useful in models that are not trained to output a set and thus don't have a notion of cardinality (eg Faster-RCNN, Yolo, SSD, retinanet, ...)
• The bipartite matching is used at training time, it is a way to train a set based model: we compute a one-to-one matching between the bounding boxes of the ground truth and the boxes predicted by the model, and then supervise each predicted box to be closer to the ground truth box it was matched to.

In a way the NMS is merely a post-processing while the bipartite matching is the core of what makes our model trainable.

I believe I have answered your question, and as such I'm closing this. Feel free to reach out if you have further concerns.