Detr: How to improve DETR on small objects (mask detection) ?

[jeromen7]

Hello,

I am currently working on a mask detector using DETR and it is working pretty well :

But the results are very poor when it comes to detecting masks in a large crowd. Here is an example :

As you can see, the detection boxes are very random, and the results are pretty bad.
How can I improve that ?

• I was thinking about dividing the image in 4 parts, zoom in on each part and detect the masks, and the reassemble the image but it seems complicated and it will increase the inference time a lot.

• The other solution I imagined was getting a lot of crowd images like the one above, labelise them and add them to my dataset. But if DETR is by definition not working well on small objects like it is said on the paper, maybe this is not a good solution...

Would you have any suggestions that my help ?
Thank you all !

[lessw2020]

Hi @jeromen7
Great work on the mask detector!
To your question, dividing the image into quadrants and doing inference on each would play to detrs strength with larger objects and would be the best approach to try imo.
I do a lot of subset, rotate, rescale stuff for my work with detr and once you get the manipulations down it works really well.
You might also try with ciou loss, but that would be more speculative how much it would help with small objects. The implementation is in my repo though so easy to try.
Hope that helps

[Epiphqny]

@jeromen7 Have you increased the number of queries of the model? It seems there are more than 100 objects in the test image.

[woctezuma]

I have roughly counted the number of green boxes, and it is very close to 100. I counted at least 80, likely more than 90.

I was thinking about dividing the image in 4 parts, zoom in on each part and detect the masks, and the reassemble the image

Do it. Or train the model from scratch with more than 100 query slots

The latter sounds more complicated, especially if you have to use 1000 query slots, or who knows, 10,000 query slots.

References:

Note that if your dataset has a substantially different average number of objects per image than coco, you might need to adjust the number of object queries (--num_queries) It should be strictly higher than the max number of objects you may have to detect, and it's good to have some slack (in coco we use 100, the max number of objects in a coco image is ~70)

https://github.com/facebookresearch/detr/issues/9#issuecomment-635357693

I'd recommend trying to fine-tune the class head, while starting from a pre-trained encoder/decoder. You'll have to keep the 100 queries if you do that, but unless you're after very marginal speed improvement it shouldn't hurt.

https://github.com/facebookresearch/detr/issues/9#issuecomment-639456002

Unless you literally have 700 items in each image, do not change num_queries and keep it at 100, this will give you 100 proposed boxes for each item. Changing num_queries will result in retraining the whole transformer, which is costly.

https://github.com/facebookresearch/detr/issues/9#issuecomment-640480904

If you're fine-tuning, I don't recommend changing the number of queries on the fly, it is extremely unlikely to work out of the box. In this case you're probably better off retraining from scratch (you can change the --num_queries arg from our training script).

https://github.com/facebookresearch/detr/issues/9#issuecomment-636407752

500 queries is quite high, it's fairly possible that the default hyper-parameters are not optimal for this setting.

https://github.com/facebookresearch/detr/issues/9#issuecomment-645521166

[jeromen7]

Thank you all for your help !
I will try the solution to split the image in quadrants, and I will come back to tell you if it worked.

[jeromen7]

Here is the result :

As you can see it works much better, still not perfect but it definitely improved. However it also takes more inference time, as expected ...
I am closing the issue, thank you again @lessw2020 @Epiphqny @woctezuma !