Detr: Multi-oriented or quadrilateral object detection?

[zobeirraisi]

Right now DETR because of rectangle bboxes of COCO-dataset produces rectangle-bboxes outputs, if we had quadrilateral bboxes (8 points), which parts of the architecture must be modified to output a quadrilateral shape bboxes?

[fmassa]

Hi,

You would need to change the number of prediction point https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/detr.py#L38, the loss for the bounding boxes https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/detr.py#L143-L162, the postprocessing https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/detr.py#L277-L282 and assignment cost for the boxes https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/matcher.py#L71-L74

Also note that the GIoU loss was very important for DETR, so you might need to find an equivalent type of loss if using different polygons.

I believe I've answered your question, and as such I'm closing this issue, but let us know if you have further questions