Detr: Image Sequence with DETR (Object Tracking)

[RajashekarY]

❓ How to do Object Tracking using DETR

as Transformers are good for long sequence models in NLP how will detr perform on object tracking sort of tasks

• giving an input video and giving an bounding box around the object to be tracked (SOT - Single Object Tracking)

• let the detector be detecting objects of specific class and identifying them with unique ID through out the video until object leaves the frame (MOT - Multi-Object Tracking)

I have seen Re3 based on LSTM for object tracking as Transformers are better compared to LSTM can idea of object tracking suitable for Transformers.

I think @alcinos and @fmassa can help me out regarding this question

[RajashekarY]

I have read the paper couple of times for now I think the place where you are applying panoptic segmentation is the place to implement tracking part by freezing the weights

Correct me if I'm wrong
thanks

[fmassa]

Hi,

Yes, a variant of DETR can be used for tracking. We have not tried this, but I see no reason why it wouldn't be a good thing to try.

[alcinos]

Hi @RajashekarY

As @fmassa mentioned, tracking is an open research direction, and DETR can indeed be a good starting point for a tracking model.
However, this code-base doesn't support it at the moment, nor in the near future, and the github issues are meant to track the defect of the current code. For research questions, please reach out via email. As such, I'm closing this issue, but feel free to reach out if you have further concerns.