Detr: Get object features from inner layers for Object Tracking

[ptamas88]

Hi,
I'm pretty new to pytorch, i have been using TF and keras until now, but this model's results are very promising!
I would like to incorporate this models detections with a DeepSort Tracking algorithm, and the tracking needs some object features with an arbitrary vector size (only important thing is that the vector size has to match for each frame).

My question is: how can I get this 256 or 512 size feature vector for each detected object from the model without rewiting the whole model class. In the first colab I have seen, and get somewhat familiar with the hooks but some help would be appreciated.

Right now i am retrieving this hook, but very uncertain about it:
hooks = [model.transformer.register_forward_hook(lambda self, input, output: transformer.append(output[0]))]

Thanks in advance,
Thomas

[fmassa]

Hi,

I would recommend using the hooks as you showed, as it's the least intrusive way.

If you would like more information on how to extract intermediate activations for a model in PyTorch, check this proposal https://github.com/pytorch/pytorch/issues/21064

I believe I've answered your question, and as such I'm closing this issue but let us know if you have further questions

[alcinos]

Here is the correct hook

object_embeddings = []
hook = model.transformer.register_forward_hook(lambda self, input, output: object_embeddings.append(output[0][-1]))

# propagate through the model
outputs = model(img)

hook.remove()

object_embeddings = object_embeddings[0]
print(object_embeddings.shape) # should be bs x 100 x 256