Detr: Why use nn.Embedding for object query in the first decoder layer?

[turboxin]

Hi all, thanks for sharing this great work!

As indicated in https://github.com/facebookresearch/detr/issues/36#issuecomment-637559227 , object querys are to zeros in the first decoder layer. The way you implemented this is by using nn.Embedding :
self.query_embed = nn.Embedding(num_queries, hidden_dim)
https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/detr.py#L39, then pass the weight to transformer:
hs = self.transformer(self.input_proj(src), mask, self.query_embed.weight, pos[-1])[0]
https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/detr.py#L65,
while in decoder, zero tensor like the embedding weight are finally feed into first decoder:
tgt = torch.zeros_like(query_embed) memory = self.encoder(src, src_key_padding_mask=mask, pos=pos_embed) hs = self.decoder(tgt, memory, memory_key_padding_mask=mask, pos=pos_embed, query_pos=query_embed)
https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/transformer.py#L55.

My question is why bother to use nn.Embedding, since after all zero tensor are passed to decoder, learnable parameters of embedding doesn't seem to be used. Please correct me if my understanding is wrong.

[fmassa]

Hi,

We do feed the learnable parameters to the model, but as "positional encoding". In https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/transformer.py#L57-L58, if you keep track of query_pos, you'll see that in https://github.com/facebookresearch/detr/blob/5e66b4cd15b2b182da347103dd16578d28b49d69/models/transformer.py#L243 we add query_pos to the "zeros" vector from the first layer.

The reason why we zero it is just to avoid passing twice the same vector in the first layer, but we could have avoided that as well and it would work.

I believe I have answered your question, and as such I'm closing this issue but let us know if you have further questions

[turboxin]

Sorry I missed that. Thank you for clarifying this for me!