Detr: Question about residual connection

[mli0603]

Hi, thank you so much for your work!

I have one question about the self-attention implementation. In the paper Attention is All You Need, the residual connection is made upon input embeddings + positional encoding as shown in the figure below.

In the paper, the figure seems to match above as shown in the paper below.

However, in the code, it looks to me that the residual connection is made upon input embeddings only (the src), also see figure below. Is this a mistake or there is a reason for such modification? Thank you!

[szagoruyko]

in the code you pointed the positional encodings are added in the first line of the function, see https://github.com/facebookresearch/detr/blob/master/models/transformer.py#L154 can you elaborate?

[mli0603]

@szagoruyko The residual connection is what I refer to. If you look at line 157, the residual connection is made upon the original input instead of the position encoded input. I.e., currently in the code it is

src = src + self.dropout1(src2)

while I think both the paper and the original transformer paper describe it as

src = q + self.dropout1(src2)

where q is the input with position encoding.

Does this clear things up a little bit? Sorry if the previous description was too confusing.

[mli0603]

@szagoruyko Another issue I see is that the positional encoding is added to the src for every encoder/decoder layer in the for loop (https://github.com/facebookresearch/detr/blob/master/models/transformer.py#L76) by with_pos_embedding (https://github.com/facebookresearch/detr/blob/master/models/transformer.py#L154). Is this necessary? In the paper, the positional encoding is only added once, which makes more sense to me.

[szagoruyko]

@mli0603 see the section 4.2 'Importance of positional encodings' in the paper for the architectural choices on where to pass positional encodings. There is also Table 3, in which the second row corresponds to vanilla Transformer where we pass positional encodings once at transformer input, the one you are referring to (also used in the demo colab). As we explain in text, passing encodings in attention directly leads to a significant performance boost.

[mli0603]

@szagoruyko Thanks for your comment on pos encoding! I am sorry I completely misunderstood that paragraph in the paper. It now makes sense.

I still wonder if there is any explicit design choices on residual connection between connecting from image featues src directly and from position encoded features q (my first question above). Maybe it is just too minor to make a difference? I really appreciate it.