Detr: will it make any sense to use zero v in the first decoder layer?

[rardz]

As in your code, tgt of the decoderlayer was firstly assigned with zeros, and use these zeros as v to calculate a new ouput with qkv attention operation, take the pre-norm forward part for example:

    def forward_pre(self, tgt, memory,
                    tgt_mask: Optional[Tensor] = None,
                    memory_mask: Optional[Tensor] = None,
                    tgt_key_padding_mask: Optional[Tensor] = None,
                    memory_key_padding_mask: Optional[Tensor] = None,
                    pos: Optional[Tensor] = None,
                    query_pos: Optional[Tensor] = None):
        tgt2 = self.norm1(tgt)
        q = k = self.with_pos_embed(tgt2, query_pos)
        tgt2 = self.self_attn(q, k, value=tgt2, attn_mask=tgt_mask,
                              key_padding_mask=tgt_key_padding_mask)[0]
        tgt = tgt + self.dropout1(tgt2)
        tgt2 = self.norm2(tgt)
        tgt2 = self.multihead_attn(query=self.with_pos_embed(tgt2, query_pos),
                                   key=self.with_pos_embed(memory, pos),
                                   value=memory, attn_mask=memory_mask,
                                   key_padding_mask=memory_key_padding_mask)[0]
        tgt = tgt + self.dropout2(tgt2)
        tgt2 = self.norm3(tgt)
        tgt2 = self.linear2(self.dropout(self.activation(self.linear1(tgt2))))
        tgt = tgt + self.dropout3(tgt2)
        return tgt

I mean, if it was the first decoderlayer, tgt was token-wisely zero, then tgt2 will be token-wisely same after the first layernorm, how will that make any sense to get weighed output from this tgt2? No matter what the q and k is, nothing but a featureless bias will be learned I think.

[szagoruyko]

Your understanding is correct. The first decoder self attention does not receive any data dependent inputs, so we pass zeros as inputs. It could be removed to save some parameters and compute, but we keep for simplicity.

We should add a comment explaining this in the code.

[rardz]

Your understanding is correct. The first decoder self attention does not receive any data dependent inputs, so we pass zeros as inputs. It could be removed to save some parameters and compute, but we keep for simplicity.

We should add a comment explaining this in the code.

As a matter of fact, not only in the first decoderlayer, but also all the subsquent layers, the self attention part can be skipped actually. Your choice is the 3-sublayered version, a little strange because such 3-sublayered decoder are more used in autogressive tasks like seq2seq. A 3-sublayered decoder will give a better performance?

[szagoruyko]

no, in subsequent layers self attention has meaningful data dependent inputs and needed for communication, so can't be skipped.

[rardz]

OK, never mind for my wrong understanding.

[fmassa]

Hi,

I believe we have answered your question, and as such I'm closing the issue, but let us know if you have any further questions.