Opennmt-py: About MLP (or Bahdanau) global attention

[jihunchoi]

As far as I know, one of the key differences between Luong- and Bahdanau-style attention mechanism is the computation path, as stated in Luong et al's paper (page 3 or 1414).

According to the paper, Luong-style attention does not require the previous decoder state in computing attention weights.
Thus, when teacher forcing is used for training, one can pre-compute all decoder hidden states and then compute attention weights of all timesteps at once.

In contrast, in the Bahdanau-style attention, the alignment function (the one in page 12--14 of Bahdanau et al's paper) takes the previous decoder state as input, and therefore it is impossible to compute all attention weights using a single operation (and RNN computation should be unrolled).

However, according to the code, computation paths of the two seem to be the same.
Since I'm not aware of the most recent NMT trends, this could be a trivial question; but I want to make it clear.

[yoonkim]

Hi. It is true that with the bilinear (i.e. Luong) attention, we can pre-compute the decoder hidden state and calculate the attention matrix in one step. However, Luong also proposes "input-feeding" which uses the context vector from the previous decoding step as input to the current decoding step. This prevents the pre-computation strategy.

Hope this clarifies things!

[jihunchoi]

Oh I didn't know there exists InputFeedRNNDecoder class. Then it all makes sense!