Fairseq: Incremental decoder

[Johnzdh]

Does anyone know the source of the "incremental decoder" ? Thanks a lot!

[myleott]

Incremental decoding is a special mode at inference time, where the model only receives a single previous output and must produce the next output (i.e., incrementally). Thus the model must cache any long-term state that is needed about the sequence, e.g., hidden states, convolutional states, etc.

Decoders can declare that they are "incremental" by implementing the FairseqIncrementalDecoder class: https://github.com/pytorch/fairseq/blob/21b8fb5cb1a773d0fdc09a28203fe328c4d2b94b/fairseq/models/fairseq_incremental_decoder.py#L11-L12

The beam search code is also aware of IncrementalDecoders and has some special logic to reorder the buffered state in each child module as beam search progresses. For example:
https://github.com/pytorch/fairseq/blob/21b8fb5cb1a773d0fdc09a28203fe328c4d2b94b/fairseq/sequence_generator.py#L245-L247

[Johnzdh]

@myleott Get it! Thanks a lot.