Farm: Utilize all max_seq_len tokens in Language Model adaptation or training from scratch

[Timoeller]

While training from scratch or adjusting Language Models to other domains (LM adaptation/ LM finetuning), FARM does not always use max sequence length number of tokens.
FARM only fetches two sentences, not accounting for their length. If these two sentences are shorter than max seq len tokens computation is wasted by zero padding.

Describe the solution you'd like
We need to avoid zero padding by filling up the whole sequence with text. So there is a need to fetch more than just two lines/sentences.

Additional context
Already implemented in Hugginface, see TextDataset vs LineByLineDataset (just below).

[Timoeller]

Here is the code where Farm fetches only two sentences and not two sequences that fill up all of max_seq_len.

[Timoeller]

I know @PhilipMay and @tripl3a were particularly interested in this feature.

[PhilipMay]

I know @PhilipMay and @tripl3a were particularly interested in this feature.

Yes - thanks for this implementation!

[PhilipMay]

One interesting question for me is: Do you see any more substantial differences between Hugging Faces and FARM? For me this was one very important difference (that is "fixed"now).

[Timoeller]

Honestly Huggingface doesn't do next sentence prediction NSP any longer in their script.

The changes add NSP similar to the style of the original Bert repository.
Additionally whole word masking is included by default.