Sentence-transformers: Can u tell more details about smart-batch strategy?
Can u tell more details about smart-batch strategy?
As far as I know, the input length of BERT is fixed, I don't understand how it works.
wzl789139
All 7 comments
Hi,
usually input to BERT is batched, e.g., you present 16 sentences at the same time to BERT. However, these 16 sentences can have a different number of tokens.
In order to create a matrix out of them, they must have all the same length. In other training codes, all sentences in training are padded to the same length, to the longest sentence in the training data. For example, all sentences are padded to 100 tokens.
With smart batching, sentences are only padded to the longest sentence in a batch, e.g. to the longest sentence in a batch of 16 sentences. So instead of being padded to 100 tokens, they might only be padded to 30 tokens.
This drastically improves the speed, as BERT has a quadratic runtime with the input sentence length.
nreimers
on 26 Sep 2020
AFAIK when training a BERT language model (with 512 max token length for example) the training algorithm picks more sentences and concatenates them until almost 512 is reached. So you should never have the situation of only having 30 tokens in real life.
PhilipMay
on 26 Sep 2020
Is the implementation of this function related to PyTorch's dynamic graphs？
Thanks a lot , I think this is a very good function, cause BERT is a very computationally complex model.
wzl789139
on 27 Sep 2020
It is implemented in this function:
https://github.com/UKPLab/sentence-transformers/blob/v0.3.6/sentence_transformers/SentenceTransformer.py#L356
It is not necessarily linked to Pytorch dynamic graphs, it has more to do how to generate the mini batches. This function applies padding on a per mini-batch basis, while the standard (e.g. on huggingface trainer class) is to pad all sentences in the train dataset to the same length.
nreimers
on 27 Sep 2020
hi,
smart batching is just used for finetuning,
or it is used in indexing the corpus by training model too?
ReySadeghi
on 7 Oct 2020
Hi @ReySadeghi
It is also used for encoding: Sentences are first sorted by length, and then only the minimal needed padding is applied.
nreimers
on 7 Oct 2020
I think the greatest value of this function is online prediction,which requires a very short response time.
wzl789139
on 8 Oct 2020
Related issues
zishanmuzeeb
·
4Comments
PhilipMay
·
5Comments
serge-sotnyk
·
3Comments
AMChierici
·
5Comments
hetianbaicnn
·
5Comments
Most helpful comment
Hi,
usually input to BERT is batched, e.g., you present 16 sentences at the same time to BERT. However, these 16 sentences can have a different number of tokens.
In order to create a matrix out of them, they must have all the same length. In other training codes, all sentences in training are padded to the same length, to the longest sentence in the training data. For example, all sentences are padded to 100 tokens.
With smart batching, sentences are only padded to the longest sentence in a batch, e.g. to the longest sentence in a batch of 16 sentences. So instead of being padded to 100 tokens, they might only be padded to 30 tokens.
This drastically improves the speed, as BERT has a quadratic runtime with the input sentence length.