Sentence-transformers: Input pair length in Cross Encoder

[Davin-debug]

Thank you for your useful and outstanding work!

I have a question when I use the Cross Encoder.

The input of Cross Encoder is a sentences pair or texts pair. And the pair length is max_length= 512.

So, when the length of two texts in the pair are both over 512 (after adding [cls] and [sep] token), the truncation method is only keep the first 510 tokens of text1 and drop all the tokens of text2 OR keep the first 255 tokens of both the two text and add a [SEP] token between them.

Additionally, if the truncation method is the previous one, I want to achieve the latter method，what will you recommand me to do?

Thanks a lot!

[nreimers]

Hi @Davin-debug

the cross-encoder uses the 'longest_first' truncation strategy from Hugginface tokenizer:
https://huggingface.co/transformers/preprocessing.html

"This will truncate token by token, removing a token from the longest sequence in the pair until the proper length is reached."

So if text A is e.g. 260 tokens and text B is 500 tokens, A will be truncated to 255 tokens and B will also be truncated to 255 tokens. If text A is only 200 tokens, text B will be truncated to 310 tokens.