Haystack: Configuration of DensePassageRetreiver

[mathieudumayet]

Question
Hi,
I am new to all of this, so I have a question regarding max_seq_len parameter of DPR.

As default, it is set to 256. What does it actually mean? Is it the number of tokens question+passage combined?
If not, what is it?

I divided my documents into passages which consist of less than 512 tokens each. So I wonder if I would set max_len_seq to be 256 and a passage is larger than 256 token, would DPR ignore half of each passage?

This issue is very crucial to me! :)

Thanks in advance!
Additional context
Add any other context or screenshots about the question (optional).

[tholor]

Hey @mathieudumayet ,

As default, it is set to 256. What does it actually mean? Is it the number of tokens question+passage combined?
If not, what is it?

It's the maximum sequence length for question and passage individually, i.e. long questions will be truncated to 256 tokens and similarly passages will be truncated to 256 tokens. We plan to switch this to individual params to give more fine-grained control (questions are usually way shorter than 256 tokens).

I divided my documents into passages which consist of less than 512 tokens each. So I wonder if I would set max_len_seq to be 256 and a passage is larger than 256 token, would DPR ignore half of each passage?

It would only ignore the part of the passage that is > 256 tokens.

[mathieudumayet]

Hey @tholor ,

Thank you for the explanation! BertTokenizer is the tokenizer that DPR uses, right? Because I will set max_seq_len parameter to 512 and divide documents into max. 512 tokens each.

Best,
Mathieu

[tholor]

BertTokenizer is the tokenizer that DPR uses, right?

Yes

Because I will set max_seq_len parameter to 512 and divide documents into max. 512 tokens each.

If you use a token-based split for your documents, I'd recommend splitting into something slightly smaller than 512.
You will need some extra space for the "special tokens" (e.g [CLS]). Also, in case you have meaningful "title" information attached to your documents, we will concatenate it to the "text" before truncating everything down to 512 (titles often boost the retrieval performance a bit). You can see the related code snippet here:
https://github.com/deepset-ai/haystack/blob/4fa5d9c3eba860ef0a85c46465b9ff1902ebfa12/haystack/retriever/dense.py#L145-L152

[mathieudumayet]

@tholor thank you so much for the information. It helped a lot!

Best,
Mathieu

[lalitpagaria]

@tholor Apologies to brining slightly off-topic.
How about integrating Longformer to support long documents (up to 4096 sequence length)?

[tholor]

@lalitpagaria Please see #61 and #337

[mathieudumayet]

@lalitpagaria Hi Lalit, can you say anything about the effect of increasing the number of tokens in a document from let's say 512 to 4096 tokens on the performance of the reader? Would model deliver a better answer if the number of tokens is high, because then it would understand the context better?

[lalitpagaria]

@mathieudumayet

can you say anything about the effect of increasing the number of tokens in a document from let's say 512 to 4096 tokens on the performance of the reader?

There performance issue with transformers, but now it's has been fixed. https://github.com/huggingface/transformers/pull/5811 . As mentioned in #337, you can use model with TransformersReader.

Would model deliver a better answer if the number of tokens is high, because then it would understand the context better?

As per paper, pre-trained, Longformer consistently outperforms RoBERTa (but slightly ~1%-4%) on long document task and it is less complex than high performing GNN models. I feel (not 100% sure) it is good choice if you don't want to split docs much and store as it is into the document_store.

[tholor]

From our initial tests, longformer was slower while not giving significant accuracy gains, but this might depend on the dataset.
One thing you should be aware of though: If you plan to use the DPR retriever, you'll need to split your docs into smaller passages anyway and then the longformer reader on top won't be the best choice.

[tholor]

Seems resolved. Feel free to re-open if not :)