Sentence-transformers: Is the SentenceTransformer appropriate for news clustering?

[Barcavin]

Hi,

Thanks for the great work. Those pretrained sentence encoders are quite useful. But if I understand it correctly, when it comes to longer texts like news, it won't perform as well as the sentence.

In terms of document embedding, is it more appropriate to just average the BERT embedding? or even tf-idf. What do you recommend to use for document embedding if I also want to leverage the power of BERT.

Thanks in advance

[nreimers]

Hi @Barcavin
BERTs memory requirement grows quadratic with the length of the sentence / document. Therefore, there is a limit of 512 tokens. As many words are broken down into sub-tokens, you often can only encode texts with up to 300-400 words with BERT.

In general I think BERT is the wrong choice for documents. BERT is strong on a sentence level, but it will not produce sensible results on a document level.

TF-IDF / BM25 is extremely strong on document level and really hard to beat, much much better than most fancy neural approaches. Another good option is also LSA and LDA, which produce also quite nice clusters.

Personally I would try these approach. If you tune them well, you get good clustering results with a quick runtime.

Best regards
Nils Reimers

[emtropyml]

Hi @nreimers

That's an interesting insight. Is there any research/data to back the claim that BERT is not a good choice for deriving document embedding?

Thanks

[nreimers]

Hi @emtropyml
not with respect to docs, but on a sentence level for information retrieval:
https://arxiv.org/pdf/1811.08008.pdf

On a sentence level, BM25 is hard to beat and only with sufficient training data.

For document level you have the length limit of BERT (510 word pieces limit). Also from my personal experiences, encoding longer docs (1 page or more) to a dense vector space currently does not really work well with any common embedding method.

[emtropyml]

Thanks for sharing @nreimers .

I wonder if deriving BERT embedding using this library for a paragraph comprising of say 10 sentences and well within 510 word piece tokens will give a meaningful representation? And if such embedding could be used to carry out unsupervised clustering for thousands of paragraph samples?

Also, how does it deal with sentences longer than 510 tokens? Does it simply truncate them?

Thanks

[nreimers]

Hi @emtropyml
You could try that ;)

One issue would be the training data. The approaches here so far are optimized on sentence level data. To work well for paragraphs, I think a lot of paragraph data would be needed.

In general it is quite hard to map paragraphs to a vector space. What if a paragraphs with 10 sentences takes about 10 different topics? What, if it talks only about 1 topic?

Currently the limit of the pre-trained models are 128 word pieces, BERT itself has a limit of 510 word pieces. Longer texts are truncated.

Best
Nils

[emtropyml]

Got it. Thanks!