Sentence-transformers: How to get the Avg. BERT embeddings?

[Cumberbatch08]

I have read your paper, and I want to know which layer of bert embedding used in your code.
Many people have said that the last layer is not the reasonalbe representation.
And this project recommends the last 2 layer embedding to averge, https://github.com/hanxiao/bert-as-service.

[nreimers]

Hi @Cumberbatch08,
In my experiment, bert-as-service produced quite bad results for unsupervised tasks like clustering, semantic search etc. I tested using the CLS token as well as averaging the BERT outputs.

I think these representation are not really suitable for unsupervised tasks: For unsupervised tasks, all dimensions are treated equally, i.e., each dimension is assigned the same weight for the final decision. However, BERT was not optimized for this.

The BERT CLS token / avg. BERT outputs contain a lot of useful information in some dimension, but not in all. If now all dimensions are treated equally, this leads to rather bad results.

In this repository, the default mode is to average the output of BERT. With some fine-tuning on suitable data, it generates sentences embeddings that work quite well for different unsupervised tasks.

I haven't tested what happens if you use different BERT layers and fine-tune them. But I would think that the effect is not that big, as the fine-tuning steps ensures that BERT produces nice embeddings for unsupervised tasks.

I also tested to use a max-pooling as well as using the CLS token: The differences between these variations are not that big.

Best regards
Nils Reimers

[aclifton314]

To piggy back on this topic, is it possible to get the word embeddings from BERT using this repo?

[Cumberbatch08]

Hi @nreimers , as you said, *bert-as-service produced quite bad results for unsupervised tasks like clustering, semantic search etc. *
so, in your experiment , you directly uesed bert-as-service to produced the words embeddings or the sentence embedding, the parameter layer is default value(-2)?
just like this:


In my experiment, the last layer gets the bad representation of sentence, but the second-to-last layer seems like good.

[nreimers]

Hi @aclifton314
It would be possible. In for example training_nli_bert.py you just omit the pooling layer. This will give you the BERT embedding for each token. But of course, without a pooling, there is no fixed-sized sentence embedding you could use. If you are interested to use the BERT embeddings direct, bert-as-service might be the better choice.

@Cumberbatch08
I tested REDUCE_MEAN and CLS_TOKEN.

[Cumberbatch08]

@nreimers thank you very much. you tested the average words embeddings, and the word embedding is from bert-as-service, and you use the default layer parameter?
just as you did in your experiment, the average is not a good way to represent sentence. And it is worse than the USE way.
In unsupervised task, the USE embedding maybe the better choice; but in supervised task , the fine-tune way is absolute the better way.

[nreimers]

Hi @Cumberbatch08 ,
yes, I used the default config from bert-as-service. For (my) unsupervised tasks, it didn't produce good sentence representations (USE was much better).

For supervised tasks like sentence classification I would skip sentence embeddings (and bert-as-service) completely and directly fine-tune BERT. This yields much better scores than with the intermediate step of generating a sentence embedding and then learning a classifier on-top.

Best
Nils Reimers

[Cumberbatch08]

@nreimers thank you very much. I finally confirmed which layer you used in your experiment. google has said the ouptut looks like a sentence vector, but is not meaningful.

your experiment also confirmed that.