Bert-as-service: Lower layer giving better results

[astariul]

I'm using a custom model, using Bert as service feature vectors as input.
I'm solving the problem of sentence textual similarity. I'm using the SICK dataset, but the STS-B dataset (from GLUE) is similar and could be used as well.

I tried to use the default layer, -2, and got a score of ~75%.

I tried to use concatenation of last layers (as described in the paper), ie. -1 -2 -3 -4, but the score didn't improve (actually slightly decreased).

I finally tried a low layer, -11, and got a score of ~80%.

---

Why a lower layer would give a better score ?
I don't understand...

[hanxiao]

Well, interesting findings, but I won't over-interpret this observation based on one experiment.

pooling_layer=-1 close to the training output, may be biased to the training targets. If you didn't fine tune the model, then this could be a bad representation.
pooling_layer=-12 close to the word embedding, may preserve the very original word information (with no fancy self-interaction etc.). On the other hand, you may achieve the same performance by simply using a word-embedding only.
Anything in-between [-1, -12] is then a trade-off.

Plus, they also give different speed: https://github.com/hanxiao/bert-as-service#speed-wrt-pooling_layer

just saying.

[hanxiao]

btw below is result from the original BERT paper (Sect. 5.4) Note that they use pretrained-fixed-BERT to output contextualized word embedding, and then feed to a randomly initialized two-layer 768-dimensional BiLSTM before the classification layer. And it is only 0.3 F1 behind fine-tuning the entire model

may also related to #76

[hanxiao]

my bad. I forgot to mask the sequence when doing reduce_mean, reduce_max and reduce_mean_max. this is fixed in #94 and now merged into master. please give it a try and may produce different result.

in principle, this will affect the most when max_seq_len is much longer than actual sequence length from clients.

[astariul]

Your fix is amazing ! 🥇

Updated results :

• Using layer -2, I now got a score of ~80%
• Using layer -11, I now got a score of ~82%

Awesome job !

[hanxiao]

nice, good to hear. close #85

[hanxiao]

FYI, I made a visualization of different BERT layers, may help you understand the pool_layer and pool_strategy. See: https://github.com/hanxiao/bert-as-service#q-so-which-layer-and-which-pooling-strategy-is-the-best

[chikubee]

@hanxiao
It is stated everywhere that the [CLS] token is the correct sentence representation for a fine tuned model.
I fine tuned bert base uncased model for sentiment analysis task.
The eval accuracy is really good.

I aim to interpret the false positives and what lead to them.
I thought finding similar representations in the training set as the test sample, would give me some insight so as to what lead to the prediction.

I've taken [CLS] token representation from the last layer of the fine tuned model.
The sentences I get similar to a false positive, do not look similar but they do share the same sentiment as predicted.

What is going on here?
Is it really the correct representation.

I would also like to know how this representation is learnt by CLS token, if all the tokens can look at the left and right context and itself., How is it that CLS is capturing the entire sentence's meaning?

Thanks in advance.