Bert-as-service: character embedding or word embedding?

[lucyboll]

Hello, thanks for your service, it is very useful. I notice that the word embedding is obtained for character 'h' rather than word 'hey' as follows. It seems like doesn't match with bert tokenizer.

`bc = BertClient()
x = ['hey you', 'whats up']

bc.encode(x) # [2, 25, 768]
bc.encode(x)[0] # [1, 25, 768], word embeddings for hey you
bc.encode(x)[0][0] # [1, 1, 768], word embedding for [CLS]
bc.encode(x)[0][1] # [1, 1, 768], word embedding for h
bc.encode(x)[0][8] # [1, 1, 768], word embedding for [SEP]
bc.encode(x)[0][9] # [1, 1, 768], word embedding for 0_PAD, meaningless
bc.encode(x)[0][25] # error, out of index!`

[astariul]

I am curious, how did you know the embedding correspond to which word / character ?

[EricLingRui]

I want to know why the 0_PAD matrix is meaningless

[RuiMao1988]

How can I obtain word embedding instead of character embedding please?

[bangann]

I have a question. Where to set max_seq_len and pooling_strategy?

[hanxiao]

@lucyboll thanks for pointing it out, i believe it's a bug. will fix it

[loretoparisi]

@hanxiao may this issue be related to https://github.com/google-research/bert/issues/164

[hanxiao]

@lucyboll I fix readme by providing more explanation for tokenization, see
https://github.com/hanxiao/bert-as-service#q-how-can-i-get-word-embedding-instead-of-sentence-embedding
https://github.com/hanxiao/bert-as-service#q-do-i-need-to-do-segmentation-for-chinese

[hanxiao]

@RuiMao1988 If you are using Chinese BERT released by Google, word embedding is character embedding, as this Chinese BERT is a character-based model.