Bert-as-service: Extracting most similar words in the vocabulary

[ogencoglu]

Prerequisites

• [x] Are you running the latest bert-as-service?
• [x] Did you follow the installation and the usage instructions in README.md?
• [x] Did you check the FAQ list in README.md?
• [x] Did you perform a cursory search on existing issues?

Description

I can extract word embeddings and it works smoothly. How would one implement most similar words kind of functionality? Something like the below example from gensim for _word2vec_, _GloVe_, _fastText_ etc. :

import gensim.downloader as api
word_vectors = api.load("glove-wiki-gigaword-100")  # load pre-trained word-vectors
result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])

[hanxiao]

hmm, it actually doesn't work in the way as you expect. Note that unlike word2vec/Glove/fastText, word embedding from BERT is contextualized, meaning that the value of woman embedding depends on the other words in the sentence, which also means that two woman can have completely different embedding values when they appear in two different sentences. This makes "most similar words" function spurious for bert-as-service

[gionapaolini]

What if I want to find the most relevant words from the contextualized vector?
Like using Bert as an intent classifier and then use the contextualized vector of the intent to find related words from the vocabulary.