Text: Feature request: add ELMo vectors

[manuelsh]

They are new state-of-the-art word embeddings.

See: https://allennlp.org/elmo

Thanks!

[mttk]

ELMo works on sentence level (and requires loading a pretrained model, not only a dictionary), which is _currently_ out of scope of the library. Adding ELMo and CoVe would be nice, but I'm not sure when it's going to happen.

[manuelsh]

@mttk I believe ELMo produces an embedding for each word. Extracted from the paper:

" Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus."

See also this link.

[mttk]

It does, but it produces those vectors through a BiLSTM, which requires the whole sentence as context.

Quote from the blog post: _"Embeddings from Language Models (ELMos) use language models to obtain embeddings for individual words while taking the entire sentence or paragraph into account."_

and

_"Concretely, ELMos use a pre-trained, multi-layer, bi-directional, LSTM-based language model and extract the hidden state of each layer for the input sequence of words. Then, they compute a weighted sum of those hidden states to obtain an embedding for each word."_

To obtain ELMo embeddings for each word, you pass the sentence through a pretrained BiLSTM. The weights available online are the weights of those BiLSTMs, not the word vectors.

[manuelsh]

Hi @mttk

That's the same case with other embeddings, like word2vec, where they are obtained from a model that takes context into account (e.g. n-grams) and then one can use the pre-trained vectors in other model.

I believe in https://allennlp.org/elmo there are weights from pre-trained ELMos vectors, taken with models trained with wikipedia corpus.

[manuelsh]

So when it says: "Then, they compute a weighted sum of those hidden states to obtain an embedding for each word." this is how the embeddings for each word are obtained, I think. :-)

[mttk]

It's not the same case as the other word embedding models such as word2vec generate a "word : vector" dictionary. ELMo produces a model which produces those weights. This requires, at some point, loading the model as a pytorch module.

You can read this tutorial and also download and open the weights to validate this yourself.

[manuelsh]

I see your point now... the phrase "Then, they compute a weighted sum of those hidden states to obtain an embedding for each word." confused me. Thanks @mttk !

[davidalbertonogueira]

Shouldn't be that difficult. I just implemented a BERT-based Field to handle BERT embeddings (also sentence-level) (https://github.com/google-research/bert)

This field instead of receiving batch of tokenized examples and outputting (batch, sentence_legth) of word_ids to the embedding table, calls the forward pass of the BERT model and outputs (batch, sentence_legth, embedding_dim).

The remaining of the code should be the same, with the removal of the nn.Embedding layer from the main model nn.module.

[raheelqader]

@davidalbertonogueira: is there a chance you can share the code of your BERT supported field?

[zhangguanheng66]

If ELMo is still needed, please propose in PyTorch core library since it is relevant to the embedding module.