Hub: How to use Elmo Embeddings (Word Vectors, Sentence Vectors)

[andymancodes]

Hi,

Even after trying to work with elmo and reading about it, I am not getting how to use it. It looks like for a given sentence, i have to pass the sentence through the elmo model and then I can get the elmo embeddings? But the parameters of a neural net are fixed after training. So, why not release elmo as a set of pretrained vectors like glove, why make it so hard to use?

Where could I find a clear description of how to use elmo embeddings. Does elmo have word embeddings? Does elmo only give sentence embeddings? How can I use elmo to say get word embeddings and compare their performance against Glove or compare the performance of sentence embeddings. How can I build a matrix of word embeddings as in Glove or word2vec?

I have been hearing about elmo and want to use it but so far all the nice metrics in their paper is lost on me because I can't really test it.

Tensorflow hub is awesome btw.

Cheers!

[vbardiovskyg]

Hi,

I will try to answer sequentially.

1. "It looks like for a given sentence, i have to pass the sentence through the elmo model and then I can get the elmo embeddings? " - you can embed your sentence with the elmo (and any other module) like this:

elmo = hub.Module("https://tfhub.dev/google/elmo/1")
embeddings = elmo(
    ["the cat is on the mat", "dogs are in the fog"],
    signature="default",
    as_dict=True)["elmo"]

1. "But the parameters of a neural net are fixed after training." - Parameters of the graph are not necessarily fixed. It is possible to fine-tune some parameters of the elmo module, if one sets trainable=True as in:

elmo = hub.Module("https://tfhub.dev/google/elmo/1", trainable=True)

1. "So, why not release elmo as a set of pretrained vectors like glove, why make it so hard to use?" - Even if the parameters were fixed, it doesn't mean that one can simply release them as embedding vectors. For example, elmo is purely character-based, meaning it has no vocabulary. Sure, you could define your own vocabulary and precompute embeddings using the module, but that is little bit defeating the purpose of having character level embedding.

(Btw if you wanted to do that as a speed/quality tradeoff, there is a mention of it here. You could then reexport the embedding as a TF-Hub module.)

1. "Where could I find a clear description of how to use elmo embeddings." - Description on how to use elmo module. Take a look at the exporting tool test for the shortest path from module to embeddings. If it is still not clear how to use the module, please let us know what seems to be the missing part.

2. "Does elmo have word embeddings? " - No, elmo does not have word embeddings in the sense "lookup table from word to embedding". But you still can embed words.

3. "Does elmo only give sentence embeddings? " - It gives embedding of anything you put in - characters, words, sentences, paragraphs - but it is built for sentence embeddings in mind, more info here.

4. "How can I use elmo to say get word embeddings and compare their performance against Glove or compare the performance of sentence embeddings. How can I build a matrix of word embeddings as in Glove or word2vec?" - what kind of metric do you want to use for comparison? If you want to use the STS benchmark, take a look here, but there are many more metrics. A very simple setup idea: you could for example reexport your Glove embeddings as a TF-Hub module and just plug both elmo and glove module in the linked STS benchmark.

[HimaVarsha94]

Thanks for the elaborate answer above. I am trying to use the above Elmo model and train using some custom data that I have. I made the trainable flag as True, but my weights don't change. Is there a documentation that I can look up?

[arnoegw]

Like https://tfhub.dev/google/elmo/2 says, the Elmo module only supports fine-tuning the weights for aggregating embeddings from its layers. It does not support training from scratch (new vocab and all). For a similar discussion of Universal Sentence Encoder, please see https://github.com/tensorflow/hub/issues/155

[PrashantRanjan09]

@HimaVarsha94 try having a look at : https://github.com/PrashantRanjan09/Elmo-Tutorial
See if it can help.

[HimaVarsha94]

@PrashantRanjan09 Thanks. You have two tutorials, one to use pre-trained ones and another for starting from scratch. Both I have already done. I am interested in using a pre-trained one at the beginning and training on top of it. The TensorFlow trainable flag is confusing me because I do not see the weights changing with that flag set as True.

[PrashantRanjan09]

@HimaVarsha94: There is one for incremental training as well, which addresses the issue you are facing. I also faced the same issue because when i was training with trainableflag as True, i also didnt see the weights changing. Although in the Tensorflow tutorial they mentioned that the new model hosted on hub allows that.

[Nomiluks]

Hi,
How can we train our data to obtain ELMO embeddings ?

[shikhar26]

@PrashantRanjan09 hello, i was able to train my model from scratch and was able to generate my weights file. Problem statement is to calculate the cosine similarity between sentence embedding. Please suggest the method to consume my model that gives us an aggregated result from all the layers in form of embedding for a sentence.
we used https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md#using-elmo-interactively but the output embedding we receive is a tensor containing output from each layer of the ELMO model (which are 3) and contains vectors for each word in the sentence rather then the sentence as a whole.

[MitaliSodhi]

@PrashantRanjan09 hello, i was able to train my model from scratch and was able to generate my weights file. Problem statement is to calculate the cosine similarity between sentence embedding. Please suggest the method to consume my model that gives us an aggregated result from all the layers in form of embedding for a sentence.
we used https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md#using-elmo-interactively but the output embedding we receive is a tensor containing output from each layer of the ELMO model (which are 3) and contains vectors for each word in the sentence rather then the sentence as a whole.

@PrashantRanjan09 I am also trying to do exactly the same as what you mentioned. Please let me know if you could come up with a solution.

[PrashantRanjan09]

@MitaliSodhi : since you have the vectors, the sentence representation can be an average sum of the vectors along an axis or a weighted sum. You just have to figure out how can you take sum of those vectors along an axis.
I am not sure if you can get the sentence level elmo embedding or if the capability is already provided by allenai.
However, take a look at https://www.tensorflow.org/api_docs/python/tf/math/reduce_sum
It might help.

[PrashantRanjan09]

Also, this is what Mark Neumann (from Allen AI has to say):

• Elmo does have word embeddings, which are built up from character convolutions. However, when Elmo is used in downstream tasks, a contextual representation of each word is used which relies on the other words in the sentence.
• Elmo does not produce sentence embeddings, rather it produces embeddings per word "conditioned" on the context. This is why you have to run them through the model.
• You cannot "build a matrix of word embeddings" for the reasons above.

Please see the tutorial for a comprehensive overview.
See

[jundongq]

The default output vector size is 1024 with 'elmo'. Is it possible to downsize the output, say 300 or 100, like word2vec?

[deanhoperobertson]

The default output vector size is 1024 with 'elmo'. Is it possible to downsize the output, say 300 or 100, like word2vec?

-- @jundongq did you managed to work this out?

[saadi143]

The default output vector size is 1024 with 'elmo'. Is it possible to downsize the output, say 300 or 100, like word2vec?

@jundongq Did you got any answer to reduce the dimensions of elmo embedding. E.g 300 or 256

[saadi143]

The default output vector size is 1024 with 'elmo'. Is it possible to downsize the output, say 300 or 100, like word2vec?

-- @jundongq did you managed to work this out?

@deanhoperobertson Please guide me if you know how to reduce the emlo embedding dimensions to 300.

[Arjunsankarlal]

Hi @saadi143 @deanhoperobertson @jundongq , As far as I know, these are pretrained models released by allennlp, which is trained to output those specific dimensions. So if you really want to reduce the dimensions, you can try dimensionality reduction techniques like PCA, tSNE, UMAP. Hope this helps.

[saadi143]

@Arjunsankarlal Dimension reduction such as PSA will loose the information. Is there any way to get around 300 dimensions?

[Arjunsankarlal]

Hi @saadi143, I think the only option would be training the whole ELMo model on you own from the scratch, which I am not sure how much possible it is, based on the dataset availability and the hardware requirement! If you could tell what you are really trying to achieve/solve I could share my thoughts :)

[saadi143]

@Arjunsankarlal I have to get embedding for my dataset tokens which is requirment of my thesis work. I am willing to use Elmo embedding. I get the 256 weights hdf5 file from Allen NLP but I am not sure how to use that file.
And I am also want to train the whole model. I get the link from allennlp but i didn't understand how to train the model on my own dataset. If you know how can I train the model on my dataset or any useful source please guide me.

[Arjunsankarlal]

@saadi143 Sorry, I haven't trained a ELMo model from the scratch! You can start from here.

You can try posting a question in Allennlp repo, where you might get the right guidance for training ELMo from scratch.

[saadi143]

@Arjunsankarlal Did you use pre-trained weights model? which is in hdf5 file?

[jeromeshan]

Hi, i have stupid question, how can i get just one vector from sentence using elmo

[deanhoperobertson]

The default output vector size is 1024 with 'elmo'. Is it possible to downsize the output, say 300 or 100, like word2vec?

-- @jundongq did you managed to work this out?

@deanhoperobertson Please guide me if you know how to reduce the emlo embedding dimensions to 300.

Sorry @saadi143 I didnt find a solution to this.

[eduamf]

You can do it changing the config file (.json) or, if you`re using bilm-tf, changing train_elmo.py.

The content of my config file is like this:

"bidirectional": true,
 "char_cnn": {"activation": "relu",
              "embedding": {"dim": 16},
              "filters": [[1, 16], [2, 32], [3, 64], [4, 128], [5, 256], [6, 512], [7, 1024]],
              "max_characters_per_token": 50,
              "n_characters": 262, 
              "n_highway": 1},
 "dropout": 0.1,
 "lstm": {"cell_clip": 3,
          "dim": 1024,
          "n_layers": 2,
          "proj_clip": 3,
          "projection_dim": 128,
          "use_skip_connections": true},
 "all_clip_norm_val": 10.0, 
 "n_epochs": 10,
 "n_train_tokens": 21100,
 "batch_size": 4,
 "n_tokens_vocab": 1136,
 "unroll_steps": 20,
 "n_negative_samples_batch": 16} 

[njfm0001]

As ELMo is trained on a corpus, is there any way to perform a lookup of the ten most similar words according to a given word?