Opennmt-py: How to use pretrained embeddings from Glove?

[ijauregiCMCRC]

I am trying to train the model with pretrained word embeddings. The embeddings I am trying to use are from Glove, such as the Common Crawl 840B.

What I try to do is to run train.py file by adding the embedding '.txt' file in the -pre_word_vecs_enc argument. But the following error appears:

`Loading data from 'PaCo_data/En_eu_preprocessed.atok.low.train.pt'

• vocabulary size. source = 50003; target = 50004
• number of training sentences. 117056
• maximum batch size. 64
  Building model...
  Intializing params
  Traceback (most recent call last):
  File "OpenNMT-py/train.py", line 440, in
  main()
  File "OpenNMT-py/train.py", line 413, in main
  encoder.embeddings.load_pretrained_vectors(opt.pre_word_vecs_enc)
  File "/data/ijauregi/Desktop/CMCRC/OpenNMT/OpenNMT-py/onmt/Models.py", line 57, in load_pretrained_vectors
  pretrained = torch.load(emb_file)
  File "/data/ijauregi/Desktop/pyTorchPython/lib/python3.5/site-packages/torch/serialization.py", line 229, in load
  return _load(f, map_location, pickle_module)
  File "/data/ijauregi/Desktop/pyTorchPython/lib/python3.5/site-packages/torch/serialization.py", line 367, in _load
  magic_number = pickle_module.load(f)
  _pickle.UnpicklingError: invalid load key, 'O'.
  (pyTorchPython)
  `
  What should be the format of the embedding file? Do I need to change the format somehow?

[hongweizeng]

maybe you can use the initialized method in torchtext

[srush]

That's a good idea. We don't support torchtext for this yet, but we will prioritize this now that torchtext is integrated.

[hongweizeng]

It's very convenient to use the methods in torchtext since the data saved in preprocess.py are classes defined in torchtext.

but you should define the word_vec_size equal to wv_dim, where there is 300 in the picture.

[srush]

Fantastic. Maybe we can have this be a special case of "pre_word_vecs_enc"?
Something like "tt:glove.6b"?

On Aug 30, 2017 10:05 AM, "zeng" [e-mail] wrote:

It's very convenient to use the methods in torchtext since the data saved
in preprocess.py are classes defined in torchtext.
[image: image]
https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_15373554_29876163-2D9989e692-2D8dce-2D11e7-2D9630-2D107aa218cea0.png&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=wnHFZ7D4m-9MRwk-CWlvCGbWEiQX_AvUO2LuMy4Vj7c&m=jC0j3mOSAzUsg1-dr3jX-pakrBW7H8SCMQGlDTZiXak&s=Yz6_X5mIGgKLKDiR2VJ-njVtTtaFv--r3_NC5CmXWjc&e=

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_OpenNMT_OpenNMT-2Dpy_issues_210-23issuecomment-2D326000736&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=wnHFZ7D4m-9MRwk-CWlvCGbWEiQX_AvUO2LuMy4Vj7c&m=jC0j3mOSAzUsg1-dr3jX-pakrBW7H8SCMQGlDTZiXak&s=14TX4OmsFdW6aYihuCtExHydLcE4bYE2yATEeMrUwHw&e=,
or mute the thread
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AACMKpStmkZ4l1hFHLGdGRNkhZ29ZO-5FIks5sdWwHgaJpZM4PG58K&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=wnHFZ7D4m-9MRwk-CWlvCGbWEiQX_AvUO2LuMy4Vj7c&m=jC0j3mOSAzUsg1-dr3jX-pakrBW7H8SCMQGlDTZiXak&s=4Pl7h4YIx8c1rMP7wlBxQ1V7H1sWuQTswrdVSiW6L74&e=
.

[hongweizeng]

the methods implemented in torchtext, maybe just need a judge to identify the string and choose the wanted embedding.

[srush]

Adding @bpopeters who is the Pytorch Chief Embeddings Officer.

[ijauregiCMCRC]

Hi!

Thank you @srush and @xjtu-zeng for your quick replies. But I still I am a bit lost. I am very new with pytorch. I am still confused about what kind of file needs to go in the argument -pre_word_vecs_enc and -pre_word_vecs_dec. In other occasions I have used the embeddings downloaded from Glove or Word2vec. They are usually in a '.txt' file and they follow the following structure:
"
word1 embedding1
word2 embedding2
word3 embedding3
...
"

But it seems that by just giving this kind of file as an argument doesn't work here.

Thanks,

[bpopeters]

I believe the argument for pre_word_vecs_enc/dec needs to be a serialized pytorch object representing a matrix whose size is (n x m), where n is the vocabulary size and m is the dimensionality of the word vectors. The nth entry in this matrix corresponds to the nth element of the vocabulary. So in order to use word vectors in another format, you'll need to write a script that puts them in a torch.Tensor and then run torch.save on that object to serialize it to a file.

[srush]

@bpopeters I think there is a request for a special option that instead loads the torchtext embeddings as above.

[ijauregiCMCRC]

Thanks @bpopeters . I think I understand what I need to do. I started using a program called torchwordemb, which takes as input the path to the glove_embeddings.txt file, and returns:

• A dictionary with all the vocabulary mapped to unique IDs
  -The vectors matrix (size: n_vocab X emb_dim)

Do I need to serialize both documents? Because, otherwise, if I only serialize the matrix, how does then OpenNMT know how to initialize each word in the source or target dictionary with these pre-trained embeddings?

Thanks

[moonlightlane]

Hey, I have a remotely related question on this... I just tried the modified code to load pretrained word embedding from @xjtu-zeng: worked out of the box like charm!

However, I really wanted to understand how OpenNMT or torchtext handles out of vocabulary words, e.g. those word tokens that appear in source or target sentences but not in the pretrained word embeddings. I took a brief look at Embeddings.py and didn't seem to find the function/method that deals with out of vocab word tokens.

Maybe this issue is handled in torchtext? Could someone kindly explain how this issue is handled, or point to the file or places in a file that I should take a look at to understand it? Thanks so much in advance!

[srush]

There is an UNK token, that is always set to index 0. Since the vocab size is limited in training, it will learn to handle this token. At test time it can input and output UNK.

[Henry-E]

Based on how I've seen other pretrained loading things work it depends how you initialise your embedding matrix. For example here the matrix is initialised with random normals. So any word which is already in the vocabulary but doesn't appear in the pre-trained embeddings will be treated as a randomly initialised vector. And from there it depends whether you continue to train the embedding or leave them as they started.

The way that OpenNMT-py allows pretrained embeddings to be loaded all of this stuff has to be handled manually beforehand. So unless you go back and change your vocabulary as part of the preprocessing it's unlikely that the words not in the pretrained embeddings are going to register
as UNKs.

[caozhen-alex]

@ijauregiCMCRC Hi, have u got it?

[ijauregiCMCRC]

I used the following script https://github.com/ylhsieh/OpenNMT-py/blob/master/embeddings_to_torch.py and it worked for me.

[srush]

We have this checked in now.

[pltrdy]

PR https://github.com/OpenNMT/OpenNMT-py/pull/398 introduces a tool to do it easily.

see the tutorial: http://forum.opennmt.net/t/how-to-use-glove-pre-trained-embeddings-in-opennmt-py/1011

[caozhen-alex]

@ijauregiCMCRC Thank u for reply. But it seems that the link is 404.

[srush]

You can go here http://opennmt.net/OpenNMT-py/FAQ.html