Text: build_vocab with custom word embedding

[DaehanKim]

Hi :)
I want to use customized bio-word embedding to do some text classification.

And I can't find how.

Some old tutorial says there is 'wv_dir' keyword argument, which I tried and failed :

TypeError                                 Traceback (most recent call last)
<ipython-input-48-ac09f554719e> in <module>()
      1 test_field = data.Field()
      2 lang_data = datasets.LanguageModelingDataset(path='pr_data/processed_neg.txt',text_field=test_field)
----> 3 voc = torchtext.vocab.Vocab(wv_dir='bio_wordemb/PubMed-and-PMC-w2v.txt')
      4 
      5 # test_field.build_vocab(lang_data,wv_dir='bio_wordemb/PubMed-and-PMC-w2v.txt')

TypeError: __init__() got an unexpected keyword argument 'wv_dir'

Just like we can load pretrained GloVe embedding using TEXTFIELD.build_vocab(data, vectors='glove.6B.100d'), is there similar way to load customized embedding?

Any help would be much appreciated. Thanks!

[Eric-Wallace]

The wv_dir has been removed to my knowledge. In my opinion, the best way to get custom vectors is to actually just make changes to the vocab.py file. See for example how FastText is added. https://github.com/pytorch/text/blob/23ced46fcb310a2dd1a0cc065c326141f65e0b5c/torchtext/vocab.py#L333

[DaehanKim]

Thank You. I built a custom extractor and solved the problem.

[tomekkorbak]

@jekbradbury, would it be valuable to work on a PR with better support for using custom word embeddings? If so, I'd be happy to work on this.

[t-vi]

Unless I'm missing something subtle, the obvious right thing to do here is to do

                if os.path.isfile(vector):
                    vectors[idx] = Vectors(vector, **kwargs)
                else:
                    # ... do the predefined stuff currently there but better use if ... elif instead...

in Vocab.load_vectors when you have a string argument.

Needless to say, I'd be happy to send a PR with updated docs etc.

[pablogps]

Is there a way to use a pre-trained embedding that does not involve changing the source code? This seems like pretty important functionality! As a fast patch I would be very grateful if a pre-trained Spanish embedding were included in the default choices.

[bebound]

This works, xxx.vecshould be the standard word2vec format file.

from torchtext.vocab import Vectors

vectors = Vectors(name='xxx.vec', cache='./')
TEXT.build_vocab(train, val, test, vectors=vectors)

[pablogps]

Yes, you are right! There is a bit more info here.

[meghdadFar]

@bebound I tried your solution with pretrained 300-dimensional word embeddings. However, the loaded vectors are always 100 dimensional. Do you know why?

[bebound]

@meghdadFar That's weird. Can you check the output ofhead -n2 xxx.vec | wc -w? It should be 303.

[bebound]

@meghdadFar Yes

[meghdadFar]

@bebound Ok, so the vector file was a binary file. I changed it back to text. The output of head -n2 xxx.vec | wc -w is 303. But when I load them, the dimension remains 100:

xxx_vectors = Vectors(name='xxx.vec')
print(xxx_vectors.dim)

100

[bebound]

@meghdadFar I guess there is hidden cache file in that folder(The suffix may be pt, I’m not sure). Delete it and try again.

[meghdadFar]

@bebound you're absolutely right. There was an old 100-dimensional xxx.vec inside .vector_cache/. I removed it and the problem is fixed. Thanks!

[rahular]

Is there a way to initialize the embeddings from a torch tensor instead of a file? I want to use wikidata big graph embeddings which cannot be loaded into memory.

[balhafni]

Is there a way in torchtext.vocab to load the vectors from a binary file or they must be in the standard word2vec format?

[zhangguanheng66]

@balhafni I don't see the utils to load a binary file there.