Text: Randomly initialising word vectors

[Henry-E]

There doesn't seem to be the option to initialise word vectors without using pretrained embeddings. There's an option to fill in vectors for tokens missing from the pretrained embeddings with normally distributed values. It would be cool if there was a built in option to initialise embeddings from a uniform distribution without having to specify a word embedding file.

[jekbradbury]

You'd do that in your model class's __init__ by using nn.init or some other weight initializer rather than passing the weight matrix from torchtext.

[Henry-E]

Thanks for the recommendation, a bit new to this. The initialised vectors would still go into TEXT.vocab.vectors?

[jekbradbury]

No, you don’t need to put them there -- the only place where your embeddings actually need to be is in your model; TEXT.vocab.vectors offers a way to get pretrained vectors corresponding to your vocabulary and then use those to initialize your model's embeddings.

[jeremy-rutman]

....There's an option to fill in vectors for tokens missing from the pretrained embeddings with normally distributed values....

how do i find that option??

[zhangguanheng66]

We have new Vector class in torchtext/experimental/vectors.py. You can build a custom vector. See here https://github.com/pytorch/text/blob/master/torchtext/experimental/vectors.py#L219

[jeremy-rutman]

Is there an example for this somewhere? Ive been looking at the build_vocab docs and can't find any description of its arguments

[jeremy-rutman]

Can I do the following:

    TEXT.build_vocab(train_data) 
    vocab_size = len(TEXT.vocab)
    embedding_vectors = torch.FloatTensor(np.random.rand(vocab_size,embedding_length)
    word_embeddings = nn.Embedding(vocab_size, embedding_length)
    word_embeddings.weight = nn.Parameter(embedding_vectors, requires_grad=True)

if I am after randomly-initialized embeddings ?

[zhangguanheng66]

you don't need the last line.

word_embeddings = nn.Embedding(vocab_size, embedding_length)

should set the vector randomly.