Text: RuntimeError: index out of range when using dataset loaded using torchtext

[Rj7]

Hello,
I have a problem using torchtext to load data for machine translation. I load my data using torchtext TranslationDataset class like this

train_data,val_data,test_data = datasets.TranslationDataset.splits(root='../data',
                                                    train='training',
                                                    exts=('.de', '.en'), 
                                                    fields=(DE, EN)
                                                   )

The words gets numericalized based on the number of words in given data. My problem occurs when I build vocab based on some minimum frequence EN.build_vocab(train_data.src,min_freq=100). The vocab size gets reduced to 2383 instead of 31760. But the word indexes are still over 2383.

self.embedding = nn.Embedding(vocab_size, hidden_size, padding_idx=1)
embedded = self.embedding(word_inputs)

This gives me out of range problem when I use this numericalized input in the embedding layer as the vocab_size is only 2383 but the index of some words are over 2383.

[divyanshj16]

I want to take this up.

[DocMorg]

Have same problem

[DocMorg]

wait, NVM, i thought that i need size+1 cause it starts from zero and it worked xD
sorry for inconvenience