Text: Adding vocab to field object

[SamLynnEvans]

Hi Everyone,

Is there any way of adding more vocabulary to a Field object that already has had its vocab built?

for example if at one point I have run this sort of code:

TEXT = field()
TEXT.build_vocab(dataset_a)

Is it possible to add vocab to TEXT from another dataset without erasing the current vocab built from dataset_a?

Thanks!

[mttk]

There's two options:

the build_vocab method accepts a variable number of datasets. So, if you want to add the vocabulary from dataset_b to dataset_a, you can simply do it by calling TEXT.build_vocab([dataset_a, dataset_b])

In the second case, where you just have a list of words and not another dataset, I'd suggest extending the stoi and itos variables of the built vocab (TEXT.vocab.stoi, TEXT.vocab.itos) with the new words.
This will be a bit more complicated if you also need to add vectors for the new words. A PR is in progress which will make this easier to do in the near future.

[skurzhanskyi]

@mttk, but if I do not have test data before the training and I want just to update the vocabulary to predict test?

[mttk]

I'd say the best way (right now) is to manually extend stoi and itos with the new words and concatenate the vectors to your embedding matrix (where you have to make sure that indices match).

Allowing the vocab to be extended after being constructed seems like a reasonable change, but I'm not sure when I'll get the time to implement it.

[skurzhanskyi]

Thanks a lot for the quick answer. Currently, there's the method Vocab.extend. It also may be helpful, but in this case, you should create your own Vocab for the test set. Not sure about it at the moment.

[mttk]

It would definitely be helpful. Extending with another vocab object seems reasonable, but you might need to define new fields for that (since calling with the existing one would overwrite it). Calling on a raw dataset might be simpler.

[skurzhanskyi]

Should I extend a new field or the old one? Because when I tried to extend the old one (what seems logically) I get significantly worse results. Maybe the reason is that when you use your new field in data.TabularDataset it becomes somehow related to your test data. So, you should extend the new field, but to keep all indexing changes in your embedding. Am I doing right?

[mttk]

Which field you are extending shouldn't matter (basically all that fields do is convert strings to indices and tokenize). Can you first manually check that both the str-index mappings are good after you extend the fields?

If it still doesn't work, paste some minimal reproduction if you can and I'll look into it.

[skurzhanskyi]

Thanks a lot for your help, Martin.
Steps that helped to solve this:

• create new text field (with the same parameters as previous one)
• load new dataset using this field
• build vocabulary for this field for this dataset
• extend previous filed's vocabulary with the new one
• link new dataset with the old filed (which was extended) like
  data_test.fields['text'] = OLD_TEXT_FIELD

[mttk]

Cool, thanks for the workflow. I'll hopefully get around making this easier sometime soon.

[tu-artem]

Hi @mttk, what is the status of this one? I want to work on it if it is not in progress already

[mttk]

@tu-artem feel free to start working on it, it's not yet in progress

[tu-artem]

Some challenges I am currently facing:

1. Currently there is an extend() method of Vocab which simply goes through itos of an argument and adds them to current vocab. I want to make it more general to accept Counter object as in Vocab.__init__() and also keep in mind original max_size and min_freq arguments. But changing it this way may harm code that is using current entend() method
2. If vocab is built with specials_first=False it becomes unclear how to add new words, since specials will no longer be at the end if itos/stoi
3. I am not sure how to treat a Vocab with existing vectors

Any advice is highly appreciated!