Text: Ignoring UNK words

[manuelsh]

Cannot find the way to ignore UNK words when numericalising, i.e. instead by substituting them by a 0, it just ignore that word.

Is that implemented?

This is useful in classification problems, when you just want to remove 'UNK' words.

[rumaak]

https://torchtext.readthedocs.io/en/latest/data.html#field

postprocessing – A Pipeline that will be applied to examples using this field after numericalizing but before the numbers are turned into a Tensor. The pipeline function takes the batch as a list, and the field’s Vocab. Default: None.

I tried this, but unsuccessfully. When used pipeline deleting 0 from sequence, I get this error:

ValueError: expected sequence of length 12 at dim 1 (got 11)

I don't think there is anything closer to your use case than postprocessing, yet that doesn't seem to work. If anyone could try it out, it would be great.

Minimal example I used:

import torchtext
from torchtext.data import *

def func(*x):
    y = x[0]
    return [z for z in y if z != 0]

pipe = Pipeline(func)
field = Field(postprocessing=pipe)

# 2 arbitrary datasets, ds2 must contain words ds1 doesn't contain
ds1 = TabularDataset(fname1,'tsv',[('text',field)])
ds2 = TabularDataset(fname2,'tsv',[('text',field)])

field.build_vocab(ds1)
itr = Iterator(ds2,4)
next(iter(itr)).text

[mttk]

So, the lifecycle of a dataset w.r.t. a field goes as follows:

1. preprocess is called on example-level (row-by-row) when loading data, but at this moment the vocabulary is not yet built
2. build_vocab is called on the pre-processed dataset, mapping the strings to indices
3. process is called (pad & numericalize) on an already constructed batch (by an Iterator)
4. (optional) postprocess is called

You can't fit this part into 4, since the batches have to be fixed-size, and you can't fit it into 1 because you don't yet know the vocabulary. What I would suggest (for now) is manually placing that part in "2.5".

After you have built your vocab, you can access stoi. Then you can do something like this:

def clean(example, vocab):
    return [word for word in example if word in vocab]
# train here is the `Dataset` instance obtained through .splits()
for example in train:
  # assuming field name is 'text'
  example.text = clean(example.text, TEXT.vocab.stoi)

[manuelsh]

Would be great to add that as an option when, for example, creating the vocabulary.

[mttk]

Agreed, but the best way is not yet completely clear to me because I dislike going over the whole dataset so many times (1. loading, 2. building vocab, 3. filtering and 4. batching). And you can filter UNKs only after you've constructed the whole vocab.

This filtering would ideally happen _during_ numericalization, but it doesn't fit into the current lifecycle so I'd like to restructure that a bit. For now, a PR doing this in the least painful way (something like dataset.filter_examples(vocab)) would be welcome.

[rumaak]

@mttk why reopen?

[mttk]

@rumaak I remembered that this is just a temporary solution :)

[zhangguanheng66]

@mttk is there anything I can do to resolve the issue?