Text: Custom dataset: It seems tokenization is always being done at a character level

[davidsbatista]

❓ Questions and Help

It seems tokenization is being done at a character level

    import torchtext

    class Example(object):
        def __init__(self, news_text, category):
            self.text = news_text
            self.category = category

    sample_1 = Example('a simple sentence', 'label_1')
    sample_2 = Example('any other thing', 'label_2')
    texts = [sample_1, sample_2]

    TEXT = torchtext.data.Field(sequential=True)
    LABEL = torchtext.data.Field(sequential=False)

    dataset = torchtext.data.Dataset(texts, fields=[('text', TEXT), ('category', LABEL)])
    TEXT.build_vocab(dataset)

    print(TEXT.vocab.freqs)

I get this

Counter({'e': 5, ' ': 4, 'n': 4, 't': 3, 'a': 2, 's': 2, 'i': 2, 'h': 2, 'm': 1, 'p': 1, 'l': 1, 'c': 1, 'y': 1, 'o': 1, 'r': 1, 'g': 1})

Even if I try to pass a custom tokenizer, simple one based on split(), which I I think is also the default one, the same result happens.

 import torchtext

 class Example(object):
     def __init__(self, news_text, category):
         self.text = news_text
         self.category = category

 sample_1 = Example('a simple sentence', 'label_1')
 sample_2 = Example('any other thing', 'label_2')
 texts = [sample_1, sample_2]

 TEXT = torchtext.data.Field(sequential=True, tokenize=lambda x: x.split())
 LABEL = torchtext.data.Field(sequential=False)

 dataset = torchtext.data.Dataset(texts, fields=[('text', TEXT), ('category', LABEL)])

 TEXT.build_vocab(dataset)

 print(TEXT.vocab.freqs)

The same thing:

Counter({'e': 5, ' ': 4, 'n': 4, 't': 3, 'a': 2, 's': 2, 'i': 2, 'h': 2, 'm': 1, 'p': 1, 'l': 1, 'c': 1, 'y': 1, 'o': 1, 'r': 1, 'g': 1})

I think I'm missing something simple in setting up the custom dataset but can't figure it out what it is. I also looked through the build_vocab() and it seems to me that the tokenization is never triggered there, and I can't understand where tokenization is called.

Anyone might know how to solve this?

[mttk]

The tokenization for each Field is called in the example.from* class here. What you are looking for can be done like this:

import torchtext
from torchtext.data import Example

# 1. Define the fields
TEXT = torchtext.data.Field(sequential=True, tokenize=lambda x: x.split())
LABEL = torchtext.data.Field(sequential=False)
fields = [('text', TEXT), ('category', LABEL)]

# 2. Manually construct examples
# The first argument is a list of Fields in the same order as the ones in the `fields` list.
# Alternatively, you can use .fromdict where the first argument would be a dict where the keys 
#  match the keys for the fields argument (and the fields argument is a dictionary)

sample_1 = Example.fromlist(['a simple sentence', 'label_1'], fields)
sample_2 = Example.fromlist(['any other thing', 'label_2'], fields)
texts = [sample_1, sample_2]

# 3. Manually construct the dataset
dataset = torchtext.data.Dataset(texts, fields=fields)

# 4. Build vocab
TEXT.build_vocab(dataset)
print(TEXT.vocab.freqs)

> Counter({'a': 1, 'simple': 1, 'sentence': 1, 'any': 1, 'other': 1, 'thing': 1})

This is the correct way to do this when manually (inline) constructing samples. When loading data from disk, you can use TabularDataset, where the Example construction is handled internally.

[davidsbatista]

A few minutes after posting here I realised my error and found the Example class from torchtext.data; Thanks anyway for your descriptive answer!