Text: Getting multiple elements from a single Field

[pasinit]

Hi all, I am processing text fields with huggingface tokenizers and this works great. Besides the token ids, however, I also need to compute at tokenization time the token type ids corresponding to each token ad to return them so to feed them to the network. Therefore, in order not to duplicate the tokenization phase, I need my text field processor to output two different things, tokens and token type ids. Is anyone aware if this is supported or if there is any work around to cope with this problem?

Thanks for your help in advance!

[zhangguanheng66]

@mttk @bentrevett

[bentrevett]

I don't think this is possible to do with a single field. You will have to have two different fields which means, unfortunately, each example will have to be tokenized twice.

Using a pre-trained tokenizer:

#assuming data.csv is something like:
#    pos, this film was great
#    neg, the worst film i have ever seen
#    pos, wow five stars easily
#    neg, what a waste of my life

from tokenizers import BertWordPieceTokenizer
from torchtext import data
import functools

tokenizer = BertWordPieceTokenizer("pretrained_vocab.txt") #or whichever you want to use

def get_token_ids(tokenizer, s):
    output = tokenizer.encode(s)
    return output.ids

def get_type_ids(tokenizer, s):
    output = tokenizer.encode(s)
    return output.type_ids

token_id_tokenizer = functools.partial(get_token_ids, tokenizer)
type_id_tokenizer = functools.partial(get_type_ids, tokenizer)

#you also need to correctly set the unk and pad tokens here
TOKENS = data.Field(tokenize=token_id_tokenizer, use_vocab=False)
TYPES = data.Field(tokenize=type_id_tokenizer, use_vocab=False)
LABEL = data.LabelField()

fields = [('label', LABEL), (('token_ids', 'type_ids'), (TOKENS, TYPES))]

train_data, test_data = data.TabularDataset.splits(
                            path = '',
                            train = 'data.csv',
                            test = 'data.csv',
                            format = 'csv',
                            fields = fields
)

You might be able to do it with a single tokenize by hacking around using Example.fromX from here, but I am unsure.

[pasinit]

Hi @bentrevett thanks for your answer. What I did was something similar to what you suggested at the end by manually creating an Example via the setattr() function. Furthermore, I also changed the batch_init function of torchtext.data.Batch so as it can handle fields that do not need to be processed. Now it simply has a meta_field field so that one can simply set fields without any processor and still they are passed to the network. It is a bit hacky but I think it adds a lot of flexibility. I am posting my code here in case anyone can find it useful.

Change Batch batch_init method

# in the  main file
def my_batch_init(self, data=None, dataset=None, device=None):
    if data is not None:
        self.batch_size = len(data)
        self.dataset = dataset
        self.fields = dataset.fields.keys()  # copy field names
        self.input_fields = [k for k, v in dataset.fields.items() if
                             v is not None and not v.is_target]
        self.target_fields = [k for k, v in dataset.fields.items() if
                              v is not None and v.is_target]
        self.meta_fields = [k for k, v in dataset.fields.items() if v is None]
        for (name, field) in dataset.fields.items():
            if field is not None:
                batch = [getattr(x, name) for x in data]
                setattr(self, name, field.process(batch, device=device))
            else:
                batch = [getattr(x, name) for x in data]
                setattr(self, name, batch)
setattr(Batch, "__init__", my_batch_init)

Manually create example:

# within a Dataset subclass
@staticmethod
 def _load_dataset(path, tokenizer, start_tok, sep_tok, end_tok, max_len, **kwargs):
        examples = list()

        with jsonlines.open(path) as lines:

            for example in tqdm(lines):
                label = example["answerKey"]
                question = example["question"]
                context = question["stem"]
                for choice in question["choices"]:
                    choice_label = choice["label"]
                    choice_text = choice["text"]
                    tokens, type_ids = fun_tti(tokenizer, context, choice_text,
                                               start_tok, sep_tok, end_tok, max_len, **kwargs)

                    e = Example()
                    setattr(e, "id", example["id"])
                    setattr(e, "cqa", tokens)
                    setattr(e, "tti", type_ids)
                    setattr(e, "label", label == choice_label)

                    examples.append(e)
        return examples


def fun_tti(tokenizer, c, a, start_token, sep_token, end_token, max_len, **kwargs):
    context, answer = c, a

    c = [start_token] + tokenizer.tokenize(context.lower(), **kwargs)
    c += [sep_token]
    a = tokenizer.tokenize(answer.lower(), **kwargs) + [end_token]
    if max_len is not None:
        l = len(c)+len(a)
        if l > max_len:
            c = c[:-(l - max_len + 1)] + [sep_token]
    cqa = c +  a
    assert max_len is None or len(cqa) <= max_len
    tti = [0] * len(c) + [1] * len(a)

    return cqa, tti