Farm: Slow Tokenizer if custom vocab was added

[tholor]

Describe the bug
Tokenizer becomes very slow with large custom vocab.

Additional context
This was introduced after switching to the Tokenizers from the transformers repo

There are related issues reported in the transformers repo:

• https://github.com/huggingface/transformers/issues/1830
• https://github.com/huggingface/transformers/issues/1621
• https://github.com/huggingface/transformers/pull/1881

To Reproduce

• Add custom vocab to tokenizer via tokenizer.add_tokens()
• Load some data into the data silo, e.g. run examples/lm_finetuning.py

System:

• OS: Ubuntu 18.04
• GPU/CPU: Both
• FARM version: master @ 484d26c5d6b01eb5ee59b793cd6ee8597af8b808

[tholor]

There are new fast tokenizers for BERT implemented in rust: https://github.com/huggingface/transformers/pull/2211

We should have a look if they are compatible and solve the issue with custom vocab here.

[julien-c]

Yes, please check it out and let us know here.

Repo is at https://github.com/huggingface/tokenizers

[tholor]

Hey @julien-c,

I finally found some time to test. Seems really promising! Great work!

:heavy_check_mark: Same tokenization behaviour as BertTokenizer (see test)
:heavy_check_mark: Speed: ~ 7.8 x faster! (Tested via tokenizing SQuAD train set with 42 Mio chars)
:white_check_mark: Speed remains the same with custom vocab < 300. Somehow it's about 4x slower for custom vocab = 400 (using add_tokens())

The only blocker for us right now:
:x: The Tokenizer objects can't be pickled and are therefore not usable with python's multiprocessing. As we make heavy use of multiprocessing during preprocessing, we can't really use them right now. Seems that others have a similar issue. Not sure how much of work is needed for fixing this, but for the XLM-R python tokenizer it was a very easy fix.

[julien-c]

Hi @tholor, mind opening an issue on tokenizers too, cross-referencing this one? cc @n1t0

[salmanmashayekh]

In the add_tokens method, why don't we simply integrate new_tokens into the self.vocab? We are using the following CustomVocabBertTokenizer and it does not slow down when new_tokens are added:

from transformers import BertTokenizer, WordpieceTokenizer
from collections import OrderedDict


class CustomVocabBertTokenizer(BertTokenizer):
    def add_tokens(self, new_tokens):
        new_tokens = [token for token in tokens if not (token in self.vocab or token in self.all_special_tokens)]

        self.vocab = OrderedDict([
            *self.vocab.items(),
            *[
                (token, i + len(self.vocab))
                for i, token in enumerate(new_tokens)
            ]
        ])

        self.ids_to_tokens = OrderedDict([(ids, tok) for tok, ids in self.vocab.items()])
        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab, unk_token=self.unk_token)

        return len(new_tokens)

[tholor]

Hi @salmanmashayekh, sorry for the late reply. I haven't seen your comment until now.
This seems like a simple, scalable workaround. However, I am not sure if it has any unintended side effects (e.g. on saving/loading) in Transformers. Have you investigated the behavior in Transformers?
It could make sense to raise a PR there, since this is something useful to everybody and not only FARM users.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs.