Text: [RFC] Subword models: BPE, Unigram subword models and Unicode equivalence

[cpuhrsch]

Motivation

There are multiple libraries that implement subword models within the compression-based space. There is fastBPE, SentencePiece, YouTokenToMe, etc.

As far as I can tell there are four major, orthogonal pieces these libraries implement together
1) White space substitution (e.g. replace whitespace with U+2581 or @@)
2) Unicode equivalence, e.g. NFKC
3) BPE as defined by https://arxiv.org/abs/1508.07909 (not actually using the bytes, but characters instead)
4) Subword segmentation using a unigram language model
as defined by https://arxiv.org/pdf/1804.10959.pdf

In particular I'd like to quote https://arxiv.org/pdf/1804.10959.pdf

BPE and the unigram language model share the same idea that they encode a text using fewer bits with a certain data compression principle (dictionary vs. entropy). Therefore, we expect to see the same benefit as BPE with the unigram language model. However, the unigram language model is more flexible as it is based on a probabilistic language model and can output multiple segmentations with their probabilities, which is an essential requirement for subword regularization.

Now, of course, there are also other approaches to dealing with subwords such as creating subword dictionaries (similar to regular perfect hash-table dictionaries) or via the hashing trick (imperfect hash-table).

All of these pieces can be backed by various "engines" that change and compete with each other in performance over time. However, I think we could find an opportunity here to create a simple, orthogonal interface for some of these well-established concepts. Not only can be then pick the best engine, we can also create something PyTorch specific that e.g. returns torch.Tensors for direct numericalization.

In particular there are a few questions I think are worth asking
1) Are these pieces comprehensive to model the referenced approaches?
2) Are there many minute differences between these implementations that makes this simple split impossible?

[vincentqb]

By the way, when loading a dataset, do we expect any of the transforms to be applied (say white space substitution), or do we let the user apply the ones desired?

[zhangguanheng66]

I agree that we should create a simple interface to support those subword models. It could be a subward transform to process the raw text data.

[zhangguanheng66]

By the way, when loading a dataset, do we expect any of the transforms to be applied (say white space substitution), or do we let the user apply the ones desired?

For basic users, we provide them the data that are ready for use. At the same time, we keep the flexibility for advanced user to choose the transforms. At least for tokenizers, each dataset has a default one but subject to change.

[chenyangyu1988]

@zhangguanheng66 @vincentqb I agree that we should keep flexibility for advanced user to choose the transforms. Usually a pre-trained model is tightly coupled with a transform, for example: RoBERTa is using gpt-2 BPE for pre-training, so we also need to use the same transform when doing fine-tuning no matter the dataset.

[cpuhrsch]

For now I'd treat all of these pieces as functionals. Whitespace substitution and unicode equivalence are straightforward string manipulations, so they can be a few functions for the different options. BPE and the unigram subword language model could be viewed as compression techniques for existing vocabularies given a certain dataset. They could be transforms for vocabularies with a certain interface.