Gluon-nlp: subword methods (BPE/wordpiece)

[szha]

I'd like to start a discussion on how we integrate subword methods such as BPE and wordpiece.

I've been working on some BPE tokenization with https://github.com/google/sentencepiece and have been quite happy with it. It provides pip packages on all platforms and is shipped with the binary inside. It carries an apache 2.0 license.

The places where subword methods are needed are subword functions in vocab and tokenizer/detokenizer.

cc @leezu @szhengac

[szha]

The encode function in sentencepiece supports both returning the token and returning the index, which conforms to our tokenizer interface and vocab interface, respectively.

[szhengac]

I think subword methods need to be separated from tokenizer/detokenizer, because tokenizer/detokenizer is data independent, while learning BPE/wordpiece is data dependent. In general, the learning of subwords is performed after the tokenization.

[szha]

I agree that BPE is slightly different from tokenizer/detokenizer, but for a different reason. BPE does a bit more than tokenization in that it adds boundary characters to mark the start and end of words. Considering the case where we need to use pre-trained BPE, the action it performs fits the transform function abstraction, just like tokenizer. IMO we can first deal with using learned BPE first so that MT models can be used to perform inference on new data.

[szha]

380

[szha]

We got the sentencepiece integration now. For subword-nmt, I haven't picked this up yet so it's unlikely I will get to it in 0.5.0.

[leezu]

I intend to add a wrapper for https://github.com/VKCOM/YouTokenToMe