Opennmt-py: how to integrate wordpiece?

[mrpega]

I'm thinking of integrating the BERT model hidden states from (https://github.com/huggingface/pytorch-pretrained-BERT) but BERT takes in wordpiece while current OpenNMT-py takes in words. Any ideas or simple hacks to get around it?

Thanks!

[rishibommasani]

OpenNMT-py doesn't specify a tokenization schema - If you want to use a word-level tokenization algorithm (such as the Moses/NLTK/SpaCy/BERT BasicTokenizer in the Pytorch-pretrained-BERT repo) or a subword tokenization algorithm (such as WordPiece or BPE), you should do that and then format the output to have one example per line with whitespaces separating words/subwords/whatever your tokens are. Refer to http://opennmt.net/OpenNMT-py/extended.html where the Moses Tokenizer is used.

[guotong1988]

Same question. Thank you!

[guotong1988]

"whitespaces separating words/subwords/whatever your tokens are"