Gluon-nlp: BPE's default alpha with sentencepiece

Created on 8 Jun 2020  Β·  2Comments  Β·  Source: dmlc/gluon-nlp

Description

As BPE-dropout is applied to sentencepiece recently, it can be tokenized based on sampling.
alpha = 1 is for optimally training not for inference.

Default alpha=1 isn't appropriate because most users and models provided by gluonnlp expect to deterministically tokenize.

https://github.com/google/sentencepiece/issues/371

To Reproduce

path = gluon.utils.download('https://kobert.blob.core.windows.net/models/kogpt2/tokenizer/kogpt2_news_wiki_ko_cased_818bfa919d.spiece')
tok = nlp.data.SentencepieceTokenizer(path)
tok('μ•ˆλ…•ν•˜μ„Έμš”.')  
['▁', 'μ•ˆ', 'λ…•', 'ν•˜', 'μ„Έ', 'μš”', '.']
tok = nlp.data.SentencepieceTokenizer(path, 0, 0.5)                                                                                                                                              
tok('μ•ˆλ…•ν•˜μ„Έμš”.')                                                                                                                                                                               
['▁', 'μ•ˆ', 'λ…•', 'ν•˜', 'μ„Έμš”', '.']
tok('μ•ˆλ…•ν•˜μ„Έμš”.')                                                                                                                                                                               
['β–μ•ˆ', 'λ…•', 'ν•˜', 'μ„Έμš”', '.']
tok('μ•ˆλ…•ν•˜μ„Έμš”.')                                                                                                                                                                               
['β–μ•ˆλ…•', 'ν•˜', 'μ„Έμš”', '.']
tok('μ•ˆλ…•ν•˜μ„Έμš”.')                     
tok = nlp.data.SentencepieceTokenizer(path, num_best=0, alpha=0)
tok('μ•ˆλ…•ν•˜μ„Έμš”.')                                                                                                                                                                               
['β–μ•ˆλ…•ν•˜μ„Έμš”', '.']
bug good first issue
Source
picture haven-jeon
👍2

Most helpful comment

Nice observation, I've fixed it in the new version: See https://github.com/dmlc/gluon-nlp/pull/1225

picture sxjscience on 10 Jun 2020
👍2

All 2 comments

agreed. feel free to propose a PR to update this.

szha picture szha on 8 Jun 2020

Nice observation, I've fixed it in the new version: See https://github.com/dmlc/gluon-nlp/pull/1225

sxjscience picture sxjscience on 10 Jun 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rondogency picture rondogency  Β·  4Comments
zburning picture zburning  Β·  3Comments
imgarylai picture imgarylai  Β·  7Comments
ktoetotam picture ktoetotam  Β·  4Comments
szha picture szha  Β·  4Comments