Gluon-nlp: gluonnlp.Vocab hard coded id of <unk> token

[hankcs]

Hi gluon-nlp team,

gluonnlp.Vocab always assigns 0 as id of <unk>, causing trouble in some situations where non-zero ids represent valid tokens. The same goes to other special tokens. When I need a mapping of PAD, ROOT, UNK = 0, 1, 2, the current implementation doesn't allow this. Besides, <unk> will be used in decoding phase, so I can't simply disable it.

Is there any possibilities for a new feature enabling flexible assignment of ids to special tokens?

Thanks.

[szha]

Yes, I think allowing flexible index assignment would be very useful, especially if we could reassign unknown token. We are likely going to still keep the assumption that the index of each token is unique, and spans from 0 to vocab_size - 1

[szha]

@astonzhang @leezu do you have some insights to share on this?

[astonzhang]

If you do not want out-of-vocab token to map to UNK, specify unknown_token to None in Vocab:
https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/vocab/vocab.py#L141

Then you can specify unk-like string like '[unk]' or 'unk' or 'Unk' in your vocab, which will be treated as a normal/valid token.

One of our future work may consider using extra_tokens in Vocab arguments to allow users to append special tokens like PAD, BOS, and EOS. However, UNK may act differently because some users want any out-of-vocab token to map to UNK, while others may just treat UNK as a valid token. Thus, we may separate its behavior from those of PAD, BOS, EOS, ROOT etc.

[szha]

@hankcs described the use case to me that he needs to use a special token ROOT to derive a mask, while handling unknown tokens. He wants to assign ROOT to be 0.

One solution that I can think of is to allow users to specify unknown token both in the unknown_token field and the reserved_tokens field, and Vocab class can use the later to decide what index these special tokens should have.

[eric-haibin-lin]

This is also a problem for pre-trained models by others such as BERT. This pre-trained model comes with parameters and vocab, which whose UNK index is not 0. It would be much more flexible if we can change unk idx in gluon nlp vocab.

[leezu]

Note that we need to update some of the files on S3 when removing the hard-coding.

https://github.com/dmlc/gluon-nlp/blob/c80b7712e3c00b81db318981d46ea9cb49471e5b/src/gluonnlp/_constants.py#L33-L35