Datasets: Replacement for tfds.deprecated.text.SubwordTextEncoder

[stefan-falk]

Unfortunately there is no statement addressing the deprecation of tfds.deprecated.text.SubwordTextEncoder.

Why was the SubwordTextEncoder deprecated? Will there be a replacement and what can/should we use instead?

[IIIBlueberry]

I think it was replaced by tensorflow/text

[stefan-falk]

@IIIBlueberry Hm, they do not seem to have a SubwordTextEncoder yet as far as I can see.

There's discussions about it in https://github.com/tensorflow/text/issues/417 but it seems nothing has materialized so far.

[Conchylicultor]

We've warned of this changed in our release notes and were raising warning when user was using it.

The reason it is deprecated is because it was non maintained, had known bug and performance issues.

Tensorflow text has a few subword tokenizer, like text.BertTokenizer or SentencepieceTokenizer. There is also pretrained tokenizer that you can install from TF-Hub: like https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/2

[stefan-falk]

Indeed, it was stated in v3.2.0.

BertTokenizer and SentencepieceTokenizer do not look like they can create a vocabulary for you so I guess this is something we have to do on our own now - except I am overlooking something.

@Conchylicultor thank you for clarification though. I guess my question is answered. :)

[Conchylicultor]

Yes, creating vocabulary isn't straightforward now. I believe TF-Text plan to make it easier. You should submit an issue on their repository to ask them directly.

[stefan-falk]

@Conchylicultor Will do, thank you! :)

[MarkDaoust]

I believe TF-Text plan to make it easier.

Yes. TFText has all the necessary classes to replace this. It's also implemented in tensorflow so you can export this preprocessing with the model.

I'm working on getting a tutorial published on how to do this.