Ludwig: [Feature Request] Allow using custom languages/models for spaCy NLP

[ZeroAurora]

Is your feature request related to a problem? Please describe.
Other related issues: #408 #251
I trained a Chinese model for spaCy, linked it to [spacy's package folder]/data/zh (using spacy link) and want to use that for ludwig. However, when I tried to set the config for ludwig, I received an error, which tell me that there is no way to load the Chinese model.

ValueError: Key chinese_tokenizer not supported, available options: dict_keys(['characters', 'space', 'space_punct', 'underscore', 'comma', 'untokenized' (...) 'bert'])

Describe the use case
By allowing using custom languages for spacy, users using other language would be able to process their texts quicker and easier.

Describe the solution you'd like
Here's the current solution...

input_features:
  -
    name: input
    type: text
    preprocessing:
      word_tokenizer: english_tokenize

...which I think could be changed to this...

input_features:
  -
    name: input
    type: text
    preprocessing:
      word_tokenizer: spacy_tokenize
      spacy_model: zh #(or en, xx, etc.)

Describe alternatives you've considered
I've considered not to use spacy but to use a custom script to simply split sentences to words using some processors like "jieba". However, by using this method I would lose nearly all benefits from NLP.

Additional context
I think that's all :)
I don't know whether my advice could be accepted. But if it got solved I would be very thankful.
BTW since I'm not a native English speaker, there may be some mistakes. Please don't mind it :p

[w4nderlust]

We are thinking about changing the way you define the tokenizer to be more flexible, and that would allow you to do what you are looking for.

In the mean time, if you are using the API, you can do the following:

from ludwig.utils.nlp_utils import language_module_registry
from ludwig.utils.strings_utils import tokenizer_registry
from ludwig.utils.strings_utils import BaseTokenizer

language_module_registry['zh'] = 'your_model_name'  # for example 'en_core_web_sm'

class ChineseTokenizer(BaseTokenizer):
    def __call__(self, text):
        return process_text(text, load_nlp_pipeline('zh'))

tokenizer_registry['chinese_tokenizer'] = ChineseTokenizer

After you do this, you can refer to chinese_tokenizer in your model configuration within the same script where you run the code above.

[w4nderlust]

In the meantime, spaCy added Chinese as language in release 2.3, this commit https://github.com/uber/ludwig/commit/2ad48fd896a8c46ccfb91c3ffd448f0c442985ee adds new Chinese preprocessing to Ludwig.
We are still thinking about adding a more flexible way to specify custom spacy models though.

[brightsparc]

Hi @w4nderlust, would this be a way to also support the new fast tokenizer to improve speed up pre-processing (esp for multi-threadding)?

from ludwig.utils.strings_utils import BaseTokenizer
from ludwig.utils.strings_utils import tokenizer_registry
from tokenizers import BertWordPieceTokenizer # Import fast tokenizers

class FastTokenizer(BaseTokenizer):
    def __init__(self,
                 pretrained_model_name_or_path,
                 **kwargs
                 ):
        super().__init__()
        self.tokenizer = BertWordPieceTokenizer(f"{pretrained_model_name_or_path}/vocab.txt", **kwargs)

    def __call__(self, text):
        return self.tokenizer.encode(text).ids

tokenizer_registry['fast_tokenizer'] = FastTokenizer

How would this work when using the ludwig CLI, is there a way to configure this - I wondering if this something that you could bundle with the main codebase?

Alternatively, there is an option to pass the "use_fast" argument for AutoTokenizer which could be enabled if you add the library to the distribution.

        self.tokenizer = AutoTokenizer.from_pretrained(
            pretrained_model_name_or_path, use_fast=True
        )

see: https://github.com/uber/ludwig/blob/master/ludwig/utils/strings_utils.py#L1166

[w4nderlust]

@brightsparc the code you posted will likely work. To make it available through the CLI one would have to add it among the other tokenizers in string_utils.py.
Although my understanding is that in the next version of transformers, the fast tokenizers will be the default, so maybe that would be sufficient.
Alternatively, one could add the use_fast parameter to init as you have shown, it would work, but there's a bit more to it, one would also need to add that parameter to the functions using the tokenizer, which requires a bit more work, although not too much.
If you are interested in helping out on this I can point you in the right direction for a PR.

[brightsparc]

Okay, I tested with adding this custom tokenizer in the registry and it worked.

@w4nderlust If use_fast will become the default, can we just add this to the AutoTokenizer for now, or are you saying that you would like to be able to pass this down through configuration?

[w4nderlust]

@brightsparc good to hear!

Regarding the use_fast I'm not 100% sure why it is not the default already, but if it isn't there's probably some compatibility issue they are ironing out, so setting it to true in Ludwig by default before HF does doesn't seem a good idea to me. So either we wait or we set it to false by default and allow people to pass it down through configuration (which, again, is not complicated to implement).

[brightsparc]

Hi @w4nderlust I am happy to support a PR to pass through configuration, but it's not clear how I would go about that. I see there is a kwargs parameter - is that were I should check the value from? I also notice that truncate and padding is setup to true in the code, should these also be optional arguments?

[w4nderlust]

truncate and padding should be set like that. @ANarayan can give more context, but I believe they shouldn't be touched.
Regarding use_fast, one would add it to function signature, and then look for all the occurrences of the tokenizers_registry that are related to tet features (for instance inside create_vocab and add_data and add_metadata inside text_feature.py) and add the parameter there too. Finally, add the parameter to defaults.py.
Let me know if this helps, otherwise I can also look into it myself, no worries.

[ANarayan]

Hi @brightsparc , so truncation is set to true because when working with pre-trained encoders, if length of the encoded sequence is greater than the model's maximum admissible input size, this leads to errors in downstream training. With regards to the padding being set to true, sequences in Ludwig are tokenized one at a time and the padding is done internally so the value assigned to padding doesn't really matter. Hope this helps!

[brightsparc]

Okay thanks @ANarayan this makes sense. @w4nderlust I have created a PR #1012 for this fix and have tested with my fork.

Also, FYI I am working on a AWS SageMaker example that I should be able to share with the community shortly.