Transformers: Using whole word masking on training LM from scratch

[uunal]

❓ Questions & Help

Details

Hello everyone,
I wanted to use _whole-word-masking_ in training LM from scratch. I could not have found how to apply this option using Trainer.
I thought this option should be managed in "class DataCollatorForLanguageModeling", but I could not find options for _whole-word-masking._
Am I looking at wrong place OR it is not implemented yet?
If not, is it possible to do with run_language_modeling.py?

A link to original question on Stack Overflow: https://stackoverflow.com/questions/62061578/how-to-use-whole-word-masking-on-training-lm-from-scratch

Any help is appreciated!
Thanks

[usuyama]

I think it's not implemented yet.

@julien-c any suggestion/thoughts for pretraining with wwm?

[usuyama]

NVIDIA/Megatron-LM does wwm on the fly in __ getitem __

We can do something similar in DataCollatorForLanguageModeling or in the dataset

https://github.com/NVIDIA/Megatron-LM/blob/22c0e300670672e4e0a8604bd6ab89bc28c970a6/megatron/data/bert_dataset.py#L148

[uunal]

Thanks for the suggestion, I'll look into it.

[luffycodes]

@usuyama The Megatron example is for the BERT dataset which uses wordpiece tokenization. Any suggestions how to do wwm for GPT2 tokenizer?

[usuyama]

related #6491

[uunal]

Check if still looking for an answer:
https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/data/data_collator.py#L301