Gluon-nlp: [BERT] Data generation script enhancement for large corpus

[eric-haibin-lin]

We need to enhance the data generation script for BERT. Specifically,

• avoid padding during dataset generation to reduce I/O and storage overhead
• use numpy for fast serialization
• use process pool for parallel processing

[eric-haibin-lin]

505

[szha]

I wonder if we should provide the processed dataset to the community. Do you think that would be useful?

[szha]

The complication is in book corpus. I'm not sure what we can do to redistribute its derivatives.

[eric-haibin-lin]

we can at least offer wikipedia, or the scripts for preprocessing wikipedia if there is any license restriction

[AnakTeka]

we can at least offer wikipedia, or the scripts for preprocessing wikipedia if there is any license restriction

I'm trying to do pre-train for BERT, but steps on this page https://gluon-nlp.mxnet.io/model_zoo/bert/index.html does not explain how to prepare the dataset for training using Wikipedia, although the is sample_text.txt for I can't figure it out how to prepare wikipedia dumps to match sample_text.txt format, any help?

[eric-haibin-lin]

@AnakTeka are you able to use the pre-processed wikipedia dataset here http://gluon-nlp.mxnet.io/master/model_zoo/bert/index.html#pre-training-from-scratch? Or you want to pre-process from scratch by yourself?

[AnakTeka]

@AnakTeka are you able to use the pre-processed wikipedia dataset here http://gluon-nlp.mxnet.io/master/model_zoo/bert/index.html#pre-training-from-scratch? Or you want to pre-process from scratch by yourself?

I'm trying to pretrain on non English language

[eric-haibin-lin]

@AnakTeka you can find some (not actively maintained) preprocessing script in https://github.com/eric-haibin-lin/text-proc