Gluon-nlp: Data and preprocessing for BERT

[mandarjoshi90]

Hi folks,
Thanks for the fantastic effort for reproducing BERT. Would it be possible to share the corpus details and preprocessing code? More specifically,

1. Wikipedia dump version and extraction code
2. Books preprocessing

Alternatively, if you could share the txt files used for create_pretraining_data.py, that works too!

Thanks!

[eric-haibin-lin]

Hi @mandarjoshi90 I am cleaning up the preprocessing script and extraction code. Stay tuned!

[eric-haibin-lin]

Hi @mandarjoshi90

Unfortunately I cannot re-distribute books corpus dataset due to license issues. I uploaded the processed texts for English wikipedia at s3://bert-pretraining/datasets/enwiki/enwiki-dec-doc/, which can be used for create_pretraining_data.py. The data is based on the wikipedia dump on 2018 December.

I will PR the preprocessing code after I test it locally.

[mandarjoshi90]

Thanks so much!

[forresti]

For other datasets (e.g. squad, mnli, imdb, ...), gluon-nlp has scripts to download and preprocess the data to enable reproducible research. Would it be possible to create such a script that enables a user to automatically download and build the Wikipedia dataset?
(I believe the Books dataset is not officially online anymore.)

I have a feeling that there will be a lot of people doing the pretraining soon.

One other question: What's the exact URL of the s3://bert-pretraining/datasets/enwiki/enwiki-dec-doc/ file that @eric-haibin-lin mentioned in the above post?

[eric-haibin-lin]

@forresti I removed public access to the bucket due to security concern. I’m happy to copy the cleaned-up Wikipedia dataset to a public bucket if that’s useful for you.

Do you still need the script to cleanup the corpus?

[forresti]

@eric-haibin-lin
That would be fantastic if you could provide the cleaned up dataset.

For your long-term solution to making it easy to pretrain BERT models, I see at least 2 possibilities:
(1) Gluon-NLP could provide a static cleaned up Wikipedia dataset that is compatible with scripts/bert/run_pretraining.py. And, run_pretraining.py could look for the cleaned up dataset and download it if it's not already on disk.
(2) Gluon-NLP could provide a script that downloads the latest Wikipedia dump and then cleans it up in such a way that it becomes compatible with scripts/bert/run_pretraining.py.

I think (1) is a more attractive solution because it allows the whole community to use exactly the same data. But, long-term, maybe it'd be worthwhile to have both.

[forresti]

I'm working on preparing a Wikipedia dataset for pretraining.

Here's what I've done so far:

# get latest wikipedia dump
wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2

# https://github.com/attardi/wikiextractor
python wikiextractor/WikiExtractor.py enwiki-latest-pages-articles.xml.bz2 # saves in /path/to/text

# create a tiny training set (using just 1 of the ~13000 files generated by WikiExtractor)
python gluon-nlp/scripts/bert/create_pretraining_data.py --input_file /path/to/text/AA/wiki_00 --output_dir /path/to/train --dataset_name book_corpus_wiki_en_uncased --dupe_factor 10 --num_workers 16

Now, if I look at what's produced by WikiExtractor, it looks something like this:

<doc id="12" url="https://en.wikipedia.org/wiki?curid=12" title="Anarchism">
Anarchism 

Anarchism is an anti-authoritarian political philosophy that rejects hierarchies deemed unjust and advocates their replacement with self-managed, ...

I notice two things here:

1. Before each article, there's a <doc id=...>, and after each article, there's a </doc>.
2. Before each article is a title (which is often just one word, and at most a few words).

So, here's my question to anyone who's done this before:
When you preprocess the Wikipedia data, do you also use something like WikiExtractor? If so, do you typically remove the <doc> and Title lines before running create_pretraining_data.py?

Finally: Once I get this all running correctly, I'll try to automate the Wikipedia preprocessing procedure (for possibility (2) above), and if it works, I will submit a PR. (Or... if one of the real Gluon-NLP developers is working on a script for this, then I'm sure your version will be better.)

[eric-haibin-lin]

Hi @forresti sorry for the late response. I was traveling for the past few days. I pushed some text cleansing code in https://github.com/eric-haibin-lin/text-proc (the doc is out-dated, but you have some idea what's inside).

Yes I used WikiExtractor, and replaced the <doc> token with newlines before running the training code. And, actually, create_pretraining_data.py is not necessary anymore, because now the training script supports generating pre-training samples on the fly (see http://gluon-nlp.mxnet.io/master/model_zoo/bert/index.html#run-pre-training)

Currently the code in https://github.com/eric-haibin-lin/text-proc is not very organized, if you can submit a PR with a script that automates the whole thing, that will be awesome.

In the meanwhile, I'll upload the pre-processed Wikipedia dataset tomorrow

[forresti]

Thanks! I will start working with your new code tomorrow.

[eric-haibin-lin]

uploaded to https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/enwiki-197b5d8d.zip
@forresti please let me know if the link doesn't work

[aninrusimha]

Hello! I tried the link that you mentioned here, but was going through the data and encountered a problem with how the sentences were split. It looks to be based on characters, and this leads to sentences being split as follows:

The word "anarchism" is composed from the word "anarchy" and the suffix -ism, themselves derived respectively from the Greek , i.e.
"anarchy" (from , "anarchos", meaning "one without rulers"; from the privative prefix ἀν- ("an-", i.e.
"without") and , "archos", i.e.
"leader", "ruler"; (cf.
"archon" or , "arkhē", i.e.
"authority", "sovereignty", "realm", "magistracy")) and the suffix or ("-ismos", "-isma", from the verbal infinitive suffix , "-izein").

Does this matter? I'll try to run it through the nltk punkt tokenizer (http://www.nltk.org/_modules/nltk/tokenize/punkt.html), but it'll take me a while to try to run any sort of pre training.

[forresti]

uploaded to https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/enwiki-197b5d8d.zip
@forresti please let me know if the link doesn't work

Thanks!! I was able to download and extract the enwiki-feb data from this link.
It looks like the data has the <doc></doc> tags removed, but we'll still need to use your text-proc repo code to remove other tags such as <ref> and <mapframe>. No problem.

Also, @aninrusimha and I are going to collaborate on implementing and automating the preprocessing (the two of us trace our roots back to the same advisor at UC Berkeley, and we decided to collaborate). We hope to provide a PR for this once the bugs are worked out. I'm looking forward to hearing your thoughts on @aninrusimha's question above.

[eric-haibin-lin]

Hello! I tried the link that you mentioned here, but was going through the data and encountered a problem with how the sentences were split. It looks to be based on characters, and this leads to sentences being split as follows:

The word "anarchism" is composed from the word "anarchy" and the suffix -ism, themselves derived respectively from the Greek , i.e.
"anarchy" (from , "anarchos", meaning "one without rulers"; from the privative prefix ἀν- ("an-", i.e.
"without") and , "archos", i.e.
"leader", "ruler"; (cf.
"archon" or , "arkhē", i.e.
"authority", "sovereignty", "realm", "magistracy")) and the suffix or ("-ismos", "-isma", from the verbal infinitive suffix , "-izein").

Does this matter? I'll try to run it through the nltk punkt tokenizer (http://www.nltk.org/_modules/nltk/tokenize/punkt.html), but it'll take me a while to try to run any sort of pre training.

In general, I don't think that's a big issue, as during pre-training we're trying to fit as many consecutive sentences as possible in the 512 sequence length. So very likely you'll see something like

The word "anarchism" is composed from the word "anarchy" and the suffix -ism, themselves derived respectively from the Greek , i.e. "anarchy" (from , "anarchos", meaning "one without rulers"; from the privative prefix ἀν- ("an-", i.e. "without") and , "archos", i.e. "leader", "ruler"; (cf.

as the first segment in the "sentence pair" for the next "sentence" prediction task. I put quote around the word sentence because BERT is actually doing next segment prediction (with multiple sentences in a segment) instead of next sentence prediction.

[eric-haibin-lin]

@forresti nice catch. What I wrote in text-proc is not perfect, so this is a miss on my end. If you can add additional regex to filter out these <ref> and <mapframe>, that will be great.

[forresti]

@eric-haibin-lin

It looks like your script in the README of your text-proc filters for <ref> and <mapframe>. But, I don't think all the filters from your README were applied to your dataset. It won't be hard to fix, though.

[eric-haibin-lin]

Pre-processed openweb text available at http://repo.mxnet.io/gluon/dataset/webtext-split-86f9dbdd.zip

[RaitoBezarius]

Hi there, I wanted to know if there were statistics on the cost/time of training BERT including the tuning of hyperparameters, as this has not been mentioned in the original paper, neither by this reproduction, thank you in advance!

[eric-haibin-lin]

@RaitoBezarius it depends on the number of samples and dataset size. For QA on SQuAD v1.1 (100k samples, seq-len 384) with fp16 it takes about 45 mins with bert base, with a single V100 GPU, for a single run with 2 epochs.