Gluon-nlp: [BERT] Functionality for BERT Pre-training

Created on 21 Dec 2018 · 5Comments · Source: dmlc/gluon-nlp

[x] support truncated_normal for better initialization for pre-training https://www.tensorflow.org/api_docs/python/tf/initializers/truncated_normal. We can follow the pytorch implementation https://github.com/pytorch/pytorch/pull/5015/files . And add inverse err function operator in MXNet: https://github.com/apache/incubator-mxnet/pull/13229
[x] support Adam with fixed weight decay
[ ] Update compare_tf_gluon.py with pre-training output for masked language model and next sentence prediction
[ ] Training and evaluation script

help wanted

Source

eric-haibin-lin

Most helpful comment

From Comprehend perspective, I have observed Text classification performance to be not so great for extremely short documents. This might be due to the fact that BERT is pretrained on long docs of wikipedia. It would definitely help comprehend custom classification in trying out pretraining BERT on additional data sources which cater additional domains like Twitter, Amazon reviews, kindle books & other noisy settings.

sravanbabuiitm on 29 Jan 2019

👍3

All 5 comments

Would someone be interested to take truncated_normal? I can PR the implementation for inverse err operator to MXNet, and the rest can be done in python I think

eric-haibin-lin on 28 Dec 2018