Gluon-nlp: ALBERT

[xf05888]

ALBERT

Official Implementation: https://github.com/google-research/google-research/tree/master/albert

[sxjscience]

I'm working on implementing ALBERT using the new numpy interface of MXNet. I haven't confirmed whether the mentioned repo could reproduce the result.

[eric-haibin-lin]

I am not sure if we should release ALBERT with the numpy interface in gluonnlp. That somehow creates a diverging experience and inconsistent APIs in gluonnlp.

[sxjscience]

I’m trying to implement it in a separate repo as part of a research project... So I could better figure out how to use numpy.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Haibin Lin [e-mail]
Sent: Wednesday, October 2, 2019 10:46:48 PM
To: dmlc/gluon-nlp [e-mail]
Cc: Xingjian SHI [e-mail]; Assign [e-mail]
Subject: Re: [dmlc/gluon-nlp] ALBERT (was: How to convert keras ALBERT model to mxnet?) (#955)

I am not sure if we should release ALBERT with the numpy interface in gluonnlp. That somehow creates a diverging experience and inconsistent APIs in gluonnlp.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/issues/955?email_source=notifications&email_token=ABHQH3W7B2HVJPHOYYSHYILQMWBMRA5CNFSM4I4QUGTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAHCF3Y#issuecomment-537797359, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABHQH3QQNTCTVDRISUTKDCLQMWBMRANCNFSM4I4QUGTA.

[kaonashi-tyc]

Is it possible to including the Embedding Factorization as part of the existing BERT model as an option? It might be more tricky to have parameter sharing in the current BERTEncoder, since it will affect how the model performs.

[szhengac]

@kaonashi-tyc. To add Embedding Factorization, we only need to do two things: 1) reduce embedding size for Embedding layer; 2) add a Dense layer before and after the first and the last multi-head attention layers, respectively.

[xf05888]

Well, Is there a mechanism that can convert Tensorflow or Pytorch checkpoint to MXNet?

[leezu]

You can take a look at the conversion utils https://github.com/dmlc/gluon-nlp/tree/master/scripts/bert/conversion_tools and https://github.com/dmlc/gluon-nlp/tree/master/scripts/language_model/conversion_utils

[kaonashi-tyc]

@szhengac the only complexity would be that we are re-using the Dense layer in the decoder setting. Some modification is to get_decoder call.

I can commit to offer a PR if possible.

[vdabravolski]

Wonder if anyone is still working on this ticket? @sxjscience?

[sxjscience]

@vdabravolski I'm working on it for the new numpy version of GluonNLP, which facilitates the new numpy-like programming interface of MXNet. Currently it is still in a private repository and I'll try to upstream it as soon as possible. This will be a major update to the whole package.

[qqaatw]

Wondering how is the progress of this ticket? @sxjscience

[sxjscience]

See https://github.com/dmlc/gluon-nlp/pull/1225 .