Models: 🙏 [Help wanted for Model Garden] MobileBERT (a Compact Task Agnostic BERT for Resource Limited Devices)

[jaeyounkim]

Help wanted: Research paper code and models

If you want to contribute, please leave a comment to express your interest.

Research paper

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Current implementation

• TensorFlow 1 code at Google Research repository

Status

• In progress

Mentor

• Hongkun Yu (@saberkun)

Contributors

• @vufg

Tasks

• Convert the TensorFlow 1 checkpoints to TensorFlow 2
  
  
  
  • Optimized Uncased English: uncased_L-24_H-128_B-512_A-4_F-4_OPT
  
  
• Reproduce results on finetuning tasks
• Convert a pre-trained model to a TensorFlow Lite format

Development environment

• Development branch: mobilebert

  • Development work should be done using the mobilebert branch.
  • All changes will be merged into the master branch after finishing the implementation.

• Target

  • Repository: Official NLP directory
  • Directory: https://github.com/tensorflow/models/tree/master/official/nlp/mobilebert

Requirements

• Follow our contribution guidelines

License

By contributing, you agree that your contributions will be licensed under Apache License 2.0.

[saberkun]

https://github.com/huggingface/transformers/issues/4185

[vufg]

Working on it

[Harsh188]

https://github.com/huggingface/transformers/pull/4901#issue-432363045 Seems to have completed the requirements?

[chenGitHuber]

Hi Harsh188,

Thanks for your interest. We have released the mobile_bert_encoder.py and will release the converted checkpoint and more soon.

huggingface/transformers#4901 (comment) Seems to have completed the requirements?