Gluon-nlp: [Discussion] Roadmap

[szha]

Hi,

Let's start a discussion here about the roadmap towards 0.10 and 1.0. We are looking for:

• New features that are useful to your research
• Improvements and patches to existing features

If you have any item that you'd like to propose to have in the roadmap, please do:

• Create (or locate existing) issue for the item, note the issue number.
• Comment in this issue: 1) the above issue number, 2) one sentence of what the item is about and why it's useful to you.
• Indication on whether you'd be willing to help out on the item.

Features

The following features have been proposed to include in GluonNLP 0.7.0 (subject to change)

Models

Language modeling

• [ ] ALBERT https://github.com/dmlc/gluon-nlp/issues/955
• [x] SciBERT pre-trained models (#650)
• [x] Release GluonNLP's pre-trained models (#642)
• [ ] Korean BERT https://github.com/dmlc/gluon-nlp/issues/939
• [x] GPT/GPT-2 training from scratch (#592) OpenGPT-2
• [ ] [SRU](https://arxiv.org/abs/1709.02755)
• [ ] [ERNIE](https://arxiv.org/abs/1905.07129)
• [ ] XLNet training from scratch
• [ ] Transformer-XL training from scratch
• [ ] ALBERT
• [ ] XLM-Roberta

Word Embedding

• [ ] [Poincaré GloVE Embedding](https://arxiv.org/abs/1810.06546)
• [ ] [ICLR2019] Adaptive Input Representations for Neural Language Modeling
• [ ] [NIPS2017] Poincaré Embeddings for Learning Hierarchical Representations

NER

• [x] BERT for NER (#593 #612)

Memory networks and transformer

• [ ] Augmenting Self-attention with Persistent Memory
• [ ] [Large Memory Layers with Product Keys](https://arxiv.org/pdf/1907.05242.pdf)
• [ ] Adaptive Attention Span in Transformers (https://www.aclweb.org/anthology/P19-1032)
• [ ] Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (https://arxiv.org/pdf/1907.00235.pdf)
• [ ] related: Hybrid computing using a neural network with dynamic external memory
• [ ] related: Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Visualization

• [ ] Visualizing and Measuring the Geometry of BERT
• [ ] Understanding Black-box Predictions via Influence Functions
• [ ] [NeurIPS2019] Visualizing and Measuring the Geometry of BERT (https://arxiv.org/pdf/1906.02715.pdf)
• [ ] [Arxiv2019] Visualizing and Understanding the Effectiveness of BERT (https://arxiv.org/pdf/1908.05620.pdf)
• [ ] [Arxiv2019] What Does BERT Look At? An Analysis of BERT’s Attention (https://arxiv.org/pdf/1906.04341.pdf)
• [ ] related: Bertviz: https://github.com/jessevig/bertviz

Quantization

• [ ] [ICLR2019] Learning Recurrent Binary/Ternary Weights (https://openreview.net/pdf?id=HkNGYjR9FX)
• [ ] [JMLR2018] Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (http://www.jmlr.org/papers/volume18/16-456/16-456.pdf)

Multi-Task/Transfer Learning

• [ ] MT-DNN and multi-task learning (#633)
• [ ] [ICML2019] Parameter-Efficient Transfer Learning for NLP (http://proceedings.mlr.press/v97/houlsby19a.html)
• [ ] [ACL2019] MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (https://arxiv.org/pdf/1905.13453.pdf)

Machine translation

• [ ] Cross-lingual BERT (#657)
• [ ] [MASS](https://github.com/microsoft/MASS)
• [ ] [ACL2016] Incorporating Copying Mechanism in Sequence-to-Sequence Learning (https://arxiv.org/pdf/1603.06393.pdf)
• [ ] [EMNLP2018] Understanding Back-Translation at Scale (https://aclweb.org/anthology/D18-1045)
• [ ] [ICLR2019] Universal Transformers (https://arxiv.org/pdf/1807.03819.pdf)
• [ ] [ICLR2019] Pay Less Attention With Lightweight and Dynamic Convolutions (https://arxiv.org/pdf/1901.10430.pdf)
• [ ] Document MT: [EMNLP2018] Document-Level Neural Machine Translation with Hierarchical Attention Networks (https://www.aclweb.org/anthology/D18-1325)

Tokenization

• [ ] Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

Text classification

• [ ] Text-matching models (#616)
• [ ] Region Embedding (#437)
• [ ] Heirarchical Attention Models for Document (HAN)

Topic modeling

• [ ] LDA

Knowledge distillation

• [ ] [EMNLP2019] Patient Knowledge Distillation for BERT Model Compressions (https://arxiv.org/pdf/1908.09355.pdf)
• [ ] [ICLR2020 Submission] TinyBERT

APIs

• [ ] fp16 trainer API (#674)
• [x] BERT export API (#659, replaces static BERT scripts)
• [x] LAMB optimizer (#677)
• [x] GLUE datasets (#673)
• [x] Vocab special token registration (#572)
• [x] Make Vocab unknown token index configurable (#393)
• [ ] ROUGE metric (#561)
• [ ] subword-nmt BPE support (#374)
• [ ] vocab look-up with case-insensitive backoff (#367)
• [ ] configurable experiment management
• [ ] natural question dataset

Scripts

• [ ] ELMo feature extraction script (#595)
• [ ] Deep Biaffine Attention Dependency Parser follow-up (#427)

Documentation

• [x] Add documentation for return types of datasets and transform functions (#405)

Demos

• [ ] Demo application for NER
• [ ] Demo application for text classification
• [ ] [Demo application for question answering](https://demo.allennlp.org/reading-comprehension)
• [ ] Demo application for dependency parsing
• [ ] Demo application for machine translation
• [ ] Demo application for text generation

cc @dmlc/gluon-nlp-team

Related Projects

• QA: QANet + BiDAF
• NER: BiLSTM + CRF
• BERT

[haven-jeon]

633 MTL(MultiTask Learning) is one of the most important topic in NLP. Implementing BigBird is good start point to develop MTL in gluon-nlp.

I can make room for this topic.

[Ishitori]

Shall we consider LAMB optimizer - https://arxiv.org/abs/1904.00962?

@szha: #677

[szha]

657 BERT for XNLI/NMT. @fhieber and team expressed interest in this.

[kenjewu]

Up to now, gluonnlp has basically perfect components. There are many specific models for the specific tasks of natural language processing. Can we write some standard classic models? Anyone with a related task requirement can call this model directly for a quick experiment.

[szhengac]

Distributed training module may be a good feature as many current SOTA nlp models typically require a lot of GPUs. pytorch/fairseq also supports distributed training across multiple machines.

[fierceX]

I think we can add some text matching models #616

[sravanbabuiitm]

1. We can look towards adding HAN (Heirarchical Attention Models for Document classification : https://www.cs.cmu.edu/~./hovy/papers/16HLT-hierarchical-attention-networks.pdf) in scripts for text classification.
2. We can add HAN model and fastText models to the d2l.ai book

[markusdr]

JSON/Yaml config files that specify models and experiments would be great. That's a very popular feature of AllenNLP and other toolkits.

https://github.com/dmlc/gluon-nlp/issues/392

For an example AllenNLP config file, see https://github.com/allenai/allennlp/blob/master/tutorials/tagger/experiment.jsonnet

See also their tutorial:
https://github.com/allenai/allennlp/blob/master/tutorials/tagger/README.md#using-config-files

Quote:

This means that most of your experiment can be specified declaratively in a separate configuration file, which serves as a record of exactly what experiments you ran with which parameters.
Now you can change various aspects of your model without writing any code. For instance, if you wanted to use a GRU instead of an LSTM, you'd just need to change the appropriate entry in the configuration file.

[eric-haibin-lin]

Add "Deep Relevance Matching Model" for relevance matching
Here is one implementation in keras: https://github.com/sebastian-hofstaetter/neural-ranking-drmm

[eric-haibin-lin]

Add Simple Recurrent Unit (SRU) https://arxiv.org/pdf/1709.02755.pdf

[eric-haibin-lin]

ERNIE: BERT with Knowledge pretraining: https://arxiv.org/pdf/1904.09223.pdf

[eric-haibin-lin]

Topic models like LDA
Lemmatization and Stemming for preprocessing
Expose sparse n gram representation or BOW representation for sentences/documents to user.

https://github.com/dmlc/gluon-nlp/issues/822

[eric-haibin-lin]

MASS: MASS: Masked Sequence to Sequence Pre-training for Language Generation
https://arxiv.org/pdf/1905.02450.pdf

[eric-haibin-lin]

Pay Less Attention With Lightweight and Dynamic Convolutions https://arxiv.org/pdf/1901.10430.pdf

[eric-haibin-lin]

Poincaré Word Embedding: https://arxiv.org/pdf/1705.08039.pdf

[eric-haibin-lin]

GPT-2 Training from Scratch

[eric-haibin-lin]

Transformer/BERT visualization:
[Arxiv2019] Visualizing and Measuring the Geometry of BERT
[ICML2017] Understanding Black-box Predictions via Influence Functions

[eric-haibin-lin]

Memory networks:
* [Nature2016] Hybrid computing using a neural network with dynamic external memory
* [NeurIPS2016] Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
* [Arxiv2019] Large Memory Layers with Product Keys (https://arxiv.org/pdf/1907.05242.pdf)

[sundeepteki]

• Transformer-XL - https://github.com/kimiyoung/transformer-xl
• RoBERTa - https://github.com/pytorch/fairseq/tree/master/examples/roberta
• XLM - Cross-lingual LM pretraining - https://github.com/facebookresearch/XLM/
• Multilingual pre-trained encoder (BERT et al.) trained on Alexa data from all languages that Alexa currently supports or plans to support by 2020

[eric-haibin-lin]

Adaptive softmax and adaptive embedding https://arxiv.org/pdf/1809.10853.pdf

[leezu]

Adaptive softmax and embedding is already part of scripts/language_model. Let's clarify the roadmap as: move support to main API.

[eric-haibin-lin]

pointer mechanism for attention https://github.com/dmlc/gluon-nlp/issues/951

[eric-haibin-lin]

ALBERT: https://github.com/google-research/google-research/tree/master/albert

[eric-haibin-lin]

XLM-Roberta: https://github.com/pytorch/fairseq/tree/master/examples/xlmr

[fierceX]

I can try tinybert.

[szha]

We are working on the numpy version of GluonNLP and will adjust the positioning of this package. We will consider the roadmap items here on related areas as well as more recent research.