Gluon-nlp: [API] [BERT] Add option to return intermediate results at each layer

[eric-haibin-lin]

Currently the BERTModel API only returns the encoding of the last layer. We should support returning intermediate encoding at each layer like ELMo. To do this, we can just add an extra argument to BERTModel to configure the model.
https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/model/transformer.py#L480-L484

[eric-haibin-lin]

@Ishitori just curious if you have started this task? @kenjewu is working on a segmentation task and also wants this option. Just to check here to avoid duplicating work

[Ishitori]

@eric-haibin-lin, yep, I am working on that

[eric-haibin-lin]

The PR is merged. Thanks @Ishitori and @kenjewu

[nahidalam]

@eric-haibin-lin I see this PR is merged. Is there any example of how to use it?

[eric-haibin-lin]

There is not specific example just for this. You can just set the output_attention or output_all_encodings to be True when you use the gluonnlp.model.get_bert_model API, and additional tensors will be returned as outputs.

If you have any good example/tutorial in mind to show the usage, contribution is welcome! :)

[eric-haibin-lin]

@Ishitori I remember you were working on a tutorial to visualize the intermediate attention weights for BERT, is that already done? Any plan to contribute it back to gluonnlp?

[Ishitori]

Unfortunately, I haven't finished working on that due to lack of time. As soon as it is ready, I will contribute it back in a form of a tutorial.