Gluon-nlp: [Script] Add ELMo feature extraction script

[eric-haibin-lin]

The model zoo section on gluon-nlp website does not show ELMo as one of the model. The main reason is that we do not have script for ELMo, but a tutorial. This leads to misunderstanding that gluon-nlp does not have pre-trained ELMo model. I suggest we add an entry to ELMo in the model zoo section and provide a script like #587

[imgarylai]

@eric-haibin-lin is anyone working on this issue? I can take care of it.

[szha]

@imgarylai Not at the moment. Let us know if you need any help with it!

[imgarylai]

I'm writing the script for this issue. I want to say thank you for the tutorial first. The tutorial is a good reference for me. The script is almost done. Several details I want to confirm with you to make sure I understand it correctly.

In the last step of this tutorial, it stops at showing the shape of features of the first batch, including two sentences. And for each feature, the shape represents batch_size, sequence_length, embedding_size. However, there are three features. These three features are not explained in the tutorial. I look it up from the source code. Do these three represent mask, forward, backward respectively? This is my first question.

If my previous understanding is correct, I have the following questions. Assume there is a sentence in a length of 5 and it is the first sentence in the first batch for simplicity. I want to get the features from both forward and backward layers of this sentence.

The idea should be like the following code, right?

batch = next(iter(data_loader))
# get the first batch

features = get_features(*batch)
# get features for first batch

_, forward, barkward = features

forward_feature = forward[0][0][:5]
barkward_feature = barkward[0][0][:5]
# The first index is the index in batch and the second one is the index of the sentence in that batch.

And according to my assumption, since the length of the sentence is 5, I should just get the first five embedding from the forward. And for the barkward_feature, is it same as like forward? My question is we still use the first five embeddings not the last 5, right? (I check the code. It should have been reversed already.)

Thanks for your patient to read these long issue! Please let me know if I misunderstand in any part of this tutorial.

[szha]

Hi Gary.

These three features are not explained in the tutorial.

They represent the activation of each of the three layers. See ELMoBiLM.
The first dimension of each extracted feature ndarray has size 2 which represents the features from forward and backward directions.

Also, note that there's not enough validation on this functionality so far, so it would be great to at least use this to reproduce one downstream task once done. https://github.com/dmlc/gluon-nlp/issues/523

[szha]

I think the above points are worth mentioning in the tutorial if you'd help with that too :)

[imgarylai]

@szha Thanks. I think I understand it. The result represents the following representations right?

[imgarylai]

And yes, sure. Of course! I will update the tutorial later.

[szha]

They represent the output of the three bi lstm layers

[imgarylai]

@szha I read the paper again. I realized I left stupid comments... sorry for that.
I'm also interested in reproducing downstream tasks. I can start from there.

[eric-haibin-lin]

Hi @imgarylai is this still on your roadmap?