Allennlp: Train ELMO for a new domain

[jeniyat]

1) What is the easiest way to train and get ELMO embedding for a new dataset? I am trying to implement NER on a new domain. Is there any tutorial which can help me to train a new embedding? What are the files that I need to update?

2) If the data is saved in conll format (WORD POS-TAG CHUNK-TAG NER-TAG), which is the best file to work with? I am thinking of using: https://github.com/allenai/allennlp/blob/master/allennlp/data/dataset_readers/conll2003.py and changing file_path in line 102 for reading data.

3) How to call the NER model?

4) If I want to check the NER performance of NER on CONLL data, in 3 cases
a) only character embedding, b)only context embedding c) only word embedding. What are the functions/classes I need to change/use for this task?

[qiuwei]

I am not the devs of allennlp, but had some experience with it.

1. Use the tensorflow implementation. Just follow the instructions there. https://github.com/allenai/bilm-tf
2. You could prepare the data in conll03 format, since allennlp already has a datareader for it. Or you can implement your own datareader. It's pretty straightforward as well.
3. Train, evaluate, test in standard Allennp way, just prepare a json file which describes your setup. You can find example config files such as https://github.com/allenai/allennlp/blob/master/training_config/ner_elmo.jsonnet
4. You can tweak the config file mentioned above to do all of the 3 experiments without changing any function/classes. That's the beauty of allennlp.

[jeniyat]

Thanks for the response!
4) Can you please specify which parameter to change in https://github.com/allenai/allennlp/blob/master/training_config/ner_elmo.jsonnet to get 3 embeddings: a) only character embedding, b)only context embedding c) only word embedding.

[matt-peters]

You can use the scalar_mix_parameters (see https://github.com/allenai/allennlp/pull/1992/) to effectively select just a single layer from the biLM. The model doesn't have separate character and word embeddings, it only has character CNNs + highway layers to form its context insensitive word embedding.

[jeniyat]

4) I found the answer in the "Text Field Embedder" section of the configuration tutorial:
So these 3 experiments can be done by removing the keys from the "token_embedders" of the "text_field_embedder": in the ner_elmo.jsonnet
Please let me know, if this is the correct interpretation of the config.josnet

[schmmd]

Just a heads up--we're adding code to train ELMo models from AllenNLP directly and plan to have something by the end of the year!