Flair: Question about camparison using ELMo on Ontonotes.

[PengLU1101]

Hi Flalr team,

I found you get a new state of art on Ontonotes NER task, your result is 89.71.
Did you ever try to tune the hyperparameter by using ELMo emb instead of yours?
Since the previous best result didn't use it.

Thanks,
Peng

[alanakbik]

Hi @PengLU1101 good point - we've compared against ELMo in all the (English) tasks we used in our paper, i.e. CoNLL-2000, CoNLL-03 and PoS tagging but I don't think we did the same for Ontonotes.

If you're interested, I could do a training run with ELMo and report back the numbers - we have an ELMo embedding class implemented in Flair that we use for this. However, I would not do a full hyperparameter search myself until the next time we want to do a thorough evaluation of different approaches. If you are interested in doing this comparison, let me know and I can give you the ELMo embeddings class for Flair.

(I was actually thinking of adding this class to the official release of Flair so that people can easily compare ELMo and Flair - but our current implementation has an AllenNLP dependency which is huge and would make the Flair framework a lot less lean, so still kind of undecided on this.)

[PengLU1101]

Cool, thanks so much.
If you can run with ELMo and report the results, that would be great!!!
And I did some test on Ontonotes with ELMo, my model highly depends on the AllenNLP module as well.
My result on ontonotes was about 89.66, yet I didn't search full hyper-parameters space. I also wonder if the 89.71 is your best result.

[alanakbik]

Ok we will, probably beginning of next week! We did not do parameter sweeps, so 89.71 is what we got on a run with parameters that generally tend to work well (I think we used mini batch size of 32, initial learning rate of 0.1, 256 LSTM states and 1 layer, patience of 2 or 3).

[tmills]

It would be really convenient if you were to post the Elmo embeddings class somewhere (or email to me), as I am very interested in this kind of comparison for my own tasks.

[alanakbik]

Ok will do! Fixing the serialization issues (#174) is the current priority but we can prepare the ELMo class directly after.

[stefan-it]

An interesting comparison would be with the latest BERT release: https://github.com/google-research/bert#using-bert-to-extract-fixed-feature-vectors-like-elmo - but at the moment I haven't look into details about the BERT implementation.

[alanakbik]

Absolutely - ideally I'd like to have embeddings classes for ELMo and BERT in Flair, to better compare and mix/match embeddings.

[stefan-it]

This project sounds very promising and can load the TF model with pytorch: https://github.com/huggingface/pytorch-pretrained-BERT :)

[stefan-it]

@alanakbik will the elmo embedding layer be part of the upcoming 0.4 release?

[alanakbik]

We are currently discussing adding both ELMo and BERT as embedding layers to Flair - hopefully we manage by 0.4!

[alanakbik]

We added ELMOEmbeddings to the release-0.4 branch. See #260

[alanakbik]

We just released flair 0.4 with both ELMo and BERT embeddings. Now you can mix and match Flair, BERT, ELMo and classic word embeddings. Check the new tutorial on these embeddings for info on how to use them.

We'd be happy to hear your experiences how they compare :) you can report experiments in #308

[tmills]

Thanks Alan, I've been experimenting with this functionality in the 0.4
branch and it's excellent to use! On my data, in preliminary results, Bert
and Flair were within about a point, and Elmo was a few points worse. If I
write it up I'll post more formal results.

On Fri, Dec 21, 2018 at 4:17 AM Alan Akbik [e-mail] wrote:

We just released flair 0.4 with both ELMo and BERT embeddings. Now you can
mix and match Flair, BERT, ELMo and classic word embeddings. Check the new
tutorial on these embeddings
https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_4_ELMO_BERT_FLAIR_EMBEDDING.md
for info on how to use them.

We'd be happy to hear your experiences how they compare :) you can report
experiments in #308 https://github.com/zalandoresearch/flair/issues/308

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/zalandoresearch/flair/issues/173#issuecomment-449325962,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAN3DXsXZ_dqByPwr2blPvHqOrqW4-7aks5u7KcogaJpZM4X9eQM
.

[alanakbik]

@tmills this is great - really look forward to the results!