Flair: Add Albert embeddings?

[krzysztoffiok]

In a recent paper https://arxiv.org/pdf/1909.11942.pdf a small and said to be SOTA version of BERT-like model is proposed. Wouldn't it be nice to include this model in Flair?

[stefan-it]

Hi @krzysztoffsiuwa ,

please have a look at this issue: https://github.com/huggingface/transformers/issues/1370

Once the albert model is included in transformers, then we can support it in Flair (I'll add an embedding layer whenever it is fully supported)

I'm also very excited about the NER results for Albert 🤗

[krzysztoffiok]

Thanks for a very quick response, great so it's probably just a matter of time now.

[yahshibu]

Hello, huggingface/transformers has been updated to v2.2.0. It supports ALBERT. Thanks!

[krzysztoffiok]

So now we are only waiting for Flair support.

[stefan-it]

I'm current doing some experiments for NER with ALBERT and Flair. I'll post some results today + add the implementation :)

[stefan-it]

With the xxlarge (v1, uncased) ALBERT model I could achieve 92.03% on CoNLL-2003 for NER 😄

So it outperforms all other "classic" BERT models, see a comparison here.

PR is coming soon!

[djstrong]

@stefan-it So, I guess loading ALBERT model using class BertEmbeddings is wrong? There is no error.
I can test your PR ;)

[stefan-it]

ALBERT for token classification has been recently added to the Transformers library.

I'll look into it again, maybe it is ready until end of this week :)

[stefan-it]

ALBERT embeddings were already added with https://github.com/flairNLP/flair/pull/1333, so I'm closing here.

You just need to pass the albert model name to the BERTEmbeddings instance:

from flair.data import Sentence
from flair.embeddings import BertEmbeddings

embeddings = BertEmbeddings(bert_model_or_path="albert-base-v2")

sent = Sentence("Berlin and Munich are nice cities .")
embeddings.embed(sent)

for token in sent.tokens:
  print(token.embedding) 

[djstrong]

Thanks. I had problem related to recognizing type of model by path/name. The solution was to change name of directory to contain "albert" in it.