Flair: Add support for XLNet

[stefan-it]

Hi,

XLNet is a new pre-training method for NLP, that adopts some ideas from the Transformer-XL architecture and was published today:

XLNet: Generalized Autoregressive Pretraining for Language Understanding.

I just read in the pytorch-pretrained-BERT repo, that they're planning to integrate it 😍

So whenever an implementation is available there, I would like to add a new embedding class XLNetEmbeddings so that it can be used in flair for downstream tasks.

[Hellisotherpeople]

I am extremely interested in this!

[vprelovac]

+1 for this

[alanakbik]

Yes that would be awesome!

[saj1919]

Huggingface just released this -> https://github.com/huggingface/pytorch-transformers.
Would like to see the implementation here as well .. hope the development is on track !!

[MarcioPorto]

Should this be closed since https://github.com/zalandoresearch/flair/pull/941 has been merged?

[alanakbik]

Yes, good point! :)

[abeermohamed1]

Hi @stefan-it and @alanakbik
Could you please advise how to use this for Arabic language embeddings?

[alanakbik]

I am not sure if there are XLNet embeddings for Arabic, check here: https://huggingface.co/models?search=xlnet

But you can use any Arabic BERT model from this list: https://huggingface.co/models?search=arabic

You can load any model you find here with the TransformerWordEmbeddings or TransformerDocumentEmbeddings classes, depending on whether you want to embed words or documents:

    embeddings = TransformerWordEmbeddings(
        model=' asafaya/bert-base-arabic',
    )

[abeermohamed1]

Thank yiu

I am not sure if there are XLNet embeddings for Arabic, check here: https://huggingface.co/models?search=xlnet

But you can use any Arabic BERT model from this list: https://huggingface.co/models?search=arabic

You can load any model you find here with the TransformerWordEmbeddings or TransformerDocumentEmbeddings classes, depending on whether you want to embed words or documents:

    embeddings = TransformerWordEmbeddings(
        model=' asafaya/bert-base-arabic',
    )

Thank you @alanakbik unfortunately there is no pretrained Arabic XLNET model on huggingface. I already tried Arabic BERT model since while