Flair: Embeddings dimension for translation

[valentinmace]

I want to use Flair Embeddings in my NMT experiments

Following your tutorials, you recommend to use Stacked Embeddings fromnews-forward, news-backward and standard word embeddings like glove

It sounds promising but such Stacked Embeddings have a dimension of 4196 which is big for a Transformer model

Do you have any recommendations for that ? I will certainly try multiple approaches but any advice would be appreciated

Thanks a lot

[stefan-it]

Hi @valentinmace,

I think you could use the news-forward-fast and news-backward-fast Flair Embeddings. These embeddings have the half size (of dimensions) compared to the "normal" news-forward embeddings. These would reduce the dimensionality :)

[valentinmace]

Hi @stefan-it

As you suggested I will start using smaller embeddings like news-forward-fast and then try with bigger ones, possibly stacked

Do you have any preference regarding where I should post my results ?

[alanakbik]

Hi @valentinmace we'd be very interested to hear how embeddings / stacked embedding influence NMT, so if you could post your results here (in this thread) that would be great!

[Icyzrw]

I've tried to change the size of the embedding2nn layer from embedding_lengthembedding_length to embedding_length(embedding_length/2), and the result didn't change much in NER task.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.