Spark-nlp: Support fastText as Word Embeddings in NerDLModel

[maziyarpanahi]

Hi,

This is a feature request which might be already possible. Facebook has introduced and extention to word2vec called fastText which has some very good Word Vectors. We can use these word vectors in training NerDLModel and other possible annotators requiring word embeddings.

It supports over 157 languages and most of their vectors from news and Wikipedia are way more recent than others (Google, Stanford).

https://fasttext.cc/docs/en/crawl-vectors.html

I have found a very nice statement about GloVe, word2vec, and fastText which I quote it here:

A key difference is Glove treats each word in corpus like an atomic entity and generates a vector for each word. In this sense Glove is very much like word2vec- both treat words as the smallest unit to train on.

Fasttext which is essentially an extension of word2vec model treats each word as composed of character n-grams. So the vector for a word is made of the sum of this character n grams. This difference enables fasttext to

Generate better word embeddings for rare words ( even if words are rare their character n-grams are still shared with other words - hence the embeddings can still be good)
Out of vocabulary words - they can construct the vector for a word from its character n-grams even if the word doesn't appear in training corpus. Both Glove and word2vec can't.
Source: https://www.quora.com/What-is-the-difference-between-fastText-and-GloVe?ch=2

[Tshimanga]

Use the .vec file that fasttext outputs. It has the same format as the glove...txt files.

Use option 2 for the format parameter in .setEmbeddingsSource()

[maziyarpanahi]

Thanks @Tshimanga, however, I have doubts about the mapping word embeddings from FastText to GloVe or Word2Vec since there is much more information inside FastText vectors.
Have you personally tested this yourself?

[Tshimanga]

This is how I have been using my own fasttext embeddings in spark nlp, and it works fine; but so far I haven't been too concerned with fully utilizing fasttext as more than a convenient cli for generating in-domain word vectors. You're almost certainly right in expecting it to be lossy. @maziyarpanahi

[albertoandreottiATgmail]

Hello,

I think the most benefit of this technique could be achieved if we implemented an annotator that can provide vectors for any input word, otherwise we're going to be limited to what we precomputed with fasttext.
So my desire would be an annotator that can separate a word in the relevant n-grams, compose the final embedding, and make it available for further processing in other annotators.
This makes more sense, isn't it?
What do you think @saifjsl , wouldn't this be a cool feature?

[saif-ellafi]

It sounds a similar feature to what we('ll) do on BERT, by using a custom tokenizer tokenize specific set of features, plus giving embeddings to entire sentences, wouldn'it?

Maybe I am adding confusion here, but the question to answer there is which annotators would benefit from such content?

[maziyarpanahi]

I’m happy that we support BERT. The reason for asking about fastText was that it is new and updated with support for many languages. So I don’t think we need to support fastText and as mentioned earlier users can still read fastText in word2vec format.

The only annotator that supports word embedding is NER_DL at the moment. I would love to have NER for multiple language especially with the high accuracy NER_DL produces.

Thank you for supporting BERT embedding :)

[maziyarpanahi]

I will be closing this feature request as I have successfully used wiki-news-300d-1M.vec, crawl-300d-2M.vec and cc.en.300.vec same as I used to use GloVe 300d.