Gluon-nlp: Add pre-trained source cc.en.300 for token embedding fasttext

[kurtjanssensai]

FaceBook Research recently released the English source files.
https://github.com/facebookresearch/fastText/commit/5bf8b4c615b6308d76ad39a5a50fa6c4174113ea
Could those be added to Gluon-NLP as well?

[szha]

Yes, I can add the English vectors and crawl-300d-2M-subword.zip

[kurtjanssensai]

Thank you very much! That will be greatly appreciated.
Also, at the moment it's not possible to use _wiki-news-300d-1M-subword.vec.zip_ as source with _load_ngrams=True_
Do I understand it correctly that this should be possible since it has been trained with subword information?

[sravanbabuiitm]

on english, i have observed wiki.en.vec to be much superior in performance over the common crawl embeddings, for other languages that arent data rich, the common crawl ones are doing good.

https://arxiv.org/pdf/1802.06893.pdf

Inline with what the common-crawl embeddings paper suggests

[leezu]

@KurtJanssensCRW unfortunately no. While those vectors were supposedly trained with subword information, the subword part of the embeddings was not published.

If you find the subword part (included in the .bin file), we'd be happy to add it.

[leezu]

@szha could you copy the following files to the GluonNLP bucket

https://lllausen-data.s3.amazonaws.com/gluonnlp/crawl-300d-2M-subword-e6b07293f.bin
https://lllausen-data.s3.amazonaws.com/gluonnlp/crawl-300d-2M-subword-927782c8e.npz

Thanks!

[szha]

@leezu yes, done

[leezu]

@KurtJanssensCRW unlike wiki-news-300d-1M-subword.vec.zip, for crawl-300d-2M-subword the ngram vectors were published and can be used via load_ngrams=True.

[kurtjanssensai]

Thank you for your effort!
I have submitted an issue to fastText to add the bin file for the English wikipedia embedding (https://github.com/facebookresearch/fastText/issues/656)

I noticed that you added crawl-300d-2M-subword but not cc.en.300 in https://github.com/dmlc/gluon-nlp/pull/336
Would it be possible to add these as well or are they inferior?

Many thanks,
Kurt

[kurtjanssensai]

FacebookResearch confirmed that the bin file for English wikipedia embeddings is avalable:
https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki-news-300d-1M-subword.bin.zip
https://github.com/facebookresearch/fastText/issues/656

[kurtjanssensai]

Would it be possible to add the bin files for the multi-language embeddings as well?
https://fasttext.cc/docs/en/crawl-vectors.html
Many thanks!

[leezu]

@KurtJanssensCRW https://github.com/dmlc/gluon-nlp/pull/368 adds the wiki-news-300d-1M-subword.bin

To clarify, the multi-language embeddings you requested 4 days ago should already be part of GluonNLP since July. Do you have any problems with the existing embeddings? I assumed facebookresearch may have updated their vectors since then, but it seems that we already include the most recent version.

[szha]

@leezu it’s worth thinking about how to better describe our embedding offerings in the new model zoo page

[kurtjanssensai]

@leezu @szha
The multi-language embeddings are indeed part of GluonNLP.
But the subword information included in the bin-files is not present.
This would allow me to use these embedding with load_ngrams=True
Thanks for taking a look at this!

[leezu]

Thanks @KurtJanssensCRW , in principle GluonNLP already contains the bin-files. For example, consider the vectors for German:

In [2]: nlp.embedding.create('fasttext', source='cc.de.300', load_ngrams=True)
Embedding file cc.de.300-fc6e4385.bin is not found. Downloading from Gluon Repository. This may take some time.
Downloading /home/leonard/.mxnet/embedding/fasttext/cc.de.300-fc6e4385.bin from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/embeddings/fasttext/cc.de.300-fc6e4385.bin...

However, it turns out that the vectors for English language were stored in a different folder (in fasttext-vectors instead of fasttext-vectors/word-vectors-v2/) in the fastText S3 bucket and were consequently not uploaded to the GluonNLP S3 bucket. Were you experiencing issues with English? Or which vectors did you have issues with?

I will upload the vectors for English too now

[leezu]

@KurtJanssensCRW the cc.en vectors are added now. Could you verify that everything works on your side? Sorry for the confusion about the vectors, I was initially not aware that the English vectors were skipped due to above path issue. Thanks for staying on this and pointing out the issues!
I will close the issue for now, but if there is any other problem feel free to re-open.