Fasttext: Learn words representation with CBOW plus position-weights

[loretoparisi]

As for the recent paper "Learning Word Vectors for 157 Languages", the model CBOW is used with position dependent weights. Using that the new pre-trained model were produced.
Is it possible to train a unsupervised model with CBOW in this version of fastText using the same approach with position weights?

[rilut]

Also, which command in fasttext corresponds to the positional-weighting?

[matanster]

I don't think this is implemented in the publicly available codebase.

[loretoparisi]

@matanster you are right, it is not, at least not yet, while the most two recent papers show models that has been trained FastText with position dependent weights.

[ghost]

Do you know if there are any updates of positional-weighting in fasttext base code?

[loretoparisi]

@omriFdna so far I'm not aware of a version that implements the positional weights bow, but I will have a look maybe someone did...

[ghost]

@loretoparisi any luck?

[loretoparisi]

@omriFdna not yet so far :

[loretoparisi]

A dataset with position weights trained with CBOW has been recently released for Wikipedia and Common Crawl - https://fasttext.cc/docs/en/crawl-vectors.html

The model parameters were:

|DIM|NGRAM|WS|NEG
|---|---|---|---|
|300|5|5|10|

It would be great to have this as training option as well.

[ghost]

I agree, they also report in their papers that position weights improves the performance, I wish it was part of the training options.

[adam2326]

+1

On Thu, Oct 11, 2018 at 2:58 AM Omri [e-mail] wrote:

I agree, they also report in their papers that position weights improves
the performance, I wish it was part of the training options.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/fastText/issues/445#issuecomment-428842667,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHUWqNswFJkmbxOlBP3L8vJjmMogNTYlks5ujuv9gaJpZM4STDKV
.

[Witiko]

I am attempting to add this feature to Gensim, see https://github.com/RaRe-Technologies/gensim/pull/2905.