Fasttext: Updating the model (online-learning)

[mino98]

Hi,
here a simple question and/or feature request: would it be possible to progressively update a model (learned via skipgram or cbow) by passing new text to fasttext?

That would be very useful for online-learning of a text stream.

Thanks.
ps: this may be similar to #279. I could not find other open issues for this topic.

[dcsan]

also interested in this, please let us know if you find a way to avoid retraining from scratch all the time. maybe a dupe of #312 ?

[mino98]

Issues #279 and #312 seem to be a duplicate of this.
Apparently, @elbamos solved it in #191 but is waiting permission to share the code.

[elbamos]

I’m unlikely to get that permission at this point :( it was not terribly hard to do though, really a days work once you’re familiar with the code.

On Nov 4, 2017, at 3:46 PM, Mino [e-mail] wrote:

Issues #279 and #312 seem to be a duplicate of this.
Apparently, @elbamos solved it in #191 but is waiting permission to share the code.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[mino98]

Thanks anyway @elbamos. Could you at least share some pointers?
Also, were you expanding the dictionary when continuing training with new text?

[loretoparisi]

@elbamos @mino98 this is why it does not make any sense to modify this library (that is provided with a license) and do not send a PR since the code is proprietary of a company and cannot be released/used due to an "another" license. In that case, better to not share the implementation at all (and I'm sorry for this of course)! Hopefully this feature will go on priority some day.

[mino98]

Sure @loretoparisi.
If anyone has good starting pointers to share, I might have a go myself.

[rajivgrover009]

did any one get around this?

[matanster]

Here's a good place to consult for deriving this enhancement:

→ loading an existing model
→ entry point into the training of the embedding(unsupervised training in the docs' lingo)
→ the c++ code ultimately called from there to kick off each epoc of training
→ the actual training is kicked off, per the training kind indicated for the run, here, where it goes on to diverge between supervised learning (the classification task), and the two flavors of unsupervised learning (word embedding generation) cbow and skipgram.
→ the gory math, which my similar use case requires tinkering, resides in and around here, by the way.

Give or take some detail and some unknowns, all it takes is adding an option of running the executable such that ― it loads an existing model and then carries on with learning a word embedding.

That said, online learning is not guaranteed to work by just iterating an existing offline-trained model with the same optimizer and/or same hyperparameters.

[lucky050619]

./fasttext supervised -input train.txt -output model -inputModel model.bin -incr

[loretoparisi]

@balaram-bhukya that is amazing from this PR!!! https://github.com/facebookresearch/fastText/pull/423
Can't wait to see it working.

[lucky050619]

@loretoparisi Me too. anyone tried this in python ??? any luck ??

[loretoparisi]

@balaram-bhukya not sure, also there are some issues right now in the PR on the dictionary size...

[alucard001]

./fasttext supervised -input train.txt -output model -inputModel model.bin -incr

May I ask how is it going? Is it worked? I read the full documentation and I did not see -incr there.

Thank you.