Fasttext: Python version: import of models trained with command line version fails

[pstroe]

Hello,

it seems that models trained with the command line interface and models trained with the Python package of fastText are not compatible. It is not possible to load a model trained with the command line version in Python:

`

In [2]: m = fasttext.load_model('models/fasttext_models/model-60-mincount5-wordngrams2-sent-stop-strat.bin')

RuntimeError Traceback (most recent call last)
fasttext/fasttext.pyx in fasttext.fasttext.load_model (fasttext/fasttext.cpp:4234)()

RuntimeError: vector::_M_default_append

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
in ()
----> 1 m = fasttext.load_model('models/fasttext_models/model-60-mincount5-wordngrams2-sent-stop-strat.bin')

fasttext/fasttext.pyx in fasttext.fasttext.load_model (fasttext/fasttext.cpp:4292)()

Exception: fastText: Cannot load models/fasttext_models/model-60-mincount5-wordngrams2-sent-stop-strat.bin due to C++ extension failed to allocate the memory
`

And vice versa.

Or am I missing anything?

Cheers,

Phillip

[EdouardGrave]

Hi @pstroe,

Thank you for reporting this issue.

What is the size of the model you are trying to load? Did you use the same version of fastText to build the command line tool and the python bindings?

Best,
Edouard

[shm007g]

Same problem here. Can anyone give me any advice? @EdouardGrave

Env: ElemantaryOS( Ubuntu16.04 ), install with 'pip install fasttext' i guess

version:
pip freeze | grep fasttext
fasttext==0.8.3

model size:

-rwxrwxrwx 1 root root 362M 7月  31 18:03 sina.567.bin
-rwxrwxrwx 1 root root 672M 7月  31 18:03 sina.567.vec

error message:

Traceback (most recent call last):
  File "fasttext/fasttext.pyx", line 152, in fasttext.fasttext.load_model
RuntimeError: vector::_M_default_append

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***/sina_test.py", line 6, in <module>
    classifier = fasttext.load_model(os.path.join(test_dir, 'sina_0604.model.bin'), label_prefix='__label__')
  File "fasttext/fasttext.pyx", line 154, in fasttext.fasttext.load_model
Exception: fastText: Cannot load ***models/sina_0604.model.bin due to C++ extension failed to allocate the memory

[shm007g]

I reinstall the fasttext linux package and python package from source.
It turns out pip package get the version of 0.8.22, which is newer than fasttext 0.8.3 I install through pip install.

And I search the fasttext 0.8.3 release history and i found it released on Feb 2017. And last stable version of fasttext linux package is v0.1.0, which is released on Dec 2017.

So this is a version incompatible problems.

[bact]

Do stick with the fastText Python binding from Facebook, and not the PyPI one (that pip install use).

PyPi (pip) one uses import fasttext (all small letters),
Facebook's one uses import fastText (Text with capital T).

They look quite similar but with small differences and incompatibilities. For example, fastText.predict() output also contains probability of label (similar to fasttext.predict_proba()).

You must build and install fastText (C++) by yourself first, then install Python binding from there by pip install ..

See https://github.com/facebookresearch/fastText/tree/master/python

[bact]

This is actually very confusing anyway, since in the fasttext PyPI page at https://pypi.org/project/fasttext/ has a link to project homepage at https://github.com/pyk/fastText.py which has a text said

The fasttext pypi (link: https://pypi.org/project/fasttext/) is now maintained by Facebook AI Research team. Read the documentation here: fastText python binding. (link: https://github.com/facebookresearch/fastText/tree/master/python)

With this description, people may expect that fasttext package in PyPI (which is "now maintained by Facebook AI Research team") will maintain some compatibility with the current C++ version, which it doesn't :(

[shm007g]

I had found fastText and fasttext, and overcome it. Thx all the way. @bact

[bact]

To add another layer of confusion, now (26 Nov 2018):

• if one pip install . the fastText from Facebook's repo, the version reported (from setup.py) will be fasttext-0.8.22 (all small caps)
• but if pip install fasttext, the version reported will be fasttext-0.8.3 (all small caps, as well)

this also gives an impression that these two packages are somehow an continuation of the same code base (which I don't think currently the case) and the one from PyPI is newer and more preferable (since it is also "now maintained by Facebook AI Research team").

[SpellOnYou]

I had found fastText and fasttext, and overcome it. Thx all the way. @bact

Me too... you solved my problem!

[Celebio]

Hi!
This issue is solved in our latest release. You can read our blog article to learn more about this here.

I will close the issue. Feel free to re-open it if you need more clarification.

Regards,
Onur

[chameleonTK]

Hi, it seems that this issue is not completely solved.

I was trying to run the code below.

import fasttext
m1 = fasttext.load_model("wv/subword_th.bin")
m2 = fasttext.load_model("wv/subword_en.bin")

Both of them are built with the same function but different dataset. I found that I cannot load subword_en.bin. It returns ValueError: vector::_M_default_append

but it works fine for subword_th.bin.

No idea what is going on ToT

subword_th.bin
https://drive.google.com/file/d/1V8-BIlijfdC5DkXB4VQQVL-1Vytu3oDL/view?usp=sharing

subword_en.bin
https://drive.google.com/file/d/1Uw39ZDbJYLu99mmITJEYwNhH_wwRCJQM/view?usp=sharing

Note: I tried fasttext earlier version but it didn't work as well.