Gensim: word2vec_inner compile fail due to import scipy.linalg.blas with scipy 0.17

[ylhsieh]

I notice that when compiling word2vec_inner.pyx it fails at from scipy.linalg.blas import fblas. Looking at scipy's documents I see that this is deprecated since scipy 0.12. Recently I updated my scipy and maybe that caused this error. I changed it to import scipy.linalg.blas as fblas to get it to work without modifying other codes. Did anyone encounter this problem?

[gojomo]

My working environments are at scipy 0.15.1, and I hadn't noticed any such warning (or failing error), either via the automated install or my manual invocations of cython-compilation.

How are you compiling that file?

[ylhsieh]

Hi, I put word2vec.py and word2vec_inner.pyx in the same folder, and use the following (modified some imports to make it easier to copy and test):

logger.info("Building word2vec_inner")
import pyximport, os
from numpy import get_include
models_dir = os.getcwd()
pyximport.install(setup_args={"include_dirs": [models_dir, get_include()]})
from word2vec_inner import train_sentence_sg, train_sentence_cbow, FAST_VERSION
logger.info("Built and imported word2vec_inner")

It results in
ImportError: Building module word2vec_inner failed: ['ImportError: cannot import name fblas\n']
Perhaps the reason roots from the 0.17 version of scipy? That they officially took out the fblas. If I have the time, I would try to downgrade scipy to 0.15 and try again. Thanks.

[ylhsieh]

I am deeply sorry to clicked the wrong button! Reopening this.

[piskvorky]

Yes, I've seen that deprecation, but forgot to investigate. Same with scipy.sparsetools, which was deprecated for unknown reasons, but on which gensim relies.

@ylhsieh can you investigate? If the fix is as simple as deleting f from fblas, then let's do it :)

[ylhsieh]

Will do. But I suggest to add some version checking in the .pyx files instead of changing it altogether, as some might still be using scipy older than 0.12. For me, changing from from scipy.linalg.blas import fblas to import scipy.linalg.blas as fblas works, and the rest of the .pyx file can remain the same. This method works for both word2vec_inner.pyx and doc2vec_inner.pyx.

[piskvorky]

Yes, can you check on scipy 0.10.1 please?

That's the latest we have to support (=version in old Debian stable); we can drop the older versions. Right now, gensim says it supports any scipy >= 0.7, but I don't think that's necessary anymore, >=0.10.1 will do.

[piskvorky]

Alternatively, we could hack .travis.yml to test against both the oldest version supported + the newest version in pip, to automate this.

[ylhsieh]

Confirmed using scipy 0.10.1 and 0.15.1, the command from scipy.linalg.blas import fblas passed. I did not install 0.16, and my scipy 0.17.1 was pulled from scipy github. So I guess we can close this issue and leave it as a future reference until they make official release of scipy 0.17. Or, should we preemptively add version detection now?

[piskvorky]

Yes, fblas works with older scipys.

The question is, does from scipy.linalg.blas import blas work too?

Is it safe to just always import scipy.linalg.blas in scipy >= 0.10.1? Or do we have to try / catch ImportError on fblas... or dynamically check scipy.version to see whether to import blas or fblas.

[ylhsieh]

Just tested in scipy 0.15.1, from scipy.linalg.blas import blas does not work.

[piskvorky]

Ah, I see. Thanks for checking @ylhsieh !

Can you add the ImportError try-catch workaround, in a pull request?

That is, if the current from scipy.linalg.blas import fblas fails, do the new import instead. And check that the workaround works both for old/new scipy :) Cheers!

[ylhsieh]

I would like to learn how to do that (pull request), but I am a noob to git and not familiar with the PR thingy... Perhaps it would be safer for others to try that? I am afraid I might do some silly stuff (like closing this issue that happened in this thread :weary: )

[piskvorky]

GitHub have a tutorial on pull requests.

It's not that complicated and maybe good practice and a good skill to have :)

And don't worry about "doing silly stuff" -- you don't have write permissions into the "real gensim", so there's no damage you can do. Experiment away!

[xarg]

I tried aliasing from scipy.linalg.blas import blas as fblas (did the Cython stuff and recompiled) but the import still doesn't work:

from gensim.models.word2vec_inner import train_sentence_sg, train_sentence_cbow, FAST_VERSION
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

I don't have enough experience with Cython to know why it's not working.

The workaround that works for me is to downgrade scipy==0.15.1

[piskvorky]

I've fixed scipy version at 0.15.1 for the Travis tests for now (the automated Travis tests started failing, because of the same issue -- fblas removed).

Just a reminder to self to revert commit 1cb6919 once this issue is resolved.

[ylhsieh]

Finally finished creating a PR. I hope I did it right to try to fix this issue.
My first PR! (hooray ??)

[piskvorky]

Lemme check :)

[piskvorky]

Fixed in #415 . Thanks @ylhsieh !

[joelthelion]

You might want to make a new release even if solely for that issue, since currently "pip install gensim" results in a broken installation and is not very welcoming for newcomers.

[piskvorky]

Yeah, will release later today.

[joelthelion]

Cool, thanks!