Faiss: Faiss import error when run in virtualenv by using own built Faiss-python

[billyean]

Summary

I have built faiss-core and faiss-python by myself. I installed python into my local virtual env and try to import faiss and I got an error, checked egg file, it does have _swigfaiss.so inside. I checked conda swigfaiss.py, it's still using old swig_import_helper, not sure if caused by this you remove it by using swig create python/swigfaiss.py as follows:

https://github.com/facebookresearch/faiss/commit/7f5b22b0fff0882ce4afd93ce54cc2833a224909#diff-8cf6167d58ce775a08acafcfe6f40966

$ ls faiss-1.5.2-py3.6/faiss
__init__.py __pycache__ _swigfaiss.so   swigfaiss.py

Platform

OS: centos 7

Faiss version: 1.5.2

Faiss compilation options:

 ./configure  --prefix=/usr --without-cuda --with-blas=/usr/lib64/libblas.so.3 --with-lapack=/usr/lib64/liblapack.so.3
make
sudo make install
make py
cd ~ && rm -rf env && python3 -m venv env
source env/bin/activate
cd ~/faiss && sudo make -C python install

Running on:

• [X] CPU
• [ ] GPU

Interface:

• [ ] C++
• [X] Python

Reproduction instructions

$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38)  [GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/__init__.py", line 18, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/swigfaiss.py", line 13, in <module>
ImportError: cannot import name '_swigfaiss'

[beauby]

Could you paste the output of import faiss._swigfaiss?

[billyean]

>>> import faiss._swigfaiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/__init__.py", line 18, in <module>
  File "/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/swigfaiss.py", line 13, in <module>
ImportError: cannot import name '_swigfaiss'

[beauby]

And the output of PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"?

[billyean]

I guess this might be related to egg file, when I do

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_swigfaiss'

after I unzip the egg file, I don't see error as below

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6/faiss/ python -c "import _swigfaiss"

[HoweChen]

Same thing happened to me.

[xxllp]

这个问题好麻烦

[WIll-Xu35]

Same problem here.

[beauby]

@HoweChen @WIll-Xu35 Please paste your generated makefile.inc.

[WIll-Xu35]

@beauby I met this problem when using pytorch docker environment, which has conda environment and it was really a mess. I gave up on that environment and installed faiss without docker or conda. The installation was successful.

[XinliYu]

Same thing here. Compiled from source with anaconda virtual env. My makefile.inc is as below.

Copyright (c) Facebook, Inc. and its affiliates.

#

This source code is licensed under the MIT license found in the

LICENSE file in the root directory of this source tree.

CXX = g++ -std=c++11
CXXCPP = g++ -std=c++11 -E
CPPFLAGS = -DFINTEGER=int -fopenmp -I/usr/local/cuda/include
CXXFLAGS = -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra
CPUFLAGS = -mpopcnt -mavx2 -mf16c
LDFLAGS = -fopenmp -L/usr/local/cuda/lib64
LIBS = -lopenblas -lcudart -lcublas
PYTHONCFLAGS = -I/home/ec2-user/anaconda3/envs/pytorch_p36/include/python3.6m -I/home/ec2-user/anaconda3/envs/pytorch_p36/include/python3.6m -I/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/numpy/core/include
SWIGFLAGS = -DSWIGWORDSIZE64

NVCC = /usr/local/cuda/bin/nvcc
CUDA_ROOT = /usr/local/cuda
CUDA_ARCH = -gencode=arch=compute_35,code=compute_35 \
-gencode=arch=compute_52,code=compute_52 \
-gencode=arch=compute_60,code=compute_60 \
-gencode=arch=compute_61,code=compute_61 \
-gencode=arch=compute_70,code=compute_70
NVCCFLAGS = -I $(CUDA_ROOT)/targets/x86_64-linux/include/ \
-Xcompiler -fPIC \
-Xcudafe --diag_suppress=unrecognized_attribute \
$(CUDA_ARCH) \
-lineinfo \
-ccbin $(CXX) -DFAISS_USE_FLOAT16

OS = $(shell uname -s)

SHAREDEXT = so
SHAREDFLAGS = -shared

ifeq ($(OS),Darwin)
SHAREDEXT = dylib
SHAREDFLAGS = -dynamiclib -undefined dynamic_lookup
SWIGFLAGS =
endif

MKDIR_P = /bin/mkdir -p
PYTHON = python
SWIG = swig
AR ?= ar

prefix ?= /home/ec2-user/faiss
exec_prefix ?= ${prefix}
libdir = ${exec_prefix}/lib
includedir = ${prefix}/include

[XinliYu]

Found the solution for me! Just unzip the egg file!
unzip faiss-1.5.3-py3.6.egg

And then a new folder called faiss will be created in the site-package folder. Then import is successful!

[mingmingfu]

I guess this might be related to egg file, when I do

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_swigfaiss'

after I unzip the egg file, I don't see error as below

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6/faiss/ python -c "import _swigfaiss"

where can I get the egg file? Could you share me the link.

[sara-eb]

@beauby I am facing the same issue:

Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/faiss-1.5.3-py3.5.egg/faiss/__init__.py", line 18, in <module>
  File "/usr/local/lib/python3.5/dist-packages/faiss-1.5.3-py3.5.egg/faiss/swigfaiss.py", line 13, in <module>
ImportError: cannot import name '_swigfaiss'

@XinliYu could you please explain more on what you did?

[nlp-study]

I guess this might be related to egg file, when I do

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6.egg/faiss/ python -c "import _swigfaiss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '_swigfaiss'

after I unzip the egg file, I don't see error as below

$  PYTHONPATH=/home/midas/env/lib/python3.6/site-packages/faiss-1.5.2-py3.6/faiss/ python -c "import _swigfaiss"

where can I get the egg file? Could you share me the link.

your python dist-packages, such as mine : /usr/local/lib/python3.5/dist-packages

[javedqadruddin]

exporting the path to _swigfaiss in the venv's activate script worked for me. Just add this to the end of your activate script:
export PYTHONPATH="${PYTHONPATH}:/path/to/your/env/lib/python3.6/site-packages/faiss/"

Then deactivate and activate again and it should work.

[mdouze]

No activity, closing.

[deimsdeutsch]

Solution:

unzip the .egg file

create the folder "faiss-1.6.1-py3.5.egg" or whatever your error while import python3 comes up.
mv faiss faiss-1.6.1-py3.5.egg

then check with import faiss

@mdouze
This is definately a bug the egg file is not extracting properly.

[sngjuk]

My case, the egg was already extracted. this one may resolve your error.

sudo apt-get install libopenblas-dev

[RaulGuo]

for me, it is numpy version problem. I have upgraded numpy version from 1.13.1 to 1.18.2 and it works. Be sure the old version numpy is totally removed, I have uninstalled 1.18.2 numpy and find that numpy is still available in ipython, then I found that 1.18.2 is not the only numpy version installed in my server.

[dvlshah]

Just install three things in the following order and your problem Is solved

pip3 install faiss
sudo apt-get install libopenblas-dev
sudo apt-get install libomp-dev

[gshguru]

If you have faiss-cpu. Just make sure that faiss is not installed. In my case, these 2 packages were conflicting.

I had installed faiss-cpu using: pip install faiss-cpu --no-cache. Then by mistake installed faiss too in the middle of development. I had to uninstall faiss to get it working