Faiss: make py: fatal error: Python.h: No such file or directory

[daisy-belle]

I am also facing same issue, i did following steps
1) Cloned FAISS
2) updated makefile.inc with anaconda python path and installed necessary dependencies like libopenblas-dev python-numpy python-dev
3) make (After this step i am not finding any _swigfaiss.so files anywhere)
4) make py (Gave following error)
$ make py
g++ -I. -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -g -fPIC -fopenmp -I~/anaconda2/envs/faissenv/include/python2.7/ -I~/anaconda2/envs/faissenv/lib/python2.7/site-packages/numpy/core/include -shared \
-o python/_swigfaiss.so python/swigfaiss_wrap.cxx libfaiss.a /usr/lib/libopenblas.so.0
python/swigfaiss_wrap.cxx:154:21: fatal error: Python.h: No such file or directory
compilation terminated.
Makefile:84: recipe for target 'python/_swigfaiss.so' failed
make: * [python/_swigfaiss.so] Error 1
I am able to run cpp implementation, but only this python wrapper is failing, let me know what i am setting wrong. As _swigfaiss.so is not generated, what went wrong while doing make?

_Originally posted by @Mahanteshambi in https://github.com/facebookresearch/faiss/issues/336#issuecomment-365565492_

[beauby]

updated makefile.inc with anaconda python path and installed necessary dependencies like libopenblas-dev python-numpy python-dev

Unless you have a specific reason, you should not do it by hand but let ./configure generate it.

make (After this step i am not finding any _swigfaiss.so files anywhere)

This is expected: python/_swigfaiss.so is generated by make py.

fatal error: Python.h: No such file or directory

This means the include path to python is wrong. You should try with absolute paths (without the ~).

[WIll-Xu35]

@beauby Hi, I've followed install.md to install faiss and met the same error: 'Python.h: No such file or directory'. The commands I have entered is:

1. ./configure --with-cuda=/usr/local/cuda-9.0 --with-python=/usr/bin/python3.5 --with-cuda-arch="-gencode=arch=compute_61,code=sm_61"

2. make

3. sudo make install

Till this step everything finished without error. But when I run 'make -C python', this error occurred:

Do you have any idea what might be the problem?

Thanks.

[beauby]

@WIll-Xu35 This usually means your python is not reporting the right paths. Could you:

1. $ python -c "import sysconfig; print(sysconfig.get_path('include'))"
2. make sure the reported path(s) exist and do contain Python.h

[WIll-Xu35]

@beauby Thank you for your help. I've tried to locate the correct path of python3.5m, which is /usr/include/python3.5m in my case. I changed the --with-python parameter and run the whole process again, the same error occurred again.

[beauby]

@WIll-Xu35 Could you make sure that the output of python -c "import sysconfig; print(sysconfig.get_path('include'))" is an existing path?

[WIll-Xu35]

@beauby I've verified that the path exists and there is a Python.h file under that directory.

[beauby]

Could you post your makefile.inc?

[WIll-Xu35]

@beauby

https://drive.google.com/open?id=1ibBBHfNEhY2_qugAnCnnkh_argM6AKZt

This is the link to my generated makefile.inc. Thank you for your help:)

[beauby]

@WIll-Xu35 Could you paste it directly here please?

[WIll-Xu35]

@beauby Sorry for the late reply. The makefile.inc looks like this:

CXX          = g++ -std=c++11
CXXCPP       = g++ -std=c++11 -E
CPPFLAGS     = -DFINTEGER=int  -fopenmp -I/usr/local/cuda-9.0/include 
CXXFLAGS     = -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra
CPUFLAGS     = -msse4 -mpopcnt
LDFLAGS      = -fopenmp  -L/usr/local/cuda-9.0/lib64 
LIBS         = -lopenblas   -lcudart -lcublas 
PYTHONCFLAGS =   -I

NVCC         = /usr/local/cuda-9.0/bin/nvcc
CUDA_ROOT    = /usr/local/cuda-9.0
CUDA_ARCH    = -gencode=arch=compute_61,code=sm_61
NVCCFLAGS    = -I $(CUDA_ROOT)/targets/x86_64-linux/include/ \
-Xcompiler -fPIC \
-Xcudafe --diag_suppress=unrecognized_attribute \
$(CUDA_ARCH) \
-lineinfo \
-ccbin $(CXX) -DFAISS_USE_FLOAT16

OS = $(shell uname -s)

SHAREDEXT   = so
SHAREDFLAGS = -shared

ifeq ($(OS),Darwin)
    SHAREDEXT   = dylib
    SHAREDFLAGS = -dynamiclib -undefined dynamic_lookup
endif

MKDIR_P      = /bin/mkdir -p
PYTHON       = 
SWIG         = swig
AR          ?= ar 

prefix      ?= /usr/local
exec_prefix ?= ${prefix}
libdir       = ${exec_prefix}/lib
includedir   = ${prefix}/include

[beauby]

PYTHONCFLAGS =   -I

and

PYTHON       = 

Those lines indicate that ./configure did not find your python binary. Could you post the output of ./configure with your options?

[WIll-Xu35]

@beauby The output of the ./configure command is:

checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking whether g++ supports C++11 features with -std=c++11... yes
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking whether make sets $(MAKE)... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for /usr/include/python3.5m... no
checking for Python C flags... ./configure: line 4138: -c: command not found

checking for swig... swig
checking how to run the C++ preprocessor... g++ -std=c++11 -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for nvcc... /usr/local/cuda-9.0/bin/nvcc
checking cuda.h usability... yes
checking cuda.h presence... yes
checking for cuda.h... yes
checking for cublasAlloc in -lcublas... yes
checking for cudaSetDevice in -lcudart... yes
checking float.h usability... yes
checking float.h presence... yes
checking for float.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdint.h... (cached) yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for unistd.h... (cached) yes
checking for stdbool.h that conforms to C99... no
checking for _Bool... no
checking for inline... inline
checking for int32_t... yes
checking for int64_t... yes
checking for C/C++ restrict keyword... __restrict
checking for size_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint64_t... yes
checking for uint8_t... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for getpagesize... yes
checking for working mmap... yes
checking for clock_gettime... yes
checking for floor... yes
checking for gettimeofday... yes
checking for memmove... yes
checking for memset... yes
checking for munmap... yes
checking for pow... yes
checking for sqrt... yes
checking for strerror... yes
checking for strstr... yes
checking for g++ -std=c++11 option to support OpenMP... -fopenmp
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking if sgemm_ is being linked in already... no
checking for sgemm_ in -lmkl_intel_lp64... no
checking for sgemm_ in -lmkl... no
checking for sgemm_ in -lopenblas... yes
checking for cheev_... yes
checking target system type... x86_64-pc-linux-gnu
checking for cpu arch... x86_64-pc-linux-gnu CPUFLAGS+=-msse4 -mpopcnt CXXFLAGS+=-m64
configure: creating ./config.status
config.status: creating makefile.inc

And the exact command I used is:
./configure --with-cuda=/usr/local/cuda-9.0 --with-cuda-arch="-gencode=arch=compute_61,code=sm_61" --with-python=/usr/include/python3.5m

Thanks.

[beauby]

--with-python=/usr/include/python3.5m: the --with-python option expects a binary, not a directory.

[WIll-Xu35]

@beauby So does it mean I should pass /usr/bin/python3.5m instead of /usr/include/python3.5m to --with-python?

But I tried and the same error still occurs. The generated makefile.inc still hasn't got PYTHONCFLAG and PYTHON value. The whole file looks like this:

CXX          = g++ -std=c++11
CXXCPP       = g++ -std=c++11 -E
CPPFLAGS     = -DFINTEGER=int  -fopenmp -I/usr/local/cuda-9.0/include 
CXXFLAGS     = -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra
CPUFLAGS     = -msse4 -mpopcnt
LDFLAGS      = -fopenmp  -L/usr/local/cuda-9.0/lib64 
LIBS         = -lopenblas   -lcudart -lcublas 
PYTHONCFLAGS =   -I

NVCC         = /usr/local/cuda-9.0/bin/nvcc
CUDA_ROOT    = /usr/local/cuda-9.0
CUDA_ARCH    = -gencode=arch=compute_61,code=sm_61
NVCCFLAGS    = -I $(CUDA_ROOT)/targets/x86_64-linux/include/ \
-Xcompiler -fPIC \
-Xcudafe --diag_suppress=unrecognized_attribute \
$(CUDA_ARCH) \
-lineinfo \
-ccbin $(CXX) -DFAISS_USE_FLOAT16

OS = $(shell uname -s)

SHAREDEXT   = so
SHAREDFLAGS = -shared

ifeq ($(OS),Darwin)
    SHAREDEXT   = dylib
    SHAREDFLAGS = -dynamiclib -undefined dynamic_lookup
endif

MKDIR_P      = /bin/mkdir -p
PYTHON       = 
SWIG         = swig
AR          ?= ar 

prefix      ?= /usr/local
exec_prefix ?= ${prefix}
libdir       = ${exec_prefix}/lib
includedir   = ${prefix}/include

And the command executed is ./configure --with-cuda=/usr/local/cuda-9.0 --with-cuda-arch="-gencode=arch=compute_61,code=sm_61" --with-python=/usr/bin/python3.5m

[beauby]

Could you try with:
./configure --with-cuda=/usr/local/cuda-9.0 --with-cuda-arch="-gencode=arch=compute_61,code=sm_61" --with-python=python3.5m (i.e. without /usr/bin in front of python3.5m)?

[WIll-Xu35]

I tried your suggested command and the first error is gone. However, there is now a bunch of new errors:

[beauby]

Your swig version is too old. Please try 3.0.12.

On Wed 8 May 2019 at 03:28, WIll-Xu35 [e-mail] wrote:

I tried your suggested command and the first error is gone. However, there
is now a bunch of new errors:

[image: Screenshot from 2019-05-08 09-26-18]
https://user-images.githubusercontent.com/40563560/57342829-5acbe380-7173-11e9-851d-47dfea0b6b66.png

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/faiss/issues/803#issuecomment-490312406,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQ46YS2T4GXV6XEMUXNJ2DPUIUEZANCNFSM4HICNI2A
.

>

Lucas Hosseini
lucas.[email protected]

[WIll-Xu35]

@beauby After installing swig=3.0.12, everything works fine. Thanks!

[beauby]

Glad to hear that 👍

[rishabh-shrivastava]

I am getting the same error:

g++_ -I. -fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp -g -fPIC -fopenmp ~/.pyenv/versions/3.5.2/include/python3.5m/ ~/.pyenv/versions/3.5.2/lib/python3.5/site-packages/numpy/core/include/ -shared \
-o python/_swigfaiss.so python/swigfaiss_wrap.cxx libfaiss.a /usr/lib/libopenblas.so.0
python/swigfaiss_wrap.cxx:154:21: fatal error: Python.h: No such file or directory
compilation terminated.
Makefile:83: recipe for target 'python/_swigfaiss.so' failed
make: * [python/_swigfaiss.so] Error 1

Python.h is located in ~/.pyenv/versions/3.5.2/include/python3.5m/

Makefile.inc:

CC=g++

CFLAGS=-fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare -std=c++11 -fopenmp
LDFLAGS=-g -fPIC  -fopenmp

SHAREDEXT=so
SHAREDFLAGS=-shared
FAISSSHAREDFLAGS=-shared

BLASLDFLAGS=/usr/lib/libopenblas.so.0

SWIGEXEC=swig

PYTHONCFLAGS=~/.pyenv/versions/3.5.2/include/python3.5m/ ~/.pyenv/versions/3.5.2/lib/python3.5/site-packages/numpy/core/include/
PYTHON=python3.5m

CC11=g++

CUDAROOT=/usr/local/cuda-9.0/

CUDACFLAGS=-I$(CUDAROOT)/include

NVCC=$(CUDAROOT)/bin/nvcc

NVCCFLAGS= $(CUDAFLAGS) \
   -I $(CUDAROOT)/targets/x86_64-linux/include/ \
   -Xcompiler -fPIC \
   -Xcudafe --diag_suppress=unrecognized_attribute \
   -gencode arch=compute_35,code="compute_35" \
   -gencode arch=compute_52,code="compute_52" \
   -gencode arch=compute_60,code="compute_60" \
   --std c++11 -lineinfo \
   -ccbin $(CC11) -DFAISS_USE_FLOAT16

BLASLDFLAGSNVCC=-Xlinker $(BLASLDFLAGS)
BLASLDFLAGSSONVCC=-Xlinker  $(BLASLDFLAGS)

@beauby Output of python -c "import sysconfig; print(sysconfig.get_path('include'))" is /usr/local/include/python2.7 which is not an existing path
Output of python3 -c "import distutils.sysconfig; print(distutils.sysconfig.get_python_inc())" is
/usr/include/python3.5m which is an existing path and Python.h is present.

[beauby]

@rishabh-shrivastava Please open a separate issue.

[dannyjeck]

Just noting for future ubuntu16.04 users that you need to update swig outside of apt-get since the latest version is the broken 3.0.8. http://weegreenblobbie.com/?p=263 has instructions that worked for me.

[Krunal79-flutter]

Already Solved At Here