Faiss: Faiss 1.5.3: build fail in windows with MSYS2 MinGW 64-bit

[leyewen]

Summary

I try to build Faiss-1.5.3-CPU in windows and finally succeeded!
But I got lots of errors.
Summarize the problems in the process.

Platform

OS: Windows 10
Faiss version: 1.5.3
Faiss compilation options: ./configure --without-cuda
msys2: msys2-x86_64-20200602
mingw64: mingw64/mingw-w64-x86_64-toolchain
gcc: gcc version 10.1.0 (Rev3, Built by MSYS2 project) (mingw64/mingw-w64-x86_64-gcc 10.1.0-3)
make: GNU Make 4.3 (mingw64/mingw-w64-x86_64-make 4.3-1 )

Running on:

• [x] CPU
• [ ] GPU

Interface:

• [x] C++
• [ ] Python

Reproduction instructions

---

Error:

In file included from Clustering.cpp:19:
Clustering.cpp:87:31: error: 'finite' was not declared in this scope; did you mean 'finitef'?
   87 |       FAISS_THROW_IF_NOT_MSG (finite (x_in[i]),

Clustering.cpp line 86

#

    for (size_t i = 0; i < nx * d; i++) {
      FAISS_THROW_IF_NOT_MSG (std::isfinite (x_in[i]),
                        "input contains NaN's or Inf's");
    }

#

---

Error:

g++ -std=c++11 -DFINTEGER=int  -fopenmp  -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra -mavx2 -mf16c -msse4 -mpopcnt -c index_io.cpp -o index_io.o
index_io.cpp:15:10: fatal error: sys/mman.h: No such file or directory
   15 | #include <sys/mman.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:40：index_io.o] error 1

require mman:
https://github.com/witwall/mman-win32
https://github.com/witwall/mman-win32/blob/master/Makefile

#

cd /c/mman-win32/
./configure
make
make libmman.dll
cp mman.h /mingw64/include/sys/mman.h
cp libmman.a /mingw64/lib/libmman.a
cp libmman.dll /mingw64/lib/libmman.dll
cp libmman.dll.a /mingw64/lib/libmman.dll.a

#

---

Error:

g++ -std=c++11 -DFINTEGER=int  -fopenmp  -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra -mavx2 -mf16c -msse4 -mpopcnt -c utils_simd.cpp -o utils_simd.o
utils_simd.cpp: In function 'void faiss::fvec_madd(size_t, const float*, float, const float*, float*)':
utils_simd.cpp:709:12: error: cast from 'const float*' to 'long int' loses precision [-fpermissive]
  709 |         ((((long)a) | ((long)b) | ((long)c)) & 15) == 0)
      |            ^~~~~~~
utils_simd.cpp:709:24: error: cast from 'const float*' to 'long int' loses precision [-fpermissive]
  709 |         ((((long)a) | ((long)b) | ((long)c)) & 15) == 0)
      |                        ^~~~~~~
utils_simd.cpp:709:36: error: cast from 'float*' to 'long int' loses precision [-fpermissive]
  709 |         ((((long)a) | ((long)b) | ((long)c)) & 15) == 0)
      |                                    ^~~~~~~
utils_simd.cpp: In function 'int faiss::fvec_madd_and_argmin(size_t, const float*, float, const float*, float*)':
utils_simd.cpp:796:12: error: cast from 'const float*' to 'long int' loses precision [-fpermissive]
  796 |         ((((long)a) | ((long)b) | ((long)c)) & 15) == 0)
      |            ^~~~~~~
utils_simd.cpp:796:24: error: cast from 'const float*' to 'long int' loses precision [-fpermissive]
  796 |         ((((long)a) | ((long)b) | ((long)c)) & 15) == 0)
      |                        ^~~~~~~
utils_simd.cpp:796:36: error: cast from 'float*' to 'long int' loses precision [-fpermissive]
  796 |         ((((long)a) | ((long)b) | ((long)c)) & 15) == 0)
      |                                    ^~~~~~~
utils_simd.cpp: In function '__m128 faiss::masked_read(int, const float*)':

mingw64 cast error:

utils_simd.cpp line 709

#

        ((((long long)a) | ((long long)b) | ((long long)c)) & 15) == 0)

#

utils_simd.cpp line 796

#

        ((((long long)a) | ((long long)b) | ((long long)c)) & 15) == 0)

#

---

Error:

g++ -std=c++11 -DFINTEGER=int  -fopenmp  -fPIC -m64 -Wno-sign-compare -g -O3 -Wall -Wextra -mavx2 -mf16c -msse4 -mpopcnt -c WorkerThread.cpp -o WorkerThread.o
ar r libfaiss.a AutoTune.o AuxIndexStructures.o Clustering.o distances.o FaissException.o hamming.o Heap.o HNSW.o Index.o index_io.o IndexBinary.o IndexBinaryFlat.o IndexBinaryFromFloat.o IndexBinaryHNSW.o IndexBinaryIVF.o IndexFlat.o IndexHNSW.o IndexIVF.o IndexIVFFlat.o IndexIVFPQ.o IndexIVFSpectralHash.o IndexLSH.o IndexPQ.o IndexReplicas.o IndexScalarQuantizer.o IndexShards.o InvertedLists.o IVFlib.o MetaIndexes.o OnDiskInvertedLists.o PolysemousTraining.o ProductQuantizer.o utils.o utils_simd.o VectorTransform.o WorkerThread.o
C:\msys64\mingw64\bin\ar.exe: Creating libfaiss.a
g++ -std=c++11 -shared -fopenmp   -o libfaiss.so AutoTune.o AuxIndexStructures.o Clustering.o distances.o FaissException.o hamming.o Heap.o HNSW.o Index.o index_io.o IndexBinary.o IndexBinaryFlat.o IndexBinaryFromFloat.o IndexBinaryHNSW.o IndexBinaryIVF.o IndexFlat.o IndexHNSW.o IndexIVF.o IndexIVFFlat.o IndexIVFPQ.o IndexIVFSpectralHash.o IndexLSH.o IndexPQ.o IndexReplicas.o IndexScalarQuantizer.o IndexShards.o InvertedLists.o IVFlib.o MetaIndexes.o OnDiskInvertedLists.o PolysemousTraining.o ProductQuantizer.o utils.o utils_simd.o VectorTransform.o WorkerThread.o -lopenblas
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: OnDiskInvertedLists.o: in function `faiss::OnDiskInvertedLists::~OnDiskInvertedLists()':
C:\faiss/OnDiskInvertedLists.cpp:386: undefined reference to `munmap'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: OnDiskInvertedLists.o: in function `faiss::OnDiskInvertedLists::do_mmap()':
C:\faiss/OnDiskInvertedLists.cpp:280: undefined reference to `mmap'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: OnDiskInvertedLists.o: in function `faiss::OnDiskInvertedLists::update_totsize(unsigned long long)':
C:\faiss/OnDiskInvertedLists.cpp:297: undefined reference to `munmap'
collect2.exe: error: ld returned 1 exit status
make: *** [Makefile:37：libfaiss.so] error 1

makefile.inc line 12

#

LIBS         = -lopenblas -lmman  

#

---

[mdouze]

@leyewen Thank you very much for these insights.
@beauby is working on an official Windows port and ran in many of the errors you see here.
I don't think we ambition to support the mmap functionality on Windows. It is unlikely that it will work smoothly anyways.

[bitsun]

I have created a branch (https://github.com/bitsun/faiss) which successfully builds FAISS under windows, however with Visual Studio 2017/2019. Note that I deliberately skip the OnDiskInvertedList. If you do not need it, it is worth a try. It passed all unit tests that i know of, except some FP16 ones with GPU. it uses cmake files, so it can also be built in Linux, (presumably works in Mac OS as well, but i did not test it).

[mdouze]

Here is the PR to follow for the official windows port: #1300
And indeed it moves to cmake to support compiling cross-platform.

[bitsun]

I am glad that Faiss team finally decides to support windows, i checked the changes that have been made, they are quite similar to the ones that i have made. BTW, how can i participate?

[leyewen]

Hearing the news, I was really happy.

[liyiersan]

I have created a branch (https://github.com/bitsun/faiss) which successfully builds FAISS under windows, however with Visual Studio 2017/2019. Note that I deliberately skip the OnDiskInvertedList. If you do not need it, it is worth a try. It passed all unit tests that i know of, except some FP16 ones with GPU. it uses cmake files, so it can also be built in Linux, (presumably works in Mac OS as well, but i did not test it).

I have read the INSTALL_CMAKE.md in your branch, but I can not understand the build-faiss, could you please show me more details abouts step 2 and step 3, images about operations may be better. As a matter of fact, I don't konw how to build faiss with Vistual Studio 2019 on Windows 10 . I would apprecite it if you could help me.

[mdouze]

No activity, closing.