Haystack: Unable to install latest haystack version on Windows

[sophgit]

Describe the bug:

I can't install the latest haystack version on Windows and therefore can't use haystack properly. Tried using pip install farm-haystack==0.4.0 and pip install git+https://github.com/deepset-ai/haystack.git. I suspect it has something to do with faiss-cpu not being compatible with Windows. Is there a way to use haystack anyways? Thanks :)

Error message:

AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
----------------------------------------
ERROR: Failed building wheel for faiss-cpu
Running setup.py clean for faiss-cpu
Failed to build faiss-cpu
Installing collected packages: faiss-cpu, farm-haystack
Running setup.py install for faiss-cpu ... error
ERROR: Command errored out with exit status 1

System:

• OS: Windows
• Haystack version (commit or version number): 0.4.0

[lalitpagaria]

Facebook officially support Conda packages on MacOS and Linux (refer thread). So in order to use it on Windows, you have to build package by yourself (refer build instructions) or use linux docker image.

[tanaysoni]

Hi @sophgit, FAISS is currently a required dependency for installing Haystack. In the next weeks, we plan to make it optional, so Haystack can be used on Windows(except the FAISSDocumentStore).

Meanwhile, you could try out the tutorials on Google Colab.

[Zenahr]

Ran into this too. Would be lovely if you could add a little note in here right next to the flag notice and probably here would also be good.

[Zenahr]

@lalitpagaria Is there a quick and dirty way to remove the faiss-dependency ourselves?

[Zenahr]

@sophgit I managed installing the pip package by using the source and modifying one line inside the requirements.txt file.

I just commented out faiss-cpu since I won't be using it for now (I hope).
Assuming the haystack source is inside a folder called ´haystack´ then this following command works (after source modification):

´ pip install haystack -f https://download.pytorch.org/whl/torch_stable.html´

[Zenahr]

@lalitpagaria furthermore uvloop is not supported on Windows either and probably won't be too soon

[lalitpagaria]

@Zenahr I am afraid it is not easy to compile on Window. Faiss team posed this -

Currently Faiss compiles only on Linux and Mac.

Porting to Windows requires:

prepare a makefile or Visual project file [using gnu make on windows is also OK]
converting some system-level calls like random, gettimeofday and pthread_create to their Windows equivalents (or use a glue library that does this) [not necessary, pthread_t is used only in Ondisk, which will not be supported anyways]
the code assumes that long = int64_t at quite some places, when long is 32 bit on Windows [FIXED]
fix the SSE includes (the SSE intrinsics are Intel standard)
find a usable BLAS implementation
optional: port the GPU version and the Python interface.
find a continuous integration that works with windows
If you are interested in undertaking this, please contact us!

I think if you are not using FaissDocumentStore then as @tanaysoni mentioned, making faiss optional or removing from requirement.txt will help. If you do need it then integrating Mivus with haystack would solve this.

[Zenahr]

@lalitpagaria Yeah that's basically what I wanted to say. Maybe we could have a branch windows where these things are accounted for and i.e. some dependencies commented out from the requirements.txt file so that it's straight-forward to work with haystack on Windows machines.

[lalitpagaria]

@Zenahr and @sophgit I raised PR. Can you please test changes on windows platform by running -

pip install git+https://github.com/lalitpagaria/haystack.git@fix_windows_installation

[lalitpagaria]

Good news! Now Faiss (CPU) supports Windows.
https://github.com/facebookresearch/faiss/releases/tag/v1.6.4 (Official release on Conda)
https://pypi.org/project/faiss-cpu/#files (community supported release on pypi)

[tholor]

Nice! We should do some benchmarking to verify performance and then upgrade. Yesterday I had some issues with the PyPI installation, but this got fixed already (https://github.com/kyamagu/faiss-wheels/issues/26)