fairseq installation failed

[amit0012]

🐛 Bug

fairseq installation failed

To Reproduce

Steps to reproduce the behavior (always include the command you ran):

(venv) C:\Users\AMIT\PycharmProjects\nmtAmit\fairseq>pip install --editable ./
Obtaining file:///C:/Users/AMIT/PycharmProjects/nmtAmit/fairseq
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing wheel metadata ... done
Collecting cython
Using cached Cython-0.29.21-cp37-cp37m-win_amd64.whl (1.6 MB)
Collecting regex
Using cached regex-2020.7.14-cp37-cp37m-win_amd64.whl (268 kB)
Collecting cffi
Using cached cffi-1.14.0-cp37-cp37m-win_amd64.whl (176 kB)
Collecting tqdm
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError( onnection object at 0x000001C34F6E9408>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/af/88/7b0ea5fa8192d1733dea459a9e3059afc87819cb4072c43263f2e
c7ab768/tqdm-4.48.0-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError( onnection object at 0x000001C34F6E9B48>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/af/88/7b0ea5fa8192d1733dea459a9e3059afc87819cb4072c43263f2e
c7ab768/tqdm-4.48.0-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError( onnection object at 0x000001C34F6E9CC8>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/af/88/7b0ea5fa8192d1733dea459a9e3059afc87819cb4072c43263f2e
c7ab768/tqdm-4.48.0-py2.py3-none-any.whl
Downloading tqdm-4.48.0-py2.py3-none-any.whl (67 kB)
|████████████████████████████████| 67 kB 434 kB/s
Requirement already satisfied: torch in c:\users\amit\pycharmprojects\nmtamit\venv\lib\site-packages (from fairseq==0.9.0) (1.5.1+cu101)
Requirement already satisfied: numpy in c:\users\amit\pycharmprojects\nmtamit\venv\lib\site-packages (from fairseq==0.9.0) (1.19.0)
Collecting sacrebleu
Using cached sacrebleu-1.4.12-py3-none-any.whl (54 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: future in c:\users\amit\pycharmprojects\nmtamit\venv\lib\site-packages (from torch->fairseq==0.9.0) (0.18.2)
Collecting mecab-python3==0.996.5
Using cached mecab-python3-0.996.5.tar.gz (65 kB)
Collecting portalocker
Downloading portalocker-1.7.1-py2.py3-none-any.whl (10 kB)
Collecting pywin32!=226; platform_system == "Windows"
Using cached pywin32-228-cp37-cp37m-win_amd64.whl (9.1 MB)
Using legacy setup.py install for mecab-python3, since package 'wheel' is not installed.
Installing collected packages: cython, regex, pycparser, cffi, tqdm, mecab-python3, pywin32, portalocker, sacrebleu, fairseq
Running setup.py install for mecab-python3 ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\amit\pycharmprojects\nmtamit\venvscripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\AMIT\AppData\Local\Temp\pip-ins
tall-lctmv_th\mecab-python3\setup.py'"'"'; __file__='"'"'C:\Users\AMIT\AppData\Local\Temp\pip-install-lctmv_th\mecab-python3\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"',
open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\AMIT\AppData\Local\Temppip-rec
ord-7q3mmpux\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\amit\pycharmprojects\nmtamit\venvinclude\site\python3.7\mecab-python3'
cwd: C:\Users\AMIT\AppData\Local\Temppip-install-lctmv_th\mecab-python3\
Complete output (10 lines):
WARNING: The wheel package is not available.
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.7
creating build\lib.win-amd64-3.7\MeCab
copying src\MeCab__init__.py -> build\lib.win-amd64-3.7\MeCab
running build_ext
error: [WinError 2] The system cannot find the file specified
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\amit\pycharmprojects\nmtamit\venvscripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\AM
IT\AppData\Local\Temp\pip-install-lctmv_th\mecab-python3\setup.py'"'"'; __file__='"'"'C:\Users\AMIT\AppData\Local\Temp\pip-install-lctmv_th\mecab-python3\setup.py'"'"';f=
getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\User
s\AMIT\AppData\Local\Temppip-record-7q3mmpux\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\amit\pycharmprojects\nmtamit\venvinclude\si
te\python3.7\mecab-python3' Check the logs for full command output.

Environment

• fairseq Version (e.g., 1.0 or master): installation failed
• PyTorch Version (e.g., 1.0): pip install torch==1.5.1+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html
• OS (e.g., Linux): windows 10
• How you installed fairseq (pip, source): both
• Build command you used (if compiling from source):
• Python version: 3.7
• CUDA/cuDNN version: 10.1
• GPU models and configuration: rtx 2060

[myleott]

Sorry, I don't have access to a Windows machine to reproduce this. If you are able to figure it out, please share the details for other people who may run into this issue!

[1mike12]

I'm trying to work this out myself, the problem is somehow related to mecab-python3 as a dependency specifically since just doing an installation of it itself through pip install mecab-python3 works with no problems.

I've narrowed down the fault to a part of the setup.py script by running the same command and trying to see where it breaks since it doesn't tell us what file is missing. Since pip cleans the temp folders, I had to run the install script with the --no-clean flag pip install --editable ./ --no-clean --verbose then run the part of the build that fails:

c:\python38\python.exe -u -c "import sys, setuptools, tokenize; sys.argv[0] = 'C:\\Users\\1mike12\\AppData\\Local\\Temp\\pip-install-dkxr7pxv\\mecab-python3\\setup.py'; __file__= 'C:\\Users\\1mike12\\AppData\\Local\\Temp\\pip-install-dkxr7pxv\\mecab-python3\\setup.py'; f=getattr(tokenize, 'open')(__file__); code=f.read().replace('\r\n', '\n'); f.close(); exec(compile(code, __file__, 'exec'))" install --record 'C:\Users\1mike12\AppData\Local\Temp\pip-record-u9dvxngc\install-record.txt' --single-version-externally-managed
 --compile --install-headers 'c:\python38\Include\mecab-python3'

Whatever is breaking is on this line:

C:\Users\{USERNAME}\AppData\Local\Temp\pip-install-{randomAlphaNumeric}\mecab-python3\setup.py

 def mecab_config(arg):
            # error on next line line
            output = subprocess.check_output(["mecab-config", arg],
                                             env=clocale_env)

when called from

inc_dir  = mecab_config("--inc-dir")

No clue where to go from here, not sure what mecab-config is supposed to be, or how it's supposed to be called since it doesn't exist on the command line

Also, why is this library needed, it seems to be specifically for Japanese, what if we have no interest in japanese support?

[polm]

Hello, I'm the mecab-python3 maintainer. Sorry you're having trouble with the package.

It looks like your installation is pulling in mecab-python3 because of sacrebleu. A recent upgrade to mecab-python3 broke that package, I have opened a PR to fix it but it hasn't been addressed yet.

Versions of mecab-python3 before 1.0 don't have Windows wheels, and will not work on Windows unless the user already has MeCab installed and some other prep ready. 1.0+ has Windows wheels.

If you aren't using Japanese, it's probably fine to comment out the relevant parts of sacrebleu for the time being.

[ypsoh]

I still want to evalutate the model using the BLEU metric, it there a way to do this without disabling --eval-bleu altogether?
I'm wondering if there are any options where we can install sacrebleu that doesn't depend on mecab-python3.

[thilakshiK]

sacrebleu has released a new version and now you can install fairseq on windows 10.

[1mike12]

Just tested installation myself. Everything worked! Should be good to close