Apex: Installation issue, no error when using but warning flag.

[Azerus13]

Hello,

I'm working on windows 10 with 1 light gpu (gtx 1050ti maxQ).
I created a conda environment that include :
python 3.7.6
pytorch 1.4.0
cudatoolkit 10.1.243

I tried to follow the quick start installation for apex on windows but got the following error :
apex RuntimeError: --cuda_ext was requested, but nvcc was not found.

As the recommended installation has failed, I've look up to conda-forge and directly installed apex with : https://anaconda.org/conda-forge/nvidia-apex

The conda installation went well. I can train a model with adam optimizer and mixed precision trainning without any error but I still got the following warning :
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")

So I've got 2 questions :
Is this warning problematic ?
If I need to re-install apex, how I can solve the missing nvcc issue ?

Any advice are welcome :)

Regards,
Thomas

[Lornatang]

@Azerus13
Please install CUDNN files with cuda10.1！

[Azerus13]

Hi @Lornatang thanks for replying.

I've re-installed cuda10.1, installed the cuDNN files, and re-installed pytorch from source.

When I run :
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .
I got the following error (not the full message) :
` Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:12:52_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.1, V10.1.243
from C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1/bin

running install
running build
running build_py
running build_ext
building 'apex_C' extension
creating C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\build\temp.win-amd64-3.7
creating C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\build\temp.win-amd64-3.7\Release
creating C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\build\temp.win-amd64-3.7\Release\csrc
Emitting ninja build file C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\build\temp.win-amd64-3.7\Release\build.ninja...
Compiling objects...
[1/1] cl /showIncludes /c /nologo /Ox /W3 /GL /DNDEBUG /MD /MD /wd4819 /EHsc -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include\TH -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include\THC -IC:\Users\thoma\Miniconda3\include -IC:\Users\thoma\Miniconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" -c C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\csrc\flatten_unflatten.cpp /FoC:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\build\temp.win-amd64-3.7\Release\csrc/flatten_unflatten.obj -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 /std:c++14
FAILED: C:/Users/thoma/AppData/Local/Temp/pip-req-build-_vs5zy5m/build/temp.win-amd64-3.7/Release/csrc/flatten_unflatten.obj
cl /showIncludes /c /nologo /Ox /W3 /GL /DNDEBUG /MD /MD /wd4819 /EHsc -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include\TH -IC:\Users\thoma\Miniconda3\lib\site-packages\torch\include\THC -IC:\Users\thoma\Miniconda3\include -IC:\Users\thoma\Miniconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" -c C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\csrc\flatten_unflatten.cpp /FoC:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\build\temp.win-amd64-3.7\Release\csrc/flatten_unflatten.obj -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 /std:c++14
CreateProcess failed: The system cannot find the file specified.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\Users\thoma\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 1311, in _run_ninja_build
    check=True)
  File "C:\Users\thoma\Miniconda3\lib\subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\setup.py", line 275, in <module>
    extras_require=extras,
  File "C:\Users\thoma\Miniconda3\lib\site-packages\setuptools\__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\thoma\Miniconda3\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Users\thoma\Miniconda3\lib\distutils\dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "C:\Users\thoma\Miniconda3\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Users\thoma\Miniconda3\lib\site-packages\setuptools\command\install.py", line 61, in run
    return orig.install.run(self)
  File "C:\Users\thoma\Miniconda3\lib\distutils\command\install.py", line 545, in run
    self.run_command('build')
  File "C:\Users\thoma\Miniconda3\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\Users\thoma\Miniconda3\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Users\thoma\Miniconda3\lib\distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
  File "C:\Users\thoma\Miniconda3\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\Users\thoma\Miniconda3\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Users\thoma\Miniconda3\lib\site-packages\setuptools\command\build_ext.py", line 87, in run
    _build_ext.run(self)
  File "C:\Users\thoma\Miniconda3\lib\distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "C:\Users\thoma\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 540, in build_extensions
    build_ext.build_extensions(self)
  File "C:\Users\thoma\Miniconda3\lib\distutils\command\build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "C:\Users\thoma\Miniconda3\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "C:\Users\thoma\Miniconda3\lib\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "C:\Users\thoma\Miniconda3\lib\distutils\command\build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "C:\Users\thoma\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 522, in win_wrap_ninja_compile
    with_cuda=with_cuda)
  File "C:\Users\thoma\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 1086, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "C:\Users\thoma\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 1324, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
Running setup.py install for apex ... error

Rolling back uninstall of apex
Moving to c:\users\thoma\miniconda3\lib\site-packages\apex-0.1.dist-info\
from c:\users\thoma\miniconda3\lib\site-packages\~pex-0.1.dist-info
Moving to c:\users\thoma\miniconda3\lib\site-packages\apex\
from c:\users\thoma\miniconda3\lib\site-packages\~pex
Replacing c:\users\thoma\miniconda3\lib\site-packages\apex-0.1.dist-info\ from c:\users\thoma\miniconda3\lib\site-packages\~pex-0.1.dist-info
Replacing c:\users\thoma\miniconda3\lib\site-packages\apex\ from c:\users\thoma\miniconda3\lib\site-packages\~pex
Cleaning up...
Removing source in C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m
Removed build tracker: 'C:\Users\thoma\AppData\Local\Temp\pip-req-tracker-zaop2lnm'
ERROR: Command errored out with exit status 1: 'C:\Users\thoma\Miniconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\setup.py'"'"'; __file__='"'"'C:\Users\thoma\AppData\Local\Temp\pip-req-build-_vs5zy5m\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record 'C:\Users\thoma\AppData\Local\Temp\pip-record-3ovom7vg\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\thoma\Miniconda3\Include\apex' Check the logs for full command output.`

[Lornatang]

@Azerus13
Re-clone apex. Keep the PyTorch version above 1.3

[Azerus13]

@Lornatang

So I tried re-cloning apex, I still got the same error message. Pytorch is version 1.5.0a0+5b92291

[Lornatang]

@Azerus13
I have successfully configured as follows:
Python: 3.7.5
PyTorch: 1.4.0
CUDA: 10.1.243
CUDNN: 7.6.4.38
Apex: 0.1.0

[Azerus13]

@Lornatang
Are you using Windows 10 ? and conda ?
Did you installed pytorch from source ?

I will try using your configuration in the next week.
Thanks for your support and quick answers.

[Lornatang]

@Azerus13
I tried it on Windows. I installed python 3.7.6.6.exe, configured the environment variables, downloaded vs2019, and followed the here to install c++ library. The only difference is that I don't use anconda, I use origin pip.
The rest of the environment is just like my list of configuration versions.

I don't recommend trying Apex on Windows.

[Azerus13]

I order to solve this problem and get cleaner environment I moved to Linux. All is working.

[RSKothari]

Did anyone manage to answer if the warnings are problematic? What happens if APEX keeps using default python fallback? I'm trying to understand effort vs benefits in this case.