Apex: Warning: apex was installed without --cuda_ext.

[amuier]

I install apex according this sentence:

python setup.py install --cuda_ext --cpp_ext

2.After that, using

import apex

to test, but it report warning as following:
Warning: apex was installed without --cuda_ext. Fused syncbn kernels will be unavailable. Python fallbacks will be used instead.
Warning: apex was installed without --cuda_ext. FusedAdam will be unavailable.
Warning: apex was installed without --cuda_ext. FusedLayerNorm will be unavailable.

Is there any problem?

[mcarilli]

Ok, that is a little strange. If the --cuda-ext and --cpp_ext options are working as intended, you should have seen c++ and cuda files being built as part of the install (long gcc and nvcc command lines). Did you see these?

[mcarilli]

I just received a new notification from this issue, but I don't see any new information posted. Is everything working on your end? Can I close this?

[matthew-z]

Hi, I have the same problem: I installed apex with the flags and I confirmed that there is fused_adam_cuda.cpython-36m-x86_64-linux-gnu.so in build/lib.linux-x86_64-3.6.

However, I still have the warning.

Warning:  apex was installed without --cuda_ext. Fused syncbn kernels will be unavailable.  Python fallbacks will be used instead.
Warning:  apex was installed without --cuda_ext.  FusedAdam will be unavailable.
Warning:  apex was installed without --cuda_ext.  FusedLayerNorm will be unavailable.

Environment:
Ubuntu18.04+CUDA10.0+CUDNN7.4.1+python3.6.6
I installed pytorch 1.0 by compiling from source.
installation log: https://gist.github.com/matthew-z/86569e055ba41b2d9dfde95fc904f72f

In [2]: import optimizers
--------------------------------------------------------------------
ImportError                        Traceback (most recent call last)
<ipython-input-2-64b3677a7f6b> in <module>()
----> 1 import optimizers

~/Documents/apex/apex/optimizers/__init__.py in <module>()
----> 1 from .fused_adam import FusedAdam

~/Documents/apex/apex/optimizers/fused_adam.py in <module>()
      1 import torch
----> 2 import fused_adam_cuda
      3
      4 class FusedAdam(torch.optim.Adam):
      5

ImportError: /home/zhaohao/.anaconda3/lib/python3.6/site-packages/apex-0.1-py3.6-linux-x86_64.egg/fused_adam_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration

I guess it is because of CUDA10? I installed it correctly on a server which is using CUDA9.0

[yaysummeriscoming]

@matthew-z weird, for me apex isn't giving warnings & my setup is the same as yours, except that I'm using ubuntu 16.04 minimal.

[matthew-z]

@yaysummeriscoming @amuier

It seems that apex relies on the latest pytorch C++ API (which is not stable yet). Thus, Installing the latest nightly pytorch and rebuilding apex again (remember to clean apex/build) can solve this problem

[yaysummeriscoming]

@matthew-z interesting, good to know! There seems to be a huge amount of work going on in pytorch right now.

[mcarilli]

The C++ API was very much in flux prior to 1.0. It's still been mildly in flux since commits started to be labelled "1.0". I'm glad upgrading to the latest pytorch 1.0 fixes the problem, but in principle, I'm surprised that an earlier version also labelled "1.0" would fail.

By the way, the official 1.0 was just released today!

@matthew-z Do you happen to know the commit hash you built from when you were seeing the original symptoms?

Another possible issue is that you shouldn't attempt to import apex from within the top level directory of your cloned repository. PYTHONPATH always implicitly includes the current directory, so may attempt to "import apex" directly from the "apex" subdirectory, rather than looking for it in the global install directory on your system (which is the correct/desired behavior).

[matthew-z]

@mcarilli

Very excited to see 1.0 finally get released.

I didn't record the exact commit hash, but I think I built it around Nov.7

Actually, I imported optimizers in the apex/apex directory. I guess it is not the issue as currently I can import optimizers successfully after re-building pytorch, but you are correct I should do "import apex.optimizers" outside the apex repository.

[enod]

With pytorch version 1.0, I'm still getting the warnings:

Warning:  apex was installed without --cuda_ext. Fused syncbn kernels will be unavailable.  Python fallbacks will be used instead.
Warning:  apex was installed without --cuda_ext.  FusedAdam will be unavailable.
Warning:  apex was installed without --cuda_ext.  FusedLayerNorm will be unavailable.

[tstandley]

I also still get the warnings after installing with
python setup.py install --cuda_ext --cpp_ext
I installed 1.0 via conda on Ubuntu 18.04:
conda install pytorch torchvision cuda100 -c pytorch

[mcarilli]

Did you see gcc and nvcc compilation during the install process? Try this:

pip uninstall apex
cd apex
rm -rf build (if it exists)
python setup.py install --cuda_ext --cpp_ext 

You should see something like https://gist.github.com/mcarilli/18a44f2988ec9e3ee03e452b5fa2432a#file-output-txt-L69-L93 if the cpp and cuda extensions are actually being built.

[tstandley]

I don't get the cpp output. I discovered this line in the output that I got:
Warning: nvcc is not available. Ignoring --cuda-ext
I tried:
sudo apt install nvidia-cuda-toolkit
But it resulted in the apex installer having C++ errors. It installed nvcc
9.1 though, which might be why.

Part of the beauty of Pytorch is that you don't have to install CUDA. How
important are these extensions for performance?

[mcarilli]

That's true, if you install Pytorch from the conda package (which is a static binary that contains all the cuda runtime code statically linked in) you only need the cuda driver on your system to use GPUs. You don't actually need nvcc accessible.

If you wish to install the extensions on bare metal, you can get cuda 10 from the official download page.

I think the easiest way forward is to use a devel container with cuda 10 preinstalled, for which there are two options, outlined here.

The performance impact of the extensions varies based on your application. If you're only using apex.parallel.DistributedDataParallel, it isn't much. If you're using sync batchnorm, it will be significant. I don't believe FusedAdam or FusedLayerNorm have Python fallbacks, so these will be unavailable.

[mangotree3]

I had this problem too and managed to solve it. In my case, the problem was that the apex module fused_adam_cuda was calling the file libcudart.so.9.1 in the wrong cuda directory /usr/local/cuda/lib that had been previously installed and therefore couldn't find it. The file was actually located in another directory /usr/local/cuda/lib64. I simply changed the env variable LD_LIBRARY_PATH to include the right dir path (/usr/local/cuda/lib64) and everything has been working fine since then.

The way the .__init__ of apex is coded made it a bit tricky to find the problem. There is a try/except trying to import the module optimizers. Even if the apex optimizers module exists and has been well installed, it will return that --cuda_ext is not installed because the optimizers module cannot be imported properly due to an error occurring as a result of its faulty cuda dependencies/imports.

Hope this helps anyone with the same problem.

[PistonY]

I also got this issue.
But I success with

pip uninstall apex
cd apex
rm -rf build (if it exists)
python setup.py install --cuda_ext --cpp_ext 

What the different between using

git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

@mcarilli

[KokeCacao]

I got

Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  Original ImportError was: ImportError('/home/k1412042720/miniconda3/envs/ml/lib/$
ython3.7/site-packages/amp_C.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration') 

only when I installed with

$ git clone https://github.com/NVIDIA/apex
$ cd apex
$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Here is my env

__Python VERSION: 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21)               
[GCC 7.3.0]                                                                                                                                                                                                                                    
__pyTorch VERSION: 1.1.0                                                                                                                                                                                                                       
__CUDA VERSION: 9.0.176                                                                                                                                                                                                                        
__CUDNN VERSION: 7501                                                                                                                                                                                                                          
__Number CUDA Devices: 1
Available devices: 1                                                                                                                                                                                                                           
Current cuda device: 0

[ptrblck]

@KokeCacao Did you check, if the installation was successful using this command?
Could you try to reinstall apex and make sure no old apex versions are installed in your current environment.

[KokeCacao]

@KokeCacao Did you check, if the installation was successful using this command?
Could you try to reinstall apex and make sure no old apex versions are installed in your current environment.

Thank you for your reply.

I didn't check whether my installation was successful, but I assume the program will delete unsuccessful installations anyway. Since I commented out the if (bare_metal_major != torch_binary_major) or (bare_metal_minor != torch_binary_minor) statement under def check_cuda_torch_binary_vs_bare_metal(cuda_dir) during the installation, this might be where the warning is coming from -- I am not certain.

(I have no problem training the network, but I guess it will be less stable.)

[ptrblck]

Which CUDA version are you using locally (nvcc --version) and which one shipped with PyTorch?
It's generally not recommended to remove this check and the warning might indeed come from this fact.

[KokeCacao]

NVCC Cuda

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

Torch

>>> torch.version.cuda
'9.0.176'

@ptrblck

[ptrblck]

@KokeCacao Could you install the latest PyTorch binaries with CUDA9.2?
You can find the install instructions here.

[KokeCacao]

@KokeCacao Could you install the latest PyTorch binaries with CUDA9.2?
You can find the install instructions here.

  cudatoolkit                                9.0-h13b8566_0 --> 9.2-0
  pytorch              1.1.0-py3.7_cuda9.0.176_cudnn7.5.1_0 --> 1.2.0-py3.7_cuda9.2.148_cudnn7.6.2_0
  torchvision                        0.3.0-py37_cu9.0.176_1 --> 0.4.0-py37_cu92

I still got the same problem after updating

Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  Original ImportError was: ImportError('libcudart.so.9.0: cannot open shared object file: No such file or directory')

[ptrblck]

The ImportError points to libcudart.so.9.0.
Could you rebuild apex and make sure that your nvcc 9.2 is being used?

[KokeCacao]

The ImportError points to libcudart.so.9.0.
Could you rebuild apex and make sure that your nvcc 9.2 is being used?

It worked after rebuild. Thank you for your patient explanation.

[ptrblck]

@ferryjoker Could you try to reinstall it and post the install log here, please?

[ptrblck]

You are missing the ./ at the end of the install command:

pip install -v --no-cache-dir ./
# or for full performance and functionality
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

[ptrblck]

Could you post the complete log by wrapping it into three backticks ```?

[Ferryboat-cmd]

RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries. Pytorch binaries were compiled with Cuda 9.0.176. In some cases, a minor-version mismatch will not cause later errors: https://github.com/NVIDIA/apex/pull/323#discussion_r287021798. You can try commenting out this check (at your own risk). Running setup.py install for apex ... error Cleaning up... Removing source in /tmp/pip-req-build-x3mm9720 Removed build tracker '/tmp/pip-req-tracker-003uce68' ERROR: Command "/home/zhangshuchen/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-req-build-x3mm9720/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-i0lukdyw/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-req-build-x3mm9720/

[ptrblck]

The CUDA version shipped with the PyTorch binaries do not match your local CUDA install.
If just the minor version creates the mismatch, you could comment the condition as described in the error message. Otherwise it is recommended to install the matching CUDA version on your machine.

[kwea123]

I got these warnings:

Warning:  using Python fallback for SyncBatchNorm, possibly because apex was installed without --cuda_ext.  The exception raised when attempting to import the cuda backend was:  No module named 'syncbn'
Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.

When it is written using Python fallback does it just mean that the performance would not be as good, or the functionality is broken? For example the first one does it mean it just uses a slower version of SyncBatchNorm or it doesn't use SyncBatchNorm at all? It doesn't matter much if it just runs slower..

[ptrblck]

It should run slower using the Python fallback.

[RSKothari]

@ptrblck the pitfalls of python fall-back aren't exactly clear. Is it safe to assume that only speed is impacted?

[IsakWesterlundBitville]

I had the same problem but it fixed itself after reinstalling cuda 10.2