Apex: Build error with --cuda_ext --cpp_ext
I spent the whole day trying to install the new apex extension without success. I was using the old one, then yesterday i wanted to upgrade. But I'm not able to install it.
I'm using Pytorch nighty with conda, with cuda100. I've tried with pytoch 1.0.1 and It did not work either.
This is what I get when I run the command from apex directory:
python setup.py install --cuda_ext --cpp_ext
Output:
torch.__version__ = 1.1.0.dev20190429
Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
from /opt/cuda/bin
Warning: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries.
Pytorch binaries were compiled with Cuda 10.0.130
running install
running bdist_egg
running egg_info
writing apex.egg-info/PKG-INFO
writing dependency_links to apex.egg-info/dependency_links.txt
writing top-level names to apex.egg-info/top_level.txt
reading manifest file 'apex.egg-info/SOURCES.txt'
writing manifest file 'apex.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/apex
copying apex/__init__.py -> build/lib.linux-x86_64-3.7/apex
creating build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/__init__.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
creating build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/__init__.py -> build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization
creating build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_initialize.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__version__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp
creating build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/__init__.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN
creating build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/__init__.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
creating build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/__init__.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/optimizers
creating build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/__init__.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/__init__.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
creating build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
running build_ext
building 'apex_C' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/csrc
gcc -pthread -B /home/amar/miniconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include/TH -I/home/amar/miniconda3/lib/python3.7/site-packages/torch/include/THC -I/home/amar/miniconda3/include/python3.7m -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /home/amar/miniconda3/compiler_compat -L/home/amar/miniconda3/lib -Wl,-rpath=/home/amar/miniconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
aalibey
All 6 comments
The reason is probably you are compiling apex with cuda 10.1, and your pytorch was compiled with cuda 10 (it says so in the warning). Cuda versions should match.
ngimel
on 30 Apr 2019
The reason is probably you are compiling apex with cuda 10.1, and your pytorch was compiled with cuda 10 (it says so in the warning). Cuda versions should match.
I downgraded nvcc to the corresponding version, And I still have the same problem.
I'm using arch linux, gcc 8 and cuda 10.
this is the output that I still get
torch.__version__ = 1.1.0.dev20190430
Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
from /opt/cuda/bin
Pytorch binaries were compiled with Cuda 10.0.130
running install
running bdist_egg
running egg_info
writing apex.egg-info/PKG-INFO
writing dependency_links to apex.egg-info/dependency_links.txt
writing top-level names to apex.egg-info/top_level.txt
reading manifest file 'apex.egg-info/SOURCES.txt'
writing manifest file 'apex.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/apex
copying apex/__init__.py -> build/lib.linux-x86_64-3.7/apex
creating build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/__init__.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
creating build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/__init__.py -> build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization
creating build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_initialize.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__version__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp
creating build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/__init__.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN
creating build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/__init__.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
creating build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/__init__.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/optimizers
creating build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/__init__.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/__init__.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
creating build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
running build_ext
building 'apex_C' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/csrc
gcc -pthread -B /home/amar/miniconda3/envs/pytorch-nighty/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/amar/miniconda3/envs/pytorch-nighty/lib/python3.7/site-packages/torch/include -I/home/amar/miniconda3/envs/pytorch-nighty/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/amar/miniconda3/envs/pytorch-nighty/lib/python3.7/site-packages/torch/include/TH -I/home/amar/miniconda3/envs/pytorch-nighty/lib/python3.7/site-packages/torch/include/THC -I/home/amar/miniconda3/envs/pytorch-nighty/include/python3.7m -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /home/amar/miniconda3/envs/pytorch-nighty/compiler_compat -L/home/amar/miniconda3/envs/pytorch-nighty/lib -Wl,-rpath=/home/amar/miniconda3/envs/pytorch-nighty/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so
/home/amar/miniconda3/envs/pytorch-nighty/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/envs/pytorch-nighty/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/envs/pytorch-nighty/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
/home/amar/miniconda3/envs/pytorch-nighty/compiler_compat/ld: build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: unable to initialize decompress status for section .debug_info
build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
Hmm it looks like this is not a Cuda error, but rather, a purely C++ error. It may be unrelated to Apex or even Pytorch for that matter: https://bbs.archlinux.org/viewtopic.php?id=242682. I'm not that familiar with arch linux unfortunately....
mcarilli
on 30 Apr 2019
Hmm it looks like this is not a Cuda error, but rather, a purely C++ error. It may be unrelated to Apex or even Pytorch for that matter: https://bbs.archlinux.org/viewtopic.php?id=242682. I'm not that familiar with arch linux unfortunately....
You're right, I had a problem with binutils, i downgraded from 2.32-1 tp 2.31.1-3
luckily Arch linux keeps versions in cache.
I've run this command to resolve the problem :
sudo pacman -U /var/cache/pacman/pkg/binutils-2.31.1-3-x86_64.pkg.tar.xz
aalibey
on 30 Apr 2019
👍2
Check this solution instead of downgrading ArchLinux packages:
https://github.com/mrjbq7/ta-lib/issues/260#issuecomment-550323380
oblitum
on 6 Nov 2019
mark
Single430
on 14 Jun 2020
👀1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
dxxz
·
3Comments
yangkky
·
4Comments
ccoulombe
·
3Comments
LightToYang
·
4Comments
TheRevanchist
·
3Comments
Most helpful comment
You're right, I had a problem with binutils, i downgraded from 2.32-1 tp 2.31.1-3
luckily Arch linux keeps versions in cache.
I've run this command to resolve the problem :
sudo pacman -U /var/cache/pacman/pkg/binutils-2.31.1-3-x86_64.pkg.tar.xz