Wav2letter: [CUDA required to build CUDA criterion backend]

Created on 28 Sep 2020  路  11Comments  路  Source: flashlight/wav2letter

Question

I get this error when trying to build the python bindings in Docker without CUDA.

Additional Context

RUN git clone -b v0.2 https://github.com/facebookresearch/wav2letter.git && \
    cd wav2letter/bindings/python && \
    pip3 install -e . && \
    cd ../../..

Full stack trace:

ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/app/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/app/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /app/wav2letter/bindings/python/
    Complete output (57 lines):
    running develop
    running egg_info
    writing wav2letter.egg-info/PKG-INFO
    writing dependency_links to wav2letter.egg-info/dependency_links.txt
    writing top-level names to wav2letter.egg-info/top_level.txt
    reading manifest file 'wav2letter.egg-info/SOURCES.txt'
    writing manifest file 'wav2letter.egg-info/SOURCES.txt'
    running build_ext
    -- The C compiler identification is GNU 8.3.0
    -- The CXX compiler identification is GNU 8.3.0
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    CMake Error at cmake/CUDAUtils.cmake:12 (message):
      CUDA required to build CUDA criterion backend
    Call Stack (most recent call first):
      src/libraries/criterion/CMakeLists.txt:28 (include)


    -- Configuring incomplete, errors occurred!
    See also "/app/wav2letter/bindings/python/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeOutput.log".
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/app/wav2letter/bindings/python/setup.py", line 109, in <module>
        zip_safe=False,
      File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/develop.py", line 140, in install_for_development
        self.run_command('build_ext')
      File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/app/wav2letter/bindings/python/setup.py", line 48, in run
        self.build_extensions()
      File "/app/wav2letter/bindings/python/setup.py", line 88, in build_extensions
        ["cmake", sourcedir] + cmake_args, cwd=self.build_temp, env=env
      File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '/app/wav2letter', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/app/wav2letter/bindings/python/wav2letter', '-DPYTHON_EXECUTABLE=/usr/local/bin/python', '-DW2L_BUILD_LIBRARIES_ONLY=ON', '-DW2L_BUILD_FOR_PYTHON=ON', '-DW2L_LIBRARIES_USE_CUDA=ON', '-DW2L_LIBRARIES_USE_KENLM=ON', '-DW2L_LIBRARIES_USE_MKL=OFF', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/app/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/app/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
WARNING: You are using pip version 19.3; however, version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
question
Source
picture loretoparisi

Most helpful comment

I encountered the same issue. I resolved it by running the following command before the pip install: export USE_CUDA=0
if you already have a build folder, delete it first

picture natspan on 29 Sep 2020
👍3

All 11 comments

I encountered the same issue. I resolved it by running the following command before the pip install: export USE_CUDA=0
if you already have a build folder, delete it first

natspan picture natspan on 29 Sep 2020
👍3

Yep, this is right, here are other env variables https://github.com/facebookresearch/wav2letter/blob/v0.2/bindings/python/setup.py#L14 which you can use to set configuration.

tlikhomanenko picture tlikhomanenko on 29 Sep 2020
👍1

Confirmed it works fine. When in docker please do

ENV USE_CUDA=0
loretoparisi picture loretoparisi on 30 Sep 2020

I want to use GPU via docker so have to put ENV USE_CUDA=1. How to solve this issue in such a case.
Log:
```
Cloning into 'wav2letter'...
Obtaining file:///app/wav2letter/bindings/python
Installing collected packages: wav2letter
Running setup.py develop for wav2letter
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/app/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/app/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
cwd: /app/wav2letter/bindings/python/
Complete output (57 lines):
running develop
running egg_info
writing wav2letter.egg-info/PKG-INFO
writing dependency_links to wav2letter.egg-info/dependency_links.txt
writing top-level names to wav2letter.egg-info/top_level.txt
reading manifest file 'wav2letter.egg-info/SOURCES.txt'
writing manifest file 'wav2letter.egg-info/SOURCES.txt'
running build_ext
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at cmake/CUDAUtils.cmake:12 (message):
CUDA required to build CUDA criterion backend
Call Stack (most recent call first):
src/libraries/criterion/CMakeLists.txt:28 (include)

-- Configuring incomplete, errors occurred!
See also "/app/wav2letter/bindings/python/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/app/wav2letter/bindings/python/setup.py", line 109, in <module>
    zip_safe=False,
  File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.7/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/usr/local/lib/python3.7/site-packages/setuptools/command/develop.py", line 140, in install_for_development
    self.run_command('build_ext')
  File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/app/wav2letter/bindings/python/setup.py", line 48, in run
    self.build_extensions()
  File "/app/wav2letter/bindings/python/setup.py", line 88, in build_extensions
    ["cmake", sourcedir] + cmake_args, cwd=self.build_temp, env=env
  File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/app/wav2letter', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/app/wav2letter/bindings/python/wav2letter', '-DPYTHON_EXECUTABLE=/usr/local/bin/python', '-DW2L_BUILD_LIBRARIES_ONLY=ON', '-DW2L_BUILD_FOR_PYTHON=ON', '-DW2L_LIBRARIES_USE_CUDA=ON', '-DW2L_LIBRARIES_USE_KENLM=ON', '-DW2L_LIBRARIES_USE_MKL=OFF', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
----------------------------------------

ERROR: Command errored out with exit status 1: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/app/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/app/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
WARNING: You are using pip version 19.3; however, version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c git clone -b v0.2 https://github.com/facebookresearch/wav2letter.git && cd wav2letter/bindings/python && pip3 install -e .' returned a non-zero code: 1

008karan picture 008karan on 21 Oct 2020

@008karan which docker image are you using? you need to use cuda docker (not cpu)

tlikhomanenko picture tlikhomanenko on 23 Oct 2020

I am pulling image FROM nvidia/cuda:10.2-base. But still nvidia-smi not working.
getting:

docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smidocker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver/library version mismatch\\\\n\\\"\"": unknown.

Any workaround for running gpu on docker? @tlikhomanenko

008karan picture 008karan on 23 Oct 2020

did you install nvidia docker?

tlikhomanenko picture tlikhomanenko on 27 Oct 2020

yes. I have.
When I do docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi
I get output. 

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.183      Driver Version: 384.183      CUDA Version: 9.0      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000001:00:00.0 Off |                    0 |
| N/A   47C    P0    71W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 00000002:00:00.0 Off |                    0 |
| N/A   60C    P0    60W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 00000003:00:00.0 Off |                    0 |
| N/A   55C    P0    56W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 00000004:00:00.0 Off |                    0 |
| N/A   43C    P0    70W / 149W |      0MiB / 11439MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

But when I do nvidia-smi inside container it says command not found.

008karan picture 008karan on 27 Oct 2020

Could you provide all commands/steps you are doing that you get "command not found"?

tlikhomanenko picture tlikhomanenko on 27 Oct 2020

So I am able to get nvidia-smi in container. But when I run my training script:

Traceback (most recent call last):
  File "train.py", line 14, in <module>
    cli_main()
  File "/app/fairseq/fairseq_cli/train.py", line 352, in cli_main
    distributed_utils.call_main(cfg, main)
  File "/app/fairseq/fairseq/distributed_utils.py", line 286, in call_main
    infer_init_method(cfg.distributed_training)
  File "/app/fairseq/fairseq/distributed_utils.py", line 149, in infer_init_method
    assert cfg.distributed_world_size <= torch.cuda.device_count()
AssertionError

torch.cuda.device_count() is giving zero. What can be reason for this. nvcc -V gives command not found.

command to run container:
docker run -d -it --rm -v /data/fairseq/new:/app/data --gpus all --name nv w2v

008karan picture 008karan on 27 Oct 2020

could you do export CUDA_VISIBLE_DEVICES=0 and try again?

tlikhomanenko picture tlikhomanenko on 28 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kamakshi-malhotra picture kamakshi-malhotra  路  5Comments
ghost picture ghost  路  5Comments
EdwinWenink picture EdwinWenink  路  4Comments
megharangaswamy picture megharangaswamy  路  5Comments
mlexplore1122 picture mlexplore1122  路  3Comments