Wav2letter: Error while installing building python bindings for wav2letter

[kamakshi-malhotra]

Bug Description

I am trying to install python bindings for wav2vec for wav2letter but I am facing below error.

Obtaining file:///home/kamakshi/app/wav2letter/bindings/python
Installing collected packages: wav2letter
Running setup.py develop for wav2letter
ERROR: Command errored out with exit status 1:
command: /home/kamakshi/env/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/kamakshi/app/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/home/kamakshi/app/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/kamakshi/app/wav2letter/bindings/python/
Complete output (67 lines):
running develop
running egg_info
writing wav2letter.egg-info/PKG-INFO
writing dependency_links to wav2letter.egg-info/dependency_links.txt
writing top-level names to wav2letter.egg-info/top_level.txt
reading manifest file 'wav2letter.egg-info/SOURCES.txt'
writing manifest file 'wav2letter.egg-info/SOURCES.txt'
running build_ext
[ 1%] Performing update step for 'CUB'
[ 1%] Performing update step for 'pybind11'
[ 2%] No configure step for 'pybind11'
[ 3%] No configure step for 'CUB'
[ 4%] No build step for 'pybind11'
[ 4%] No build step for 'CUB'
[ 5%] No install step for 'pybind11'
[ 6%] No install step for 'CUB'
[ 6%] Completed 'pybind11'
[ 6%] Completed 'CUB'
[ 7%] Built target pybind11
[ 9%] Built target CUB
[ 11%] Built target w2l-criterion-library-cuda
[ 12%] Building CXX object src/libraries/CMakeFiles/wav2letter-libraries.dir/feature/Derivatives.cpp.o
[ 12%] Building CXX object src/libraries/CMakeFiles/wav2letter-libraries.dir/lm/KenLM.cpp.o
[ 12%] Building CXX object src/libraries/CMakeFiles/wav2letter-libraries.dir/decoder/LexiconSeq2SeqDecoder.cpp.o
[ 13%] Building CXX object src/libraries/CMakeFiles/wav2letter-libraries.dir/feature/Dither.cpp.o
In file included from /home/kamakshi/app/wav2letter/src/libraries/lm/KenLM.cpp:9:
/home/kamakshi/app/wav2letter/src/libraries/lm/KenLM.h:14:10: fatal error: lm/model.hh: No such file or directory
#include
^~~~~
compilation terminated.
make[2]: * [src/libraries/CMakeFiles/wav2letter-libraries.dir/build.make:245: src/libraries/CMakeFiles/wav2letter-libraries.dir/lm/KenLM.cpp.o] Error 1
make[2]: Waiting for unfinished jobs....
In file included from /home/kamakshi/app/wav2letter/src/libraries/decoder/LexiconSeq2SeqDecoder.cpp:18:
/home/kamakshi/app/wav2letter/src/libraries/lm/KenLM.h:14:10: fatal error: lm/model.hh: No such file or directory
#include
^~~~~
compilation terminated.
make[2]: [src/libraries/CMakeFiles/wav2letter-libraries.dir/build.make:193: src/libraries/CMakeFiles/wav2letter-libraries.dir/decoder/LexiconSeq2SeqDecoder.cpp.o] Error 1
make[1]: [CMakeFiles/Makefile2:229: src/libraries/CMakeFiles/wav2letter-libraries.dir/all] Error 2
make: [Makefile:84: all] Error 2
Traceback (most recent call last):
File "", line 1, in
File "/home/kamakshi/app/wav2letter/bindings/python/setup.py", line 95, in
setup(
File "/home/kamakshi/env/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(attrs)
File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/kamakshi/env/lib/python3.8/site-packages/setuptools/command/develop.py", line 38, in run
self.install_for_development()
File "/home/kamakshi/env/lib/python3.8/site-packages/setuptools/command/develop.py", line 140, in install_for_development
self.run_command('build_ext')
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/kamakshi/app/wav2letter/bindings/python/setup.py", line 48, in run
self.build_extensions()
File "/home/kamakshi/app/wav2letter/bindings/python/setup.py", line 90, in build_extensions
subprocess.check_call(
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2.
----------------------------------------
ERROR: Command errored out with exit status 1: /home/kamakshi/env/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/kamakshi/app/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/home/kamakshi/app/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

Reproduction Steps

I have installed KenLM using following commands-

• git clone https://github.com/kpu/kenlm
• cd kenlm
• mkdir build
• cd build
• cmake ..
• make -j 4
• make install
  This finished successfully .
  After this I have installed OpenBLAS, FFTW3, libsndfile, gflags, glog using below command-
• sudo apt-get install libsndfile1-dev libopenblas-dev libfftw3-dev libgflags-dev libgoogle-glog-dev

When I try to build wav2letter using these commands then I get above error

• git clone -b v0.2 https://github.com/facebookresearch/wav2letter.git &&
• cd wav2letter/bindings/python &&
• pip3 install -e .

Platform and Hardware

OS- Ubuntu
GCC version - 8.4.0
CUDA version - 10.1.243
Directory structure - app->wav2letter,kenlm,fairseq

[tlikhomanenko]

Please see here example on docker https://github.com/facebookresearch/wav2letter/wiki/Building-Python-bindings,
you need to do export of kenlm path - path to the git repo where you run cmake and make (also install packaging and torch if you have no them pip install torch==1.2.0 packaging==19.1)

export KENLM_ROOT_DIR=/path/to/kenlm && \
cd /path/to/wav2letter/bindings/python && pip install -e .

[kamakshi-malhotra]

I have installed kenlm using following steps -

• wget -O - https://kheafield.com/code/kenlm.tar.gz |tar xz
• mkdir kenlm/build
• cd kenlm/build
• cmake ..
• make -j2
• export KENLM_ROOT_DIR= /home/speech_to_text/kenlm

My main directory is speech_to_text inside which I have installed kenlm. Also I am building python bindings for wav2letter inside this directory. But in error it is shown that it is not able to find LM in kenlm. Can you please let me know if the structure of directory while installing is correct?

Error shown ->
RROR: Command errored out with exit status 1:
command: /home/kamakshi/fair_env/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/kamakshi/speech_to_text/wav2letter/bindings/python/
Complete output (94 lines):
running develop
running egg_info
writing wav2letter.egg-info/PKG-INFO
writing dependency_links to wav2letter.egg-info/dependency_links.txt
writing top-level names to wav2letter.egg-info/top_level.txt
reading manifest file 'wav2letter.egg-info/SOURCES.txt'
writing manifest file 'wav2letter.egg-info/SOURCES.txt'
running build_ext
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- CUDA found (library: /usr/lib/x86_64-linux-gnu/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/include)
-- CUDA architecture flags: -gencodearch=compute_30,code=sm_30-gencodearch=compute_35,code=sm_35-gencodearch=compute_50,code=sm_50-gencodearch=compute_52,code=sm_52-gencodearch=compute_60,code=sm_60-gencodearch=compute_61,code=sm_61-gencodearch=compute_70,code=sm_70-gencodearch=compute_75,code=sm_75-gencodearch=compute_70,code=compute_70-gencodearch=compute_75,code=compute_75
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'cblas'
-- No package 'cblas' found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Checking for [Accelerate]
-- Checking for [vecLib]
-- Checking for [cblas - atlas]
-- Includes found
-- Looking for cblas_dgemm
-- Looking for cblas_dgemm - found
-- CBLAS Symbols FOUND
-- CBLAS library found
-- CBLAS found (include: /usr/include/x86_64-linux-gnu, library: /usr/lib/x86_64-linux-gnu/libcblas.so;/usr/lib/x86_64-linux-gnu/libatlas.so)
-- Found FFTW: /usr/include
-- FFTW found
-- Looking for KenLM
-- Using kenlm library found in /usr/local/lib/libkenlm.a
-- Using kenlm utils library found in /usr/local/lib/libkenlm.a
CMake Error at cmake/Findkenlm.cmake:79 (message):
kenlm lm/model.hh not found; please set CMAKE_INCLUDE_PATH or KENLM_INC
Call Stack (most recent call first):
src/libraries/lm/CMakeLists.txt:11 (find_package)

-- Configuring incomplete, errors occurred!
See also "/home/kamakshi/speech_to_text/wav2letter/bindings/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
See also "/home/kamakshi/speech_to_text/wav2letter/bindings/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py", line 95, in <module>
    setup(
  File "/home/kamakshi/fair_env/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/kamakshi/fair_env/lib/python3.8/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/home/kamakshi/fair_env/lib/python3.8/site-packages/setuptools/command/develop.py", line 140, in install_for_development
    self.run_command('build_ext')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py", line 48, in run
    self.build_extensions()
  File "/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py", line 87, in build_extensions
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/kamakshi/speech_to_text/wav2letter', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/kamakshi/speech_to_text/wav2letter/bindings/python/wav2letter', '-DPYTHON_EXECUTABLE=/home/kamakshi/fair_env/bin/python3', '-DW2L_BUILD_LIBRARIES_ONLY=ON', '-DW2L_BUILD_FOR_PYTHON=ON', '-DW2L_LIBRARIES_USE_CUDA=ON', '-DW2L_LIBRARIES_USE_KENLM=ON', '-DW2L_LIBRARIES_USE_MKL=OFF', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
----------------------------------------

ERROR: Command errored out with exit status 1: /home/kamakshi/fair_env/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py'"'"'; __file__='"'"'/home/kamakshi/speech_to_text/wav2letter/bindings/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

I am using torch 1.6.0 version, packaging 19.1 version, cuda version 10.1, gcc version 8.4.

[tlikhomanenko]

First, could you try to remove build folder inside wav2letter/bindings/python do export of kenlm path as you are doing now and run pip install -e . again? Does the error still occur?

[tlikhomanenko]

Also could you show me your output on echo $KENLM_ROOT_DIR?

[kamakshi-malhotra]

I was able to install wav2letter correctly. There was issue with path of kenlm. Thanks for your help!