Deepspeech: Import Error in Native Client

[ghost]

I'm trying to install the Python bindings for the native client. After doing all the steps in the native client readme I get the following error when importing the deepspeech module:

$ python client.py

Traceback (most recent call last):
File "client.py", line 3, in
from deepspeech import DeepSpeech
File "/usr/local/lib/python2.7/dist-packages/deepspeech/__init__.py", line 1, in
from .deepspeech import *
File "/usr/local/lib/python2.7/dist-packages/deepspeech/deepspeech.py", line 28, in
_deepspeech = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/deepspeech/deepspeech.py", line 20, in swig_import_helper
import _deepspeech
ImportError: /usr/local/lib/libdeepspeech.so: undefined symbol: _ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE

OS: Ubuntu 14.04.5 LTS
TensorFlow : v1.0.1 tag

To quickly test out how the native client works I created a model with ./bin/run-ldc93s1.sh

If I run the client with the Makefile everything works as expected. My first issue, let me know if I should expand on anything :)

[Cwiiis]

Looks like it either can't find libtensorflow.so or it linked against a different version of libtensorflow.so - did you run the install target before building and installing the bindings?

[ghost]

Yes, here is the console output using the instructions in the Python bindings section:

~/src/DeepSpeech/native_client$ make clean
rm -rf build dist deepspeech.egg-info
rm -f deepspeech setup.cfg python/deepspeech_wrap.cpp python/deepspeech.py

~/src/DeepSpeech/native_client$ make deepspeech
g++ -o deepspeech pkg-config --cflags sox -L/home/ubuntu/src/tensorflow/bazel-bin/tensorflow -L/home/ubuntu/src/tensorflow/bazel-bin/native_client client.cc pkg-config --libs sox -ltensorflow -ldeepspeech

~/src/DeepSpeech/native_client$ PREFIX=/usr/local sudo make install
install -d /usr/local/lib
install -s -m 0644 ../../tensorflow/bazel-bin/tensorflow/libtensorflow.so /usr/local/lib/
install -s -m 0644 ../../tensorflow/bazel-bin/native_client/libdeepspeech.so /usr/local/lib/
install -d /usr/local/bin
install -m 0755 deepspeech /usr/local/bin/

~/src/DeepSpeech/native_client$ make bindings
sed -e 's:@LIBDIRS@:/home/ubuntu/src/tensorflow/bazel-bin/tensorflow\:/home/ubuntu/src/tensorflow/bazel-bin/native_client:g' setup.cfg.in > setup.cfg
python ./setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
building '_deepspeech' extension
swigging python/deepspeech.i to python/deepspeech_wrap.cpp
swig -python -c++ -keyword -o python/deepspeech_wrap.cpp python/deepspeech.i
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/python
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I./ -I/usr/include/python2.7 -c python/deepspeech_wrap.cpp -o build/temp.linux-x86_64-2.7/python/deepspeech_wrap.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
creating build/lib.linux-x86_64-2.7
c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/python/deepspeech_wrap.o -L/home/ubuntu/src/tensorflow/bazel-bin/tensorflow -L/home/ubuntu/src/tensorflow/bazel-bin/native_client -ltensorflow -ldeepspeech -o build/lib.linux-x86_64-2.7/_deepspeech.so
running build_py
creating build/lib.linux-x86_64-2.7/deepspeech
copying python/__init__.py -> build/lib.linux-x86_64-2.7/deepspeech
copying python/deepspeech.py -> build/lib.linux-x86_64-2.7/deepspeech
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
copying build/lib.linux-x86_64-2.7/_deepspeech.so -> build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/deepspeech
copying build/lib.linux-x86_64-2.7/deepspeech/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeech
copying build/lib.linux-x86_64-2.7/deepspeech/deepspeech.py -> build/bdist.linux-x86_64/wheel/deepspeech
running install_egg_info
running egg_info
creating deepspeech.egg-info
writing deepspeech.egg-info/PKG-INFO
writing top-level names to deepspeech.egg-info/top_level.txt
writing dependency_links to deepspeech.egg-info/dependency_links.txt
writing manifest file 'deepspeech.egg-info/SOURCES.txt'
reading manifest file 'deepspeech.egg-info/SOURCES.txt'
writing manifest file 'deepspeech.egg-info/SOURCES.txt'
Copying deepspeech.egg-info to build/bdist.linux-x86_64/wheel/deepspeech-0.0.1-py2.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/deepspeech-0.0.1.dist-info/WHEEL

~/src/DeepSpeech/native_client$ sudo pip install dist/deepspeech*
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing ./dist/deepspeech-0.0.1-cp27-cp27mu-linux_x86_64.whl
Installing collected packages: deepspeech
Successfully installed deepspeech-0.0.1

~/src/DeepSpeech/native_client$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from deepspeech import DeepSpeech
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/deepspeech/__init__.py", line 1, in
from .deepspeech import *
File "/usr/local/lib/python2.7/dist-packages/deepspeech/deepspeech.py", line 28, in
_deepspeech = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/deepspeech/deepspeech.py", line 20, in swig_import_helper
import _deepspeech
ImportError: /usr/local/lib/libdeepspeech.so: undefined symbol: _ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE

I built the TensorFlow and DeepSpeech libraries using the following command:
bazel build -c opt --copt=-march=native --copt=-mtune=native --copt=-O3 //tensorflow:libtensorflow.so //native_client:deepspeech

$TFDIR is set to /home/ubuntu/src/tensorflow

[Cwiiis]

Is /usr/local/lib in your LD_LIBRARY_PATH?

[ghost]

Yes

~$ echo $LD_LIBRARY_PATH
/home/ubuntu/src/torch/install/lib:/home/ubuntu/src/torch/install/lib:/home/ubuntu/src/cntk/bindings/python/cntk/libs:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/home/ubuntu/src/mxnet/mklml_lnx_2017.0.1.20161005/lib:

[Cwiiis]

I had a similar issue on TaskCluster where the Makefile's environment was polluting setup.py's... As well as updating master, I'd recommend trying using setup.py directly rather than the Makefile, so something like:

sudo pip uninstall deepspeech
make clean
make setup.cfg
python ./setup.py bdist_wheel

And then try reinstalling the newly created .whl and trying again. Before that, I'd also check all the obvious things too, like making sure both libtensorflow.so and libdeepspeech.so are present in /usr/local/lib, and that if you run the client binary without the Makefile that it works without error (the Makefile sets up the environment to use the libraries from the TensorFlow directory, but this should be unnecessary if they're installed correctly in the system path).

[basant-kumar]

getting the same kind of error, tried all above solutions, but didn't work.

Error
$ python2 -c "import deepspeech"
or
$ python client.py

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/deepspeech/__init__.py", line 1, in
from .deepspeech import *
File "/usr/local/lib/python2.7/dist-packages/deepspeech/deepspeech.py", line 28, in
_deepspeech = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/deepspeech/deepspeech.py", line 20, in swig_import_helper
import _deepspeech
ImportError: /usr/local/lib/libdeepspeech.so: undefined symbol: _ZN10tensorflow6TensorD1Ev

OS: Ubuntu 16.04
Tensorflow: r1.0

$ echo $LD_LIBRARY_PATH

/usr/local/lib:/home/bkm/ai_ml/tools/opencv-.2.0/build/lib:/home/bkm/ai_ml/tools/dependencies/glog/build:/home/bkm/ai_ml/tools/dependencies/gflags/build/lib:/opt/intel/ipp/lib/intel64:/opt/intel/lib/ia32:/opt/intel/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin:/usr/include/linux:/usr/include:/usr/lib/x86_64-linux-gnu:/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/targets/x86_64-linux/lib:

P.S. inferencing works with make run command

[Cwiiis]

I've come across similar errors only when I've had a version mismatch somewhere. Likelihood is a different Tensorflow library to that which you compiled deepspeech against is being found by Python and there's an ABI mismatch. I would suggest that the easiest way to fix this is to make sure TensorFlow is completely uninstalled, the TensorFlow check-out is clean and rebuilding and reinstalling from scratch. If there's a system TensorFlow installed (e.g. from the TensorFlow python package), it may also be worth uninstalling that and rebuilding the Python package from the same checkout following the instructions on the TensorFlow website.

[lissyx]

@basant-kumar It would help to have the overall errors reported. I am inclined to think this is just fallout from what was fixed by https://github.com/mozilla/DeepSpeech/pull/657

[kdavis-mozilla]

We've made related fixes. If this occurs again, please reopen,

[basant-kumar]

hi, @lissyx @kdavis-mozilla , now deepspeech can be imported easily but while importing deepspeech.model, I'm getting same kind of error as shown below

$ python client.py
Traceback (most recent call last):
File "client.py", line 3, in
from deepspeech.model import Model
File "/usr/local/lib/python2.7/dist-packages/deepspeech/model.py", line 28, in
_model = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/deepspeech/model.py", line 20, in swig_import_helper
import _model
ImportError: /usr/local/lib/libdeepspeech.so: undefined symbol: _ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE

some useful outputs

$ nm /usr/local/lib/libdeepspeech.so | grep ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE
U _ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE

$ echo $LD_LIBRARY_PATH
/usr/local/lib:/home/inkers/basant/DeepSpeech/prefix/lib:/home/inkers/Lovekesh/ROS/gym-gazebo/gym_gazebo/envs/installation/catkin_ws/devel/lib:/opt/ros/kinetic/lib:/opt/ros/kinetic/lib/x86_64-linux-gnu:/opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64_lin/:/home/inkers/ai_ml/tools/caffe/build/lib/

Console output of python bindings steps

$ make deepspeech
g++ -o deepspeech pkg-config --cflags sox -L../../tensorflow/bazel-bin/tensorflow -L../../tensorflow/bazel-bin/native_client client.cc pkg-config --libs sox -ltensorflow_cc -ldeepspeech -ldeepspeech_utils

$ sudo make install
install -d /usr/local/lib
install -m 0644 ../../tensorflow/bazel-bin/tensorflow/libtensorflow_cc.so /usr/local/lib/
install -m 0644 ../../tensorflow/bazel-bin/native_client/libdeepspeech.so /usr/local/lib/
install -m 0644 ../../tensorflow/bazel-bin/native_client/libdeepspeech_utils.so /usr/local/lib/
install -d /usr/local/bin
install -m 0755 deepspeech /usr/local/bin/

$ make bindings
python ./setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
building '_model' extension
swigging python/model.i to python/model_wrap.cpp
swig -python -c++ -keyword -o python/model_wrap.cpp python/model.i
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/python
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I./ -I/usr/include/python2.7 -c python/model_wrap.cpp -o build/temp.linux-x86_64-2.7/python/model_wrap.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating build/lib.linux-x86_64-2.7
c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/python/model_wrap.o -ltensorflow_cc -ldeepspeech -ldeepspeech_utils -o build/lib.linux-x86_64-2.7/_model.so
building '_utils' extension
swigging python/utils.i to python/utils_wrap.cpp
swig -python -c++ -keyword -o python/utils_wrap.cpp python/utils.i
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I./ -I/usr/include/python2.7 -c python/utils_wrap.cpp -o build/temp.linux-x86_64-2.7/python/utils_wrap.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/python/utils_wrap.o -ldeepspeech_utils -o build/lib.linux-x86_64-2.7/_utils.so
running build_py
creating build/lib.linux-x86_64-2.7/deepspeech
copying python/model.py -> build/lib.linux-x86_64-2.7/deepspeech
copying python/utils.py -> build/lib.linux-x86_64-2.7/deepspeech
copying python/__init__.py -> build/lib.linux-x86_64-2.7/deepspeech
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
copying build/lib.linux-x86_64-2.7/_model.so -> build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/deepspeech
copying build/lib.linux-x86_64-2.7/deepspeech/model.py -> build/bdist.linux-x86_64/wheel/deepspeech
copying build/lib.linux-x86_64-2.7/deepspeech/utils.py -> build/bdist.linux-x86_64/wheel/deepspeech
copying build/lib.linux-x86_64-2.7/deepspeech/__init__.py -> build/bdist.linux-x86_64/wheel/deepspeech
copying build/lib.linux-x86_64-2.7/_utils.so -> build/bdist.linux-x86_64/wheel
running install_egg_info
running egg_info
creating deepspeech.egg-info
writing deepspeech.egg-info/PKG-INFO
writing top-level names to deepspeech.egg-info/top_level.txt
writing dependency_links to deepspeech.egg-info/dependency_links.txt
writing manifest file 'deepspeech.egg-info/SOURCES.txt'
reading manifest file 'deepspeech.egg-info/SOURCES.txt'
writing manifest file 'deepspeech.egg-info/SOURCES.txt'
Copying deepspeech.egg-info to build/bdist.linux-x86_64/wheel/deepspeech-0.0.1-py2.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/deepspeech-0.0.1.dist-info/WHEEL

$ sudo -H pip install dist/deepspeech-0.0.1-cp27-cp27mu-linux_x86_64.whl
Processing ./dist/deepspeech-0.0.1-cp27-cp27mu-linux_x86_64.whl
Installing collected packages: deepspeech
Successfully installed deepspeech-0.0.1

$ python client.py
Traceback (most recent call last):
File "client.py", line 3, in
from deepspeech.model import Model
File "/usr/local/lib/python2.7/dist-packages/deepspeech/model.py", line 28, in
_model = swig_import_helper()
File "/usr/local/lib/python2.7/dist-packages/deepspeech/model.py", line 20, in swig_import_helper
import _model
ImportError: /usr/local/lib/libdeepspeech.so: undefined symbol: _ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE

PS: same error occurs when using prebuild wheel and libraries from git repo.

[gardenia22]

657 didn't fix this issue, same error still remains on my server.

[lissyx]

Yes, it looks like I do reproduce that issue:

$ objdump -tTC ./libdeepspeech.so |grep CheckTypeAndIsAligned
0000000000000000         *UND*  0000000000000000              tensorflow::Tensor::CheckTypeAndIsAligned(tensorflow::DataType) const
0000000000000000      D  *UND*  0000000000000000              tensorflow::Tensor::CheckTypeAndIsAligned(tensorflow::DataType) const
$ objdump -tTC ./libtensorflow_cc.so |grep CheckTypeAndIsAligned
0000000003479b60 g     F .text  00000000000000d1              tensorflow::Tensor::CheckTypeAndIsAligned(tensorflow::DataType) const
0000000003479b60 g    DF .text  00000000000000d1  Base        tensorflow::Tensor::CheckTypeAndIsAligned(tensorflow::DataType) const

And yet,

ImportError: ./libdeepspeech.so: undefined symbol: _ZNK10tensorflow6Tensor21CheckTypeAndIsAlignedENS_8DataTypeE

[lissyx]

@basant-kumar @gardenia22 Can you test PR https://github.com/mozilla/DeepSpeech/pull/676 ? It fixes the issue locally for me.

[basant-kumar]

Thanks @lissyx , yes, this(676) fixed it.
appreciate it :+1:

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.