Wav2letter: Decoder Test Failed

[luozhiping]

./build/Decoder --flagsfile tutorials/1-librispeech_clean/decode.cfg

#

decode.cfg:
--lexicon=/mnt/speech/libri/data/lm/lexicon.txt
--lm=/mnt/speech/libri/data/lm/3-gram.arpa
--am=/root/speech/wav2letter/librispeech_clean_trainlogs/001_model_lists#dev-clean.lst.bin
--test=lists/test-clean.lst
--sclite=/root/speech/wav2letter
--lmweight=2.5
--wordscore=1
--beamsize=500
--beamthreshold=25
--silweight=-0.5
--nthread_decoder=4
--smearing=max
--show=true

#

mylog:
Loading the LM will be faster if you build a binary file.
Reading /mnt/speech/libri/data/lm/3-gram.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

* Error in `./build/Decoder': malloc(): memory corruption: 0x00000000154f5940 *
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f7bc27c67e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7f7bc27d113e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f7bc27d3184]
/root/speech/arrayfire/lib64/libafcuda.so.3(_Znwm+0x18)[0x7f7be1722368]
./build/Decoder[0x538e88]
./build/Decoder[0x41ae4e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f7bc276f830]
./build/Decoder[0x47b469]
======= Memory map: ========
00400000-0092f000 r-xp 00000000 fd:01 1977682 /root/speech/wav2letter/build/Decoder
00b2e000-00b32000 r--p 0052e000 fd:01 1977682 /root/speech/wav2letter/build/Decoder
00b32000-00b34000 rw-p 00532000 fd:01 1977682 /root/speech/wav2letter/build/Decoder
00b34000-00b3e000 rw-p 00000000 00:00 0
01e17000-423c0000 rw-p 00000000 00:00 0 [heap]
200000000-200100000 rw-s 00000000 00:06 405 /dev/nvidiactl
200100000-200300000 rw-s 00000000 00:06 405 /dev/nvidiactl
200300000-202b00000 rw-s 00000000 00:06 405 /dev/nvidiactl
202b00000-202c00000 rw-s 00000000 00:06 405 /dev/nvidiactl

[tlikhomanenko]

The same issue as https://github.com/facebookresearch/wav2letter/issues/404

Hi @luozhiping,

At which commit are you running the test? In the docker image this test is working fine for me, also at at CircleCI all test passed.

[luozhiping]

The same issue as #404

Hi @luozhiping,

At which commit are you running the test? In the docker image this test is working fine for me, also at at CircleCI all test passed.

HI @tlikhomanenko
I use newest commit:cc5ca00 (Commits on Sep 14, 2019)
https://github.com/facebookresearch/wav2letter/commit/cc5ca003735b2a8138c85d9039c9320387ccbeff
I build wav2letter on my ubuntu 16.04, cuda9.2, cudnn7.1.4, dirver418.67
mygpu is Tesla M40

[luozhiping]

When I run python demo, this error happen again @tlikhomanenko

~/speech/wav2letter/bindings/python# python examples/decoder_example.py ../../src/decoder/test
Loading the LM will be faster if you build a binary file.
Reading ../../src/decoder/test/lm.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

* Error in `python': malloc(): memory corruption: 0x000056293fea04d0 *
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f2b1163e7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7f2b1164913e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f2b1164b184]
/root/anaconda3/lib/libstdc++.so.6(_Znwm+0x16)[0x7f2b006d7084]
/root/speech/wav2letter/bindings/python/wav2letter/_decoder.cpython-37m-x86_64-linux-gnu.so(+0x37bde)[0x7f2ab077abde]
/root/speech/wav2letter/bindings/python/wav2letter/_decoder.cpython-37m-x86_64-linux-gnu.so(+0x2a4c9)[0x7f2ab076d4c9]
/root/speech/wav2letter/bindings/python/wav2letter/_decoder.cpython-37m-x86_64-linux-gnu.so(+0x3e742)[0x7f2ab0781742]
/root/speech/wav2letter/bindings/python/wav2letter/_decoder.cpython-37m-x86_64-linux-gnu.so(+0x27579)[0x7f2ab076a579]
python(_PyMethodDef_RawFastCallKeywords+0x264)[0x56293e5c6ea4]
python(_PyCFunction_FastCallKeywords+0x21)[0x56293e5c6fc1]
python(_PyEval_EvalFrameDefault+0x523e)[0x56293e63c2ce]
python(_PyEval_EvalCodeWithName+0x2e8)[0x56293e575528]
python(PyEval_EvalCodeEx+0x44)[0x56293e5763a4]
python(PyEval_EvalCode+0x1c)[0x56293e5763cc]
python(+0x22d304)[0x56293e690304]
python(PyRun_FileExFlags+0xa1)[0x56293e698611]
python(PyRun_SimpleFileExFlags+0x1c4)[0x56293e698804]
python(+0x23717d)[0x56293e69a17d]
python(_Py_UnixMain+0x80)[0x56293e69a3f0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f2b115e7830]
python(+0x1e3e32)[0x56293e646e32]

[tlikhomanenko]

@jacobkahn, @vineelpratap, @xuqiantong any ideas? I suspect that KenLM is not correctly installed (the model is very small, so memory issue cannot be connected with this).

[samuallin]

I met the same issue with ubuntu 16.04, cuda9.2, cudnn7.6.4, dirver410.78, gpu is GeForce GTX 1060
decoder.cfg is almost the same as in /wav2letter/tutorials/1-librispeech_clean/decode.cfg only the path configurations different
Logs:
Reading /home/ubuntu/w2l/lm/3-gram.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

* Error in `/home/ubuntu/wav2letter/build/Decoder': malloc(): memory corruption: 0x000000001cb36af0 *
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f59475837e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7f594758e13e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f5947590184]
/usr/lib/nvidia-410/tls/libnvidia-tls.so.410.78(+0x24c0)[0x7f592ab194c0]

[lingss0918]

Same error here on ubuntu 16.04, cuda10, Tesla V100.

[hiroaki-ogawa]

Hi @tlikhomanenko ,
This is fixed by adding "-DKENLM_MAX_ORDER=20" to argument list of the cmake in "wav2letter/bindings/python/setup.py.

KenLM with KENLM_MAX_ORDER=20 wrote 156 byte lm::ngram::state onto 44 byte lm::ngram::state of the python binding.

[tlikhomanenko]

@hiroaki-ogawa,

This is strange, because kenlm then should be built also with -DKENLM_MAX_ORDER=20 (so there should be consistent flags for w2l and kenlm.

[hiroaki-ogawa]

Hi @tlikhomanenko ,

This -DKENLM_MAX_ORDER=20 is needed for bulding the python binding code.

The lm::ngram::state is defined in state.hh that is shared between kenlm and the python binding code. The lm::ngram::state changes its size depending on KENLM_MAX_ORDER.

The lm::ngram::state in the library built in kenlm regards KENLM_MAX_ORDER as 20 because kenlm library is built with -DKENLM_MAX_ORDER=20. But lm::ngram::state in python binding code use KENLM_MAX_ORDER as 4 without the option.

[tlikhomanenko]

Hi @hiroaki-ogawa,

In all three places the flags should be consistent: in kenlm, w2l, w2l python bindings. That what I meant in my previous comment.

Regarding the error: I cannot reproduce it on my local machine and in the docker image everything works fine too.

[hiroaki-ogawa]

Hi @tlikhomanenko ,

I see what you ment.

Regarding the error: I cannot reproduce it on my local machine and in the docker image everything works fine too.

Hmm, this error should be happen in your environment too because bindings/python/setup.py does not include -DKENLM_MAX_ORDER=20. And generated g++ command line does not include the option too in my environment.

Do you set environment variable that change build behavior (e.g. CXX="gcc -DKENLM_MAX_ORDER=20")?
Or, to reproduce the error, how about removing bindings/python/build and rebuild the python binding code?

[tlikhomanenko]

Hi @hiroaki-ogawa,

Again I am building kenlm without DKENLM_MAX_ORDER=20 https://github.com/facebookresearch/wav2letter/blob/master/Dockerfile-CUDA-Base#L85 and w2l also without DKENLM_MAX_ORDER=20 https://github.com/facebookresearch/wav2letter/blob/master/Dockerfile-CUDA#L24. So python bindings work ok for this setup https://github.com/facebookresearch/wav2letter/blob/master/Dockerfile-CUDA#L29.

Do you mean that this issue related to the case when kenlm and w2l are built with DKENLM_MAX_ORDER=20 and setup.py doesn't contain this flag?

And generated g++ command line does not include the option too in my environment.

Yep, this is true, so you can try just set export KENLM_MAX_ORDER=20 before building w2l and running setup.py.

Or, to reproduce the error, how about removing bindings/python/build and rebuild the python binding code?

Will try, but again my question is with witch config of KENLM_MAX_ORDER did you build kenlm?

[hiroaki-ogawa]

Hi @tlikhomanenko

Thank you for your reply!

Yep, this is true, so you can try just set export KENLM_MAX_ORDER=20 before building w2l and running setup.py.

Oh, this is the one that I missed! And it can be so for other people having this problem.
The docs/installation.md only said about MKLROOTand KENLM_ROOT_DIR so I didn't set "KENLM_MAX_ORDER" as an environment variable. (I should realize this, though..)

Thank you again!

[adamchant]

Hey, I am getting the same issue on local build of wav2letter...
Every test case passes except for the DecoderTest
11/22 Test #11: DecoderTest ......................*Exception: SegFault 0.21 sec
95% tests passed, 1 tests failed out of 22

Total Test time (real) = 64.52 sec

The following tests FAILED:
11 - DecoderTest (SEGFAULT)
Errors while running CTest
Makefile:118: recipe for target 'test' failed
make: * [test] Error 8

and running the DecoderTest binary gives :
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DecoderTest
[ RUN ] DecoderTest.run
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0404 20:22:56.088800 7221 DecoderTest.cpp:90] [Serialization] Loaded emissions [235 x 29]
I0404 20:22:56.128610 7221 DecoderTest.cpp:98] [Dictionary] Number of words: 26710
Loading the LM will be faster if you build a binary file.
Reading /home/developer2/asr_toolkits/wav2letter/src/decoder/test/lm.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

I0404 20:22:56.165189 7221 DecoderTest.cpp:103] [Decoder] LM constructed.
corrupted size vs. prev_size
Aborted (core dumped)
I have tried the above solutions, rebuilding w2l with export KENLM_MAX_ORDER=20, but the issue still remains, can someone help me resolve this issue.. thanks

[tlikhomanenko]

Hi @adamchant, could you provide your system info, commit of w2l at which you are running and the way you installed (docker image or locally on the machine)?