Wav2letter: Inference failed with long audio

[hieuhv94]

Bug Description

When i inference by any type of inference (simple, multithread or interactive) with a long audio (30 minutes), inference work fine in 15 first minutes after that output is empty:
773000,774000,làm gì
774000,775000,
775000,776000,nhà tao biệt
776000,777000,lập ở xóm trại
777000,778000,đây
778000,779000,bao năm chẳng
779000,780000,làm ruộng nên trắng
780000,781000,biết làm gì
781000,782000,ngoài
782000,783000,làm cái gì cá ba
783000,784000,
784000,785000,xã
785000,786000,không bắt sao
786000,787000,
787000,788000,gì
788000,789000,nhưng bây
789000,790000,giờ dân ít
790000,791000,chơi rồi
791000,792000,chỉ thỉnh thoảng
792000,793000,vào dịch lễ
793000,794000,tết thôi
794000,795000,nên cũng
795000,796000,chẳng
796000,797000,
797000,798000,
798000,799000,
799000,800000,
800000,801000,
801000,802000,
802000,803000,
803000,804000,
804000,805000,
805000,806000,
806000,807000,
807000,808000,
808000,809000,
809000,810000,
810000,811000,
811000,812000,
812000,813000,
813000,814000,
814000,815000,
815000,816000,
816000,817000,
817000,818000,
818000,819000,
819000,820000,
820000,821000,
821000,822000,
822000,823000,
823000,824000,
824000,825000,
825000,826000,
826000,827000,
827000,828000,
828000,829000,
829000,830000,
830000,831000,
831000,832000,
832000,833000,
833000,834000,
834000,835000,
835000,836000,
836000,837000,
837000,838000,
838000,839000,
839000,840000,
840000,841000,
841000,842000,
842000,843000,
843000,844000,
844000,845000,

Anyone have same problems, and how to fix it?
Thank you

[abhinavkulkarni]

@hieuhv94: I had the same problem a while ago and I am not quite sure how I fixed it (or whether I fixed it), but are you reading the file while it is being written to (although even this shouldn't cause any problems)?

[hieuhv94]

@abhinavkulkarni Thank for your reply but i sure that i don't read file while it is being written, i recorded it before decoding.
If you remember how you fix it, please tell me, or I'll try fix it by myself :)))
Thanks!

[abhinavkulkarni]

@hieuhv94: Sorry, I meant the transcription file (rather than the audio file). But yeah, that's unlikely to be the cause behind the missing transcription.

Can you please verify that your audio is either wave or flac format, 16kz frequency, 16-bit depth integers and monochannel?

Thanks.

[hieuhv94]

@hieuhv94: Sorry, I meant the transcription file (rather than the audio file). But yeah, that's unlikely to be the cause behind the missing transcription.

Can you please verify that your audio is either wave or flac format, 16kz frequency, 16-bit depth integers and monochannel?

Thanks.

Of course, parameters of wave file:
Sample rate: 16kHz
Bitrate: 256kbps => 16-bit depth intergers
Channels: mono

[hieuhv94]

@hieuhv94: Sorry, I meant the transcription file (rather than the audio file). But yeah, that's unlikely to be the cause behind the missing transcription.

Can you please verify that your audio is either wave or flac format, 16kz frequency, 16-bit depth integers and monochannel?

Thanks.

And i print output to console not a transcripts file

[tlikhomanenko]

cc @vineelpratap @xuqiantong

[hieuhv94]

@vineelpratap , @xuqiantong Have you any ideal?

[mlexplore1122]

hi all,
does we have any update for this issues, I have the same problem? :((
I think problem from score beam when decode, because i try case with lmweight=0 and wordscore=0, streaming work normal with long audio, but when try set lmweight=0.7 and wordscore=0.8, streaming when to some chunk has no get any output.
Any idea?

[mlexplore1122]

hi hieuhv94,
can you confirm same experiment with your audio?

Bug Description

When i inference by any type of inference (simple, multithread or interactive) with a long audio (30 minutes), inference work fine in 15 first minutes after that output is empty:
773000,774000,làm gì
774000,775000,
775000,776000,nhà tao biệt
776000,777000,lập ở xóm trại
777000,778000,đây
778000,779000,bao năm chẳng
779000,780000,làm ruộng nên trắng
780000,781000,biết làm gì
781000,782000,ngoài
782000,783000,làm cái gì cá ba
783000,784000,
784000,785000,xã
785000,786000,không bắt sao
786000,787000,
787000,788000,gì
788000,789000,nhưng bây
789000,790000,giờ dân ít
790000,791000,chơi rồi
791000,792000,chỉ thỉnh thoảng
792000,793000,vào dịch lễ
793000,794000,tết thôi
794000,795000,nên cũng
795000,796000,chẳng
796000,797000,
797000,798000,
798000,799000,
799000,800000,
800000,801000,
801000,802000,
802000,803000,
803000,804000,
804000,805000,
805000,806000,
806000,807000,
807000,808000,
808000,809000,
809000,810000,
810000,811000,
811000,812000,
812000,813000,
813000,814000,
814000,815000,
815000,816000,
816000,817000,
817000,818000,
818000,819000,
819000,820000,
820000,821000,
821000,822000,
822000,823000,
823000,824000,
824000,825000,
825000,826000,
826000,827000,
827000,828000,
828000,829000,
829000,830000,
830000,831000,
831000,832000,
832000,833000,
833000,834000,
834000,835000,
835000,836000,
836000,837000,
837000,838000,
838000,839000,
839000,840000,
840000,841000,
841000,842000,
842000,843000,
843000,844000,
844000,845000,

Anyone have same problems, and how to fix it?
Thank you

[hieuhv94]

Hi @mlexplore1122
I tried the same experiemnt

hi hieuhv94,
can you confirm same experiment with your audio?

Bug Description

When i inference by any type of inference (simple, multithread or interactive) with a long audio (30 minutes), inference work fine in 15 first minutes after that output is empty:
773000,774000,làm gì
774000,775000,
775000,776000,nhà tao biệt
776000,777000,lập ở xóm trại
777000,778000,đây
778000,779000,bao năm chẳng
779000,780000,làm ruộng nên trắng
780000,781000,biết làm gì
781000,782000,ngoài
782000,783000,làm cái gì cá ba
783000,784000,
784000,785000,xã
785000,786000,không bắt sao
786000,787000,
787000,788000,gì
788000,789000,nhưng bây
789000,790000,giờ dân ít
790000,791000,chơi rồi
791000,792000,chỉ thỉnh thoảng
792000,793000,vào dịch lễ
793000,794000,tết thôi
794000,795000,nên cũng
795000,796000,chẳng
796000,797000,
797000,798000,
798000,799000,
799000,800000,
800000,801000,
801000,802000,
802000,803000,
803000,804000,
804000,805000,
805000,806000,
806000,807000,
807000,808000,
808000,809000,
809000,810000,
810000,811000,
811000,812000,
812000,813000,
813000,814000,
814000,815000,
815000,816000,
816000,817000,
817000,818000,
818000,819000,
819000,820000,
820000,821000,
821000,822000,
822000,823000,
823000,824000,
824000,825000,
825000,826000,
826000,827000,
827000,828000,
828000,829000,
829000,830000,
830000,831000,
831000,832000,
832000,833000,
833000,834000,
834000,835000,
835000,836000,
836000,837000,
837000,838000,
838000,839000,
839000,840000,
840000,841000,
841000,842000,
842000,843000,
843000,844000,
844000,845000,
Anyone have same problems, and how to fix it?
Thank you

Hi @mlexplore1122, thank you for your ideal
I tried this and results as you predicted, model worked normally with lmweight=0 and wordscore=0.
Did u fix it?

[mlexplore1122]

Hi @mlexplore1122
I tried the same experiemnt

hi hieuhv94,
can you confirm same experiment with your audio?

Bug Description

When i inference by any type of inference (simple, multithread or interactive) with a long audio (30 minutes), inference work fine in 15 first minutes after that output is empty:
773000,774000,làm gì
774000,775000,
775000,776000,nhà tao biệt
776000,777000,lập ở xóm trại
777000,778000,đây
778000,779000,bao năm chẳng
779000,780000,làm ruộng nên trắng
780000,781000,biết làm gì
781000,782000,ngoài
782000,783000,làm cái gì cá ba
783000,784000,
784000,785000,xã
785000,786000,không bắt sao
786000,787000,
787000,788000,gì
788000,789000,nhưng bây
789000,790000,giờ dân ít
790000,791000,chơi rồi
791000,792000,chỉ thỉnh thoảng
792000,793000,vào dịch lễ
793000,794000,tết thôi
794000,795000,nên cũng
795000,796000,chẳng
796000,797000,
797000,798000,
798000,799000,
799000,800000,
800000,801000,
801000,802000,
802000,803000,
803000,804000,
804000,805000,
805000,806000,
806000,807000,
807000,808000,
808000,809000,
809000,810000,
810000,811000,
811000,812000,
812000,813000,
813000,814000,
814000,815000,
815000,816000,
816000,817000,
817000,818000,
818000,819000,
819000,820000,
820000,821000,
821000,822000,
822000,823000,
823000,824000,
824000,825000,
825000,826000,
826000,827000,
827000,828000,
828000,829000,
829000,830000,
830000,831000,
831000,832000,
832000,833000,
833000,834000,
834000,835000,
835000,836000,
836000,837000,
837000,838000,
838000,839000,
839000,840000,
840000,841000,
841000,842000,
842000,843000,
843000,844000,
844000,845000,
Anyone have same problems, and how to fix it?
Thank you

Hi @mlexplore1122, thank you for your ideal
I tried this and results as you predicted, model worked normally with lmweight=0 and wordscore=0.
Did u fix it?

sorry but i have in process for debug more. because code flow in lexicon decoder is not ez to understand so i think i need more time for debug. If any facebook dev dive to their code with idea behind my hypothesis with score lm, i think it will faster to resolv.
anyway, i will update when have any news. :)))

[mlexplore1122]

hi all,
I just test with decoder offline with LM is same with decoder in infrence streaming, so as same result, can't get full output of audio ( in this case LM is just 3 gram with prune and quantinize, this version is try to downsize of lm for streaming)
When I change LM to version 4 gram with quantinize only (version official for decoder) output of long audio is good, all audio has decoded.
So I think problem is from when you have LM is good enough, we can avoid problem (and problem from LexiconDecoder.cpp logic) like issuse #894 and this issuse not resolved too.
So just wait @vineelpratap for his answer :((((, and i will continue trying .

[vineelpratap]

Hi, I think the problem could be that for very very long audios, we need to re-normalize the computed alphas (forward probabilities) . I'm looking into the best way to fix it. Will get back soon...

[vineelpratap]

I cannot reproduce the issue.

I took a librispeech audio and replicated it 100 times to create a ~30 minute audio and used simple_streaming_asr_example and it transcribed everything correctly...

This is what I did...

> cd data 
> for f in acoustic_model.bin tds_streaming.arch decoder_options.json feature_extractor.bin language_model.bin lexicon.txt tokens.txt ; do wget http://dl.fbaipublicfiles.com/wav2letter/inference/examples/model/${f} ; done
> // consider a file from Librispeech audio.flac 
> sox audio.flac audio.wav // convert to .wav
> cp audio.wav longaudio.wav && for i in {1..100};do sox audio.wav longaudio.wav longaudio.wav; done
> ./$PATH/simple_streaming_asr_example -input_files_base_path data -input_audio_file longaudio.wav 

If you can give a way for me to reproduce the issue, it will help me in debugging...

[trangtv57]

hi @vineelpratap
as some comment above, problem just occurred when language model not enough good meaning fit with audio domain. So as this idea, i just use lm_3 gram with prune 0 5 6 in kenlm. And I try to find audio with big gap difference domain, but may be acoustic model train with unsupervised so good so i hard to find audio to make reproduce error ez.
But finally i found this audio with domain from news about games can be reproduce error ( 1 hour and error occured from 42th minute to 51 minuute not all audio) like some experiment in my language. And of course when i cut audio from 42-51 alone and decode, everything work normaly. I have include my cut audio for your reproduce. Thankyou
All resource for reprocedure in my drive:
drive folder save audio and lm_3_gram_prune_056 for reproduce error