Deepspeech: wrong Inference results in DeepSpeech. CPU vs GPU?

[MuruganR96]

Inference Running on GPU:
Linux Ubuntu 16.04
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.43
totalMemory: 3.94GiB
CUDA/cuDNN version:9.0

Inference Running on CPU:
Linux Ubuntu 16.04
i5 processor.
8 GB RAM

CPU gives Correct expected Inference result. like this,

deepspeech --model output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio 1.wav
Loading model from file output_graph.pbmm
TensorFlow: v1.11.0-9-g97d851f
DeepSpeech: v0.3.0-0-gef6b5bd
Loaded model in 0.00495s.
Loading language model from files models/lm.binary models/trie
Loaded language model in 1.5s.
Running inference.
**a what is the benefit for pan card** 
Inference took 3.080s for 4.598s audio file.

but in GPU gives wrong inference for same model. like this,

deepspeech --model model_export_cv02/output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio 1.wav
Loading model from file model_export_cv02/output_graph.pbmm
TensorFlow: v1.11.0-9-g97d851f
DeepSpeech: v0.3.0-0-gef6b5bd
Loaded model in 0.304s.
Loading language model from files models/lm.binary models/trie
Loaded language model in 1.35s.
Running inference.
**e e love jesting is wicebanethexapsing and weareiosdd** 
Inference took 3.154s for 10.000s audio file.

sir, My model trained on, EC2 P3 Instance 512GB RAM, 8 GPUs 16GB.

sir what is the problem? i don't know have idea, how to resolve this?

trained on: checkout DeepSpeech v0.3.0 and pretrained model, lm, trie.

how to resolve this issue sir.
thanks sir.

[lissyx]

@MuruganR96 Your command lines are different:

deepspeech --model output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio 1.wav
deepspeech --model model_export_cv02/output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio 1.wav

Are you sure you are running the same models ?

[MuruganR96]

@lissyx sir, both same model. but different system(one is GPU, another one is CPU), different dir sir.
thanks for your quick reply sir.

[lissyx]

@lissyx sir, both same model. but different system(one is GPU, another one is CPU), different dir sir.
thanks for your quick reply sir.

Still, I insist, your pasted console output refers to different models. The one testing on GPU seems to be some Common Voice ; it might be one you trained. This output would be consistent of a model improperly trained.

[MuruganR96]

@lissyx sir, both same model. but different system(one is GPU, another one is CPU), different dir sir.
thanks for your quick reply sir.

Still, I insist, your pasted console output refers to different models. The one testing on GPU seems to be some Common Voice ; it might be one you trained. This output would be consistent of a model improperly trained.

@lissyx sir,
but both as same model. one is copied from another one. both as same, common voice indian accents(augmented) audios trained model. how is this happened? three times i tried to copy right one and tested inference in the GPU. still it shows same wrong output.

Thank you so much for your reply sir

[kdavis-mozilla]

What is the output of

diff output_graph.pbmm model_export_cv02/output_graph.pbmm

[MuruganR96]

sorry sir,
CPU Result: a what is the benefit for credit card
GPU Result: e e love jesting is wicebanethexapsing and weareiosdd

Expected output:(transcript) :what is the benefit for credit card

[kdavis-mozilla]

What is the output of

diff output_graph.pbmm model_export_cv02/output_graph.pbmm

[MuruganR96]

sorry sir, i am very nerves, can't understand. this is difference sir.

CPU Result: a what is the benefit for credit card
GPU Result: e e love jesting is wicebanethexapsing and weareiosdd

Expected output:(transcript) :what is the benefit for credit card

[kdavis-mozilla]

Could you please type the following command into you terminal then hit return

diff output_graph.pbmm model_export_cv02/output_graph.pbmm

and add the text of the output to this issue

[MuruganR96]

(deepspeech-venv003) dell@dell-OptiPlex-7050:~/Murugan_R/DeepSpeech$ diff output_graph.pbmm model_export_cv02/output_graph.pbmm
**Binary files output_graph.pbmm and model_export_cv02/output_graph.pbmm differ**

(deepspeech-venv003) dell@dell-OptiPlex-7050:~/Murugan_R/DeepSpeech$ diff output_graph.pbmm DeepSpeech_Model/output_graph.pb
**Binary files output_graph.pbmm and DeepSpeech_Model/output_graph.pb differ**

sir what it means. i don't know. both are diffient binaries?
explain me what mistakes i made?

thank you sir.

[MuruganR96]

how do i resolve this?

[lissyx]

i don't know. both are diffient binaries?
explain me what mistakes i made?

Explain us what you did, because so far, it's not good.

[lissyx]

how do i resolve this?

Can you start by running exactly the same model and share the complete output? So far, all you did confirms you are running two different models and getting different results, which is 100% consistent.

[MuruganR96]

Mm k sir. again i will do same stuff. and report to you sir.
thank you so much @lissyx sir @kdavis-mozilla sir.

[bill-kalog]

@MuruganR96 just a wild guess but could it be the case that the audiofile you are testing is different as well.
In the first case your output says
Inference took 3.080s for 4.598s audio file.
whereas in the second
Inference took 3.154s for 10.000s audio file.

So is the first audio file 5 seconds long and in the second case the same file 10 seconds long?

[MuruganR96]

I am really sorry @lissyx sir @kdavis-mozilla sir and @bill-kalog sir. this is my very careless one. same audio name but different audio content. very sorry i was disrupt lots.

@bill-kalog thanks. thank you so much sir @lissyx @kdavis-mozilla

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.