Deepspeech: Text timeline

[kaszperro]

Could you please make it possible to get exact time at which each word was said?

[lissyx]

@kaszperro I might be confused, but isn't it what https://discourse.mozilla.org/t/audacity-vamp-plugin-for-deep-speech/23349 does ?

[kaszperro]

I have to check, because there isn't a word about such a functionality.

[JanX2]

@lissyx
The plugin does VAD first and then passes the audio snippets on to Deep Speech. So no. You get some time metadata this way, but it’s very coarse-grained. Additionally, the granularity is based on the pauses in the recorded speech. So it’s kinda random.

[mleue]

I second this. It would be great to have the possibility to get timestamps for every character, natively returned by DeepSpeech.

The way I understand it, every output character is based on a particular segment of the input audio anyways, so I assume that piece of information should be there and would just have to get returned by stt.
Also I guess internally DeepSpeech uses this information already to parse characters into words.

[reuben]

On 16 Feb 2018, at 17:01, Michael Leue [e-mail] wrote:

I second this. It would be great to have the possibility to get timestamps for every character, natively returned by DeepSpeech.

The way I understand it, every output character is based on a particular segment of the input audio anyways, so I assume that piece of information should be there anyways and would just have to get returned by stt.

It is not.
Also I guess internally DeepSpeech uses this information already to parse characters into words.

It does not.

Returning that information requires extending the CTC algorithm in TensorFlow, it's a non trivial change. Patches always welcome, of course :)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[kaszperro]

Seems like a challenge 😊

[mleue]

@reuben
Thanks for the explanations. I just quickly skimmed through your article here earlier https://hacks.mozilla.org/2017/11/a-journey-to-10-word-error-rate/ and there you are talking about "slices of audio" being used to predict character probabilities, so I thought there would be some way of linking back in the opposite direction as well.

Anyways, thanks for responding.

[mathematiguy]

I'm interested in doing something like this - at a minimum, it would be quite nice to be able to get the character probabilities before they go to the CTC step. I'm going to be doing my own digging to see how this might work, but if anyone has any leads I'd appreciate it.

[kaszperro]

In my opinion, you would need a training set with timestamped words (unfortunately I'm not aware of such). Correct me if I'm wrong.

[mathematiguy]

I'm confident that we can get the character probabilities by unfreezing and running the model ourselves at a minimum, but that wouldn't give timestamps because they would still have to be inferred. And as @reuben said, ideally this would be done by CTC itself.

It's possible we could get a 'good enough' time stamp once we have the character probabilities without actually rewriting the CTC algorithm, but it's hard to say without trying.

For the time being, I'm actually mostly interested in the character probabilities, and less so in the time stamps, although I do want those further down the line.

[reuben]

No need to unfreeze the model, just fetch the "raw_logits" node which is the output of the acoustic model (and input to CTC decoder).

[lissyx]

So, some news that @reuben was too shy to share: https://github.com/mozilla/DeepSpeech/issues/1656#issuecomment-431296511

[spencer-brown]

SO excited for the word timeline!

[lissyx]

Closing because this is being taken care of in https://github.com/mozilla/DeepSpeech/pull/1893 https://github.com/mozilla/DeepSpeech/pull/1892

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.