Cognitive-services-speech-sdk: Transcription of continuous stream?

[msimecek]

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [X] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

• create SpeechFactory
• create SpeechRecognizer
• try to continuously send byte[]
• not available...

Expected/desired behavior

Similar to the ProjectOxford Speech Recognition SDK (DataClient):

_speechClient = SpeechRecognitionServiceFactory.CreateDataClient(SpeechRecognitionMode.LongDictation, "en-us", "key");
_speechClient.SendAudio(audioChunk, audioChunk.Length); // audioChunk is byte[]

OS and Version?

Windows 10

Versions

Any :)

Mention any other details that might be useful

Is there a way to continuously stream audio to the speech service and get back text transcript? Not a file, but a series of byte arrays for more than hour (live speech transcription). I'm currently solving it with the old SDK and WebSockets, but I'm looking for ways how to do it with this one (since there are limitations).

[zhouwangzw]

The SDK 0.4.0 supports audio input stream, which should allow you to handle a series of byte arrays. Currently there is a limitation on audio length. We are working on it and will support long recogniztion very soon.
Thanks,

[msimecek]

@zhouwangzw Thank you for the information!

I'm looking at the sample and I'm not clear on how to continuously write to the AudioInputStream which is referenced when creating the Speech Recognizer...

This is how I try to do it:

var factory = SpeechFactory.FromSubscription("...key...", "North Europe");
_audioStream = new MemoryStream();
_audioStreamReader = new BinaryAudioStreamReader(new AudioInputStreamFormat() {
    FormatTag = 1,
    Channels = 1,
    BitsPerSample = 16,
    SamplesPerSec = 16000
}, _audioStream);
_speechRecognizer = factory.CreateSpeechRecognizerWithStream(_audioStreamReader);
_speechRecognizer.IntermediateResultReceived += _speechRecognizer_IntermediateResultReceived;
_speechRecognizer.FinalResultReceived += _speechRecognizer_FinalResultReceived;
_speechRecognizer.RecognitionErrorRaised += _speechRecognizer_RecognitionErrorRaised;

await _speechRecognizer.StartContinuousRecognitionAsync();

And then every time there are new bytes available, they are written into the stream:

public void WriteStream(byte[] chunk)
{
    _audioStream.Write(chunk, 0, chunk.Length);
}

I never get any response...

Could you advise what is the right way to do the continuous recognition when I don't have the whole Stream at the beginning?

[zhouwangzw]

Thanks for the feedback.

The issue could be due to different behavior of AudioInputStream::Read() and MemoryStream::Read(). Currently, AudioInputStream::Read() expects 0 in case the stream hits its end and there is no more data available. If there is no data immediate available, Read() should block untils the next data becomes available. But MemoryStream::Read() returns 0 if there is no data immediately available upon call to Read(). You probably need write your own wrapper to fill the gap.
Another issue in memory stream is that you probably need to reset the position (via seek()) after write(). Otherwise the written data can be retrieved by read(). There is only one stream position which is used both by read() and write().

[msimecek]

I see, thanks for the pointers, @zhouwangzw. I'm going to play with it a little more.

I'm wondering though - what's the proper way of doing continuous recognition with this SDK? How was it designed to work? On a stream which is endless, not coming from a file.

[zhouwangzw]

For continuous recognition with stream, your own AudioInputStream::Read() returns 0 when you stream is end, and then call StopContinuousRecognitionAsync() when you get the SessionStopped event. However, currently the connection could be closed by the service after a certain time. We are currently working on reconnection support which allows you run recognition for long time.

[zhouwangzw]

Btw, the region name in FromSubscription() should be ""northeurope", instead of ""North Europe".

[msimecek]

Ok, thank you. Having the support for long recognition built directly into the SDK would be awesome! In the meanwhile, I will go with custom implementation.

Closing this issue.

[msimecek]

Thanks again, @zhouwangzw , I was able to make it work with custom AudioInputStream. I wrote a blogpost about it - posting here as a reference for others: https://codez.deedx.cz/posts/continuous-speech-to-text/

[zhouwangzw]

@msimecek Glad to know it works, and thank you for writing the nice blog to share your experience.

Thanks,