Cognitive-services-speech-sdk: Error for MULAW push stream on Windows

[garethkelly]

Hello,

I am trying to stream Twilio audio (MULAW) to Speech to Text service similar to #446 but I'm running C# on Windows.

I have GStreamer installed and using 1.11.0 of Microsoft.CognitiveServices.Speech

• OS: Windows
• Hardware - x64
• Programming language: C#

private SpeechRecognizer _recognizer;
        private PushAudioInputStream _inputStream;
        private AudioConfig _audioInput;

public async Task Start()
        {
            var config = SpeechConfig.FromSubscription(_projectSettings.AzureSpeechServiceSubscriptionKey, _projectSettings.AzureSpeechServiceRegionName);

            var audioFormat = AudioStreamFormat.GetCompressedFormat(AudioStreamContainerFormat.MULAW);

            _inputStream = AudioInputStream.CreatePushStream(audioFormat);
            _audioInput = AudioConfig.FromStreamInput(_inputStream);

            _recognizer = new SpeechRecognizer(config, _audioInput);
            _recognizer.SessionStarted += RecognizerStarted;
            _recognizer.Recognized += RecognizerRecognized;
            _recognizer.Canceled += RecognizerCancelled;

            await _recognizer.StartContinuousRecognitionAsync();
        }

I am taking the raw Twilio stream (media.payload see here) base64 decoding it and feeding it directly into the push stream with no buffer

public async Task Transcribe(byte[] audioBytes) { _inputStream.Write(audioBytes); }

I get the following error:

Message: The stream is of a different type than handled by this element.
DebugInfo: riff-read.c(262): gst_riff_parse_file_header (): /GstPipeline:pipeline/GstWavParse:wavparse:
Stream is no RIFF stream: 0x7a6f7afe
SessionId: 3aeeac302e6f410a9411751dd25512c8

Please let me know if my setup is incorrect as I could find no examples with a stream to pushstream scenario

[amitkumarshukla]

@garethkelly Thanks a lot for raising the issue. Right now for SpeechSDK to transcribe the ALaw and Mulaw stream, it has to be in the wav container. Looks like your stream is not in wave container. If you can please put your Alaw and Mulaw stream in a wav container and let us know your observation.

Thanks

[garethkelly]

@amitkumarshukla thanks for the reply.
I tried to add the audio to a Wav Container using nAudio:

public static void AddWavContainer(byte[] audioBytes, MemoryStream wavStream) { var waveFormat = new WaveFormat(16000, 16, 1); using (WaveFileWriter writer = new WaveFileWriter(new IgnoreDisposeStream(wavStream), waveFormat)) { writer.Write(audioBytes, 0, (int)audioBytes.Length); } }

And then call it
public async Task Transcribe(byte[] audioBytes) { var wavMemoryStream = new MemoryStream(); AddWavContainer(audioBytes, wavMemoryStream); _inputStream.Write(wavMemoryStream.ToArray()); }

I now get an error of:
Runtime error: Source: audio_source
Message: Internal data stream error.
DebugInfo: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline/GstAppSrc:audio_source:
streaming stopped, reason not-negotiated (-4)
SessionId: 3b43db726e5f465a811945f3b94c1250

1. Is there a better way of adding Wav Container?
2. Can I pass the audio as I receive it without buffer?

Thanks
Gareth

[amitkumarshukla]

@garethkelly If you have gstreamer installed on your machine, you can use the following to encode a pcm/wav file into mulaw.

gst-launch-1.0 -v filesrc location=whatstheweatherlike.wav ! wavparse ! audioconvert ! mulawenc ! filesink location=whatstheweatherlike.mulaw

Use the above mulaw stream to transcribe and check the result.

Could you also explain me what is your use case here. Like from where are you getting the stream and in what format?

One more thing to keep in mind here, in speech SDK we decode all the compressed stream into pcm (using gstreamer) and send the uncompressed pcm data to the service.

[garethkelly]

@amitkumarshukla My usecase is transcribing a phonecall from Twilio. I'm receiving packets of MULAW audio from Twilio (format) and a full sample repo can be found here

[amitkumarshukla]

@garethkelly Could you please capture a full Twilio stream and attach it here. I would like to investigate the stream.

[garethkelly]

@amitkumarshukla the Twilio Stream Output is attached in TwilioOutput.zip as I received it in order starting from 1.txt. Each text file contains base64 encoded message.

I was also not able to get whatstheweatherlike.mulaw to work. I converted the file as you suggested and copied SpeechRecognitionWithCompressedInputPullStreamAudio from examples:

using (var audioInput = AudioConfig.FromStreamInput(new PullAudioInputStream(new BinaryAudioStreamReader( new BinaryReader(File.OpenRead(@"Samples/whatstheweatherlike.mulaw"))), AudioStreamFormat.GetCompressedFormat(AudioStreamContainerFormat.MULAW))))

Get the error:
ErrorDetails=Runtime error: Source: wavparse
Message: The stream is of a different type than handled by this element.
DebugInfo: riff-read.c(262): gst_riff_parse_file_header (): /GstPipeline:pipeline/GstWavParse:wavparse:
Stream is no RIFF stream: 0xffffffff
SessionId: d799ec3a7a294bc7916f64988b09ccd2

Thanks for your help

Regards
Gareth

[amitkumarshukla]

whatstheweatherlike.zip

Can you try with the attached mulaw file and see if your scenario works ?

Dont you have a stream from Twilio. Because the speech SDK do not support base64 encoded data. So I doubt we will be able to decode it.

Can you provide me a simple C# code to get a stream from twilio ?

[garethkelly]

@amitkumarshukla the stream is each file containing a base64 encoding of Mulaw audio as per https://www.twilio.com/docs/voice/twiml/stream (see media.payload)

[garethkelly]

@amitkumarshukla I can confirm your file does transcribe for me so looks like GStreamer is installed and working correctly. What did you do to that file that was different from previous instructions?

The following code will simulate the Twilio stream coupled with my previously file attachment. I have placed into these files exactly what Twilio stream is giving me.

for (var i = 1; i <= 511; i++) { var path = $"TwilioOutput/{i}.txt"; var fileContents = File.ReadAllText(path); byte[] mulawBytes = Convert.FromBase64String(fileContents); }

Let me know if you the above works for you, otherwise to get a working stream from Twilio will require an account setup and additional setup.

[amitkumarshukla]

@garethkelly I would say just try to convert Twilio stream to wav file using gstreamer. Do let me know if you are able to do it. If you are able to do it, please paste me the code.

In the meantime I will see how to work with the stream that you have provided me.

[garethkelly]

@amitkumarshukla any update on this? Were you able to convert the sample MULAW stream using GStreamer? Is it supported?

[amitkumarshukla]

@garethkelly Currently we do not support twilio container format inside Microsoft speech SDK. I will suggest you to find a Twilio mulaw decoder plugin and decode the audio into pcm sample and use pull/push stream audio from Microsoft speech SDK and go ahead with your work. We might support in future some more formats and Twilio might also be one, for that please subscribe to our release notes https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/releasenotes. I am closing this issue for the time being. Thanks a lot.

[twilio-jyoung]

I too ran into this problem and only found this thread after working all weekend trying to get it to work.

@amitkumarshukla, I work for Twilio. Let me know if there is anything I can do to provide more details about how our streams work, or help you get setup with an account and unblock your testing. We have many customers who would like for this integration to work (hence my PoC efforts over the weekend).

our media payloads are mulaw/8000 base64-encoded. We've got pretty much every other major player built out and working except this one:

Amazon Transcribe - https://github.com/TwilioDevEd/talkin-cedric-node
Google Cloud Speech - https://github.com/twilio/media-streams/tree/master/node/realtime-transcriptions, https://github.com/twilio/media-streams/tree/master/java/realtime-transcriptions
IBM Watson - https://github.com/twilio/media-streams/tree/master/node/keyword-detection
Google Dialogflow - https://github.com/twilio/media-streams/tree/master/node/dialogflow-integration
and quite a few others...

[amitkumarshukla]

@twilio-jyoung As I have told that currently we do not support Twilio, but we would be really happy to provide a support for Twilio. Could you provide me a sample code (C#/C++) to convert from Twilio stream to a wave file using gstreamer? Also if possible could you attach some sample Twilio stream for our testing ?

[twilio-jyoung]

@amitkumarshukla, I'm hoping that you can help me understand how to do that. I can provide where I'm at but I am stuck in a non-working state.

Steps to reproduce:

• [ ] create a twilio account if you don't already have one. we'll give you $15 in credit, which should be more than enough to run this simple test. I can also upgrade your account and give you more credits ( and/or swag 😃 ) if needed.
• [ ] once you are logged in, buy a phone number using your credits.
• [ ] clone my broken repo - https://github.com/twilio-jyoung/MediaStreams2AzureSpeech
• [ ] follow the steps on the readme.md.

shouldn't take more than 10 minutes to get through this, and you should see this in the logs:

Message: The stream is of a different type than handled by this element. DebugInfo: riff-read.c(262): gst_riff_parse_file_header (): /GstPipeline:pipeline/GstWavParse:wavparse: Stream is no RIFF stream: 0xffffffff

I tried writing wav header using the instructions here, but it didn't seem to solve the problem, just a different error.

Message: The stream is of a different type than handled by this element. DebugInfo: gstwavparse.c(324): gst_wavparse_parse_file_header (): /GstPipeline:pipeline/GstWavParse:wavparse: File is not a WAVE file: 0x4c343255

[garethkelly]

Hi @twilio-jyoung

I was able to get it working (credit to Joris Kalz from Microsoft who gave me the solution)
Using the following two files:

https://www.codeproject.com/Articles/14237/Using-the-G711-standard
MuLawDecoder.cs
MuLawEncoder.cs

Let me know if you need me to upload my feature branch with full example.

Its a lot of code - and not even sure what it does :) so it would really be better if Microsoft supported MULAW/8000 format or Twilio was able to support streaming in WAV/PCM (was told this was on long term road map)

[twilio-jyoung]

hey @garethkelly, thanks for this! I'll start trying to plug this in now, but it you wouldn't mind passing your sample as well I would really appreciate it.

[garethkelly]

@twilio-jyoung - no problem Ive updated repo here

[twilio-jyoung]

@garethkelly, I cannot tell you how much I appreciate you sharing this with me. drop me an email and I'll ship you some twilio swag!

[amitkumarshukla]

@twilio-jyoung and @garethkelly Thanks a lot guys for doing the work around. Looks like we have more customer who needs mulaw decoding in twilio stream. We have created a workitem in our database and will try to ship this in our next release. closing it now.

[harshitazilen]

Hi, I am unable to install and setup Gsteamer on linux machine. I always face error of Gstreamer-1.0 not found. Kindly guide me.