Botframework-webchat: Text to Speech Randomly Working

Created on 10 Feb 2019 · 14Comments · Source: microsoft/BotFramework-WebChat

Using Webchat v4 with Botframework v3.
In most of the case the bot stops speaking in the middle of a phrase.
Speech-to-Text works fine though.
Also getting special characters in the bot reply like 'Good afternoon! Iâ€™m Aila, a Microsoft Virtual Conversational AI Agent. How can I help you today?'
Thoughts?

<!DOCTYPE html>
<html lang="en-US">
  <head>
    <title>Web Chat: Cognitive Services Speech Services using JavaScript</title>
    <script src="https://cdn.botframework.com/botframework-webchat/latest/webchat.js"></script>
    <style>
      html, body { height: 100% }
      body { margin: 0 }
      #webchat,
      #webchat > * {
        height: 100%;
        width: 100%;
      }
    </style>
  </head>
  <body>
    <div id="webchat" role="main"></div>
    <script>
      (async function () {
        const searchParams = new URLSearchParams(window.location.search);
        const subscriptionKey = searchParams.get('s');
        let webSpeechPonyfillFactory;
        if (subscriptionKey) {
            webSpeechPonyfillFactory = await window.WebChat.createCognitiveServicesSpeechServicesPonyfillFactory({
            region: searchParams.get('r') || 'westus',
            subscriptionKey
          });
        }
        window.WebChat.renderWebChat({
          directLine: window.WebChat.createDirectLine({secret:'XXXXXX'}),
          webSpeechPonyfillFactory
        }, document.getElementById('webchat'));
        document.querySelector('#webchat > *').focus();
      })().catch(err => console.error(err));
    </script>
  </body>
</html>

Bug

Source

Amintasn

Most helpful comment

@Amintasn - I'm helping @corinagum get to the root of this problem while she focuses on our 4.3 release going into code freeze soon.

Using your bot on Chrome 73, I went over the scenario 4x with no text-to-speech issues.
Safari did not work for text-to-speech at all
Firefox with an ad blocker did not work.
Firefox with the ad blocker turned off did work flawlessly in all attempts.

To rule out a browser issue, are you able to try this in a different browser? Firefox, Edge or Brave would be good to try out. Also check to be sure you have all add-blockers turned off since they seem to interfere.

If you still see the problem, I'd like to get you a dev build of webchat so we can zero in on this a bit more efficiently.

justinwilaby on 14 Feb 2019

👍2

All 14 comments

Hey @Amintasn, thanks for reporting this. Do you have any repro steps for the text-to-speech issue? In case this isn't repro'd on MockBot, are you able to share your published bot to test this out on?

As for the special characters, this strikes me as an encoding issue. As a quick test, could you try setting your charset to utf-8 and see if this problem persists? Setting charset on the client side is a deprecated workaround but if it does repro at least we will know the cause. :)

corinagum on 11 Feb 2019

Hey @corinagum, thanks for the quick reply.
While speaking to the bot it sometimes replies by voice, sometimes stays quiet or in some situations starts speaking and then stops in the middle of the answer.
You can try it out at https://bit.ly/2RXMWdw
Try saying:
. Hi
. What can you do for me?
. What's a chat bot?
BTW, I'm using https://cdn.botframework.com/botframework-webchat/latest/webchat.js
On the bright side, spot on the encoding issue!
Working now. :)

Thx

Amintasn on 12 Feb 2019

👍1

@Amintasn, glad the temporary workaround works for you. Please keep an eye on #1071 for the fix. :)

Let me get back to you about text-to-speech after I investigate.

corinagum on 12 Feb 2019

@corinagum Great. Thanks a lot for the support. :)

Amintasn on 13 Feb 2019

@Amintasn, Unfortunately I'm not able to repro this bug. I tried taking a look at your bot, but it looks like your bot is offline and I'm unable to connect. I also tried this out a few times on Mock Bot, and never came to an problem where the bot stopped speaking mid-utterance.

Do you happen to have any console errors when you run into this issue?

I spoke with @compulim about your problem and he asked me to give you this link to test out your Speech Services on his cognitive services app. Could you try out the text-to-speech synthesis and see if you are able to repro the voice cutting out?

What I did: added the speech subscription key in the upper right
Use Speech synthesis tab
Use the Winnie the Pooh preset, or any text that you want to test.
Use the en-US voice, assuming your bot is in English:

Please let us know any errors or information you can gather if this repros. :)

corinagum on 14 Feb 2019

@corinagum Hi Corina, thanks for your support. I guess you probably didn't notice the bot windows given it was full maximized. Now I've placed in a visible screen position. :)
Please try again at https://bit.ly/2RXMWdw when you can.
I've tried the cognitive service app and it worked well with my key.
While running the app on the browser I see the following console message.
webchat.js:67 Web Chat: Cognitive Services Speech Services support is currently in preview. If you encounter any problems, please file us an issue at https://github.com/Microsoft/BotFramework-WebChat/issues/.
In summary:
. The cognitive service is working fine on your app.
. In my application, the bot replies correctly by text
. It works fine on voice-to-text but fails to do text-to-voice properly.

Thanks!

Amintasn on 14 Feb 2019

@Amintasn - I'm helping @corinagum get to the root of this problem while she focuses on our 4.3 release going into code freeze soon.

Using your bot on Chrome 73, I went over the scenario 4x with no text-to-speech issues.
Safari did not work for text-to-speech at all
Firefox with an ad blocker did not work.
Firefox with the ad blocker turned off did work flawlessly in all attempts.

If you still see the problem, I'd like to get you a dev build of webchat so we can zero in on this a bit more efficiently.

justinwilaby on 14 Feb 2019

👍2

@justinwilaby - Thanks for your help.
I've actually found the root cause of the issue after reflecting on your points.
Yes, it fails in some browsers but it partially works on Chrome and Edge which are the most important platform for us.
Now the main cause of the text-to-speech speaking failure is that the bot at https://bit.ly/2RXMWdw asks the user if he/she has additional questions after the main question has been answered.
So when the "additional questions" type message gets into the channel it interrupts the reading task to focus on the new task.
A workaround for now is to stop using "additional questions" type messages.
On the other hand it would be great if the webchat.js could queue messages to read in queued fashion.
For example, Chrome in-browser text-to-speech service manages it that way.
I've posted the same bot app at https://bit.ly/2CYnIYf using Chrome and the latest botchat.js. CDN.
Please try the following questions in both bot instances to see how it's addressed differently by them:

Hi
What can I do for me?
Anyway, it's a suggestion for future versions.
Thx

Amintasn on 15 Feb 2019

@Amintasn - Good call with queuing messages for text-to-speech. We'll look into that. I'm glad you found a workaround and thanks again for bringing this issue to our attention.

justinwilaby on 15 Feb 2019

@justinwilaby Thanks!

Amintasn on 15 Feb 2019

Opening a new issue for the queuing text-to-speech discussion. :)

corinagum on 15 Feb 2019

@corinagum You rock Corina! :) Thanks

Amintasn on 16 Feb 2019

We are facing some issue in browsers other than chrome speak icon is not displaying in other browsers
voice issue

rgovindan29 on 4 Sep 2019

'How to' questions such as above are better suited for Stack Overflow. Please feel free to post other questions you have about developing your own features over there so the community at large may help out. If you need help with a Web Chat implementation, you can post a question to the Web Chat tag. Thank you!

corinagum on 4 Sep 2019

Was this page helpful?

0 / 5 - 0 ratings