Web-audio-api: web audio api connected to speech api

[cyrta]

Describe the feature
Briefly describe the feature you would like WebAudio to have.
Web audio API and audio processing graph is great. However, There exists to many interfaces that are not interchangable.
I think that we can make a connection between web audio api and speech api.
There could exist many use cases that could gain from it.
Any speech enhancement done as AudioWorkletNode, that would feed the SpeechRecognition or even SpeechRecognition done as a AudioWorkletNode. Furthermore the SpeechSynthesis could also get a post processing effects.

Is there a prototype?
No.
Some things can be done extracting MediaStream to buffer and feeding that to the AudioNode.

Describe the feature in more detail
I can imagine that it could be done on the low level, maybe even to connect with webrtc engine.
Maybe we can create MediaDevice that will be Sink and Source to WebAudioAPI.
Having a "virtual mic" that could be used as a input to SpeechRecognition would be one of the great features that would open possibility for a file or stream input to ASR in browser.

• Web Speech API specification

[hoch]

I played with this idea a bit, but it appears that Speech API implementations rely on the underlying speech synth provided from OS. Naturally it doesn't give the rendered audio data back to the browser. Another roadblock was to find a high quality open source speech synthesizer that does not require some sorts of backend (server).

But I think this can be prototyped with new techs at our hand(AudioWorklet, WASM and SAB) if we can get the local speech synth part done somehow.

Perhaps in V2?

[cyrta]

My claim is that it would be great if we could have a loop possible, from a sink to source.
If one would like to have a graph with a node that send sound to device (speakers) but also would like to have the input going back to the system as a microphone source node and reuse it sound.
e.g.

• this way speech recognition of a sound files or data in audio buffer would be possible using web speech api.
• effects on microphone before sending it via webrtc
  -

I think that those changes would need to be done e.g in the browser internals likechromium/content/browser/webrtc/*
and webrtc::AudioDeviceModul and chromium/media/base/audio_bus and chromium/media/base/audio_renderer_mixer.h .
Of course also media::AudioCapturerSource and WebRtcAudioCapturer::Capture

After some research, I've found that the only loop is show in this example:
getusermedia local audio rendering demo
and getusermedia audio volume rendering demo
using those piece of code: local audio demo & volume demo, all are using as a core (webrtc adapter.js
https://github.com/webrtc/adapter , https://webrtc.org/web-apis/interop/
https://developers.google.com/web/updates/2015/10/media-devices .
https://developers.google.com/web/updates/2015/10/inputdevicecapabilities .
I assume it would be almost impossible to convince people in webrtc to add speakers as audio input devices in mediaDevices.enumerateDevices()
However, if MediaStream is given a feature of screen capture, why not to give also audio capture and share capabilities. If I would like to playback a audio file while being on Hangoout so webrtc the only thing is to use "air" channel from real speakers to mic on laptop.
I still did not try let mediaStream = await navigator.getDisplayMedia({video:false, audio:true});
https://w3c.github.io/mediacapture-screen-share/ .

Finally, There is sample demo
Peer connection as input to Web Audio, and
Web Audio output as input to peer connection, chrome://flags/ webaudio input flag
code: main.js, webaudioextended.js.
I will try to use it in order to get OS speech recognition working live with browser audio input from microphone and obtain live transcription offline.

I assume It is not only synthetized by the OS itself, but Speech Api in Chrome or Mozilla uses an API first

• docs/accessibility/tts.md
  
  
  
  • OS tts calls: tts_mac.mm
  
  
  • OS tts calls: tts_mac.mm
  
  
  • OS tts calls on linux, using libspeechd, tts_linux.cc
  
  
  • OS tts calls on windows, tts_win.cc
  
  
  • chrome/browser/resources/network_speech_synthesis/tts_extension.js
  
  
  • chrome.tts api docs
  
  
  • The Chrome OS PATTS speech synthesis engine - Chromium OS
  
  
  • TTS Debug for chrome
  
  
  • Chrome TTS Demo app
  
  

there is Flite speech synth that is in pure C, so it can be ported easily to js via webassembly.
I get to know Flite very well, I think I will prepare a js version.
I found that prototype from 2010:

• https://hacks.mozilla.org/2010/05/fast-javascript-and-audio-speech-synthesis-in-your-browser/
• https://vimeo.com/12039415

yes, we can do it in V2

[hoch]

Thanks for your research! It really helps.

My claim is that it would be great if we could have a loop possible, from a sink to source.
If one would like to have a graph with a node that send sound to device (speakers) but also would like to have the input going back to the system as a microphone source node and reuse it sound.
e.g.

Technically, AudioDestinationNode has one output. You can plug it back to the graph if you want.
https://webaudio.github.io/web-audio-api/#AudioDestinationNode

this way speech recognition of a sound files or data in audio buffer would be possible using web speech api.

I am confused. Why do you need a loop-back to do such thing? In general, the speech recognition would generate a string and the speech synthesizer would be triggered by a string. Clearly there is a non-audio connection between two entities. I don't understand how audio loop-back helps this situation.

I guess we could have a SpeechRecognizerNode (extends AnalyserNode) and a SpeechSynthesizerNode (extends AudioScheduledSourceNode). Then one can make a connection from SSN to SRN to create a loop-back. Certainly possible, but not sure if there is a real-world use case for it. The connection from SRN to SSN can be useful, but it the user code must deliver the generated string to produce the synthesized voice. Also it doesn't seem like a practical use case.

I think that those changes would need to be done e.g in the browser internals likechromium/content/browser/webrtc/*
and webrtc::AudioDeviceModul and chromium/media/base/audio_bus and chromium/media/base/audio_renderer_mixer.h .
Of course also media::AudioCapturerSource and WebRtcAudioCapturer::Capture

Not sure why WebRTC is relevant to this feature request. Can you clarify here? If that's the part needs to be changed as you claim, perhaps it should be address from the WebRTC side.

I assume It is not only synthetized by the OS itself, but Speech Api in Chrome or Mozilla uses an API first
docs/accessibility/tts.md
OS tts calls: tts_mac.mm
OS tts calls: tts_mac.mm
OS tts calls on linux, using libspeechd, tts_linux.cc
OS tts calls on windows, tts_win.cc
chrome/browser/resources/network_speech_synthesis/tts_extension.js
chrome.tts api docs
The Chrome OS PATTS speech synthesis engine - Chromium OS
TTS Debug for chrome
Chrome TTS Demo app

Actually most of links you provided here is the evidence that TTS on web heavily relies on the underlying TTS service. Based on the code, the actual speech synthesis is done by either from OS or remote server. I am aware that ChromeOS uses its own module but that can't be generalized to all the browser implementations.

there is Flite speech synth that is in pure C, so it can be ported easily to js via webassembly.
I get to know Flite very well, I think I will prepare a js version.

That's great! I am also interested in this area, so perhaps we can start writing an explainer first. A working prototype with Audio Worklet would help the cause as well.

[guest271314]

@cyrta See https://github.com/guest271314/SpeechSynthesisRecorder. Note, at Chromium/Chrome webkitSpeechRecognition actually records local (user) audio and sends that recording to an external service Does webkitSpeechRecognition send recorded audio to a remote web service by default?
https://bugs.chromium.org/p/chromium/issues/detail?id=816095

[mdjp]

The speech api does not have the ability to output to a buffer so this is not currently possible in waapi.

[guest271314]

@mdjp Yes, the Web Speech API does not have the ability to output to a buffer.

[guest271314]

@mdjp

The speech api does not have the ability to output to a buffer so this is not currently possible in waapi.

That is not technically correct.

To demonstrate the statement is that it is possible to get the output of speechSynthesis.speak() as buffer

1. Plug in headphones to the device, e.g., a desktop or laptop machine.
2. Run the below code at console

navigator.mediaDevices.getUserMedia({audio:true})
.then(stream => {
  const text = [...Array(10).keys()].join(" ");
  const handleVoicesChanged = e => {
    const voice = speechSynthesis.getVoices().find(({name}) => name.includes("English"));
    const utterance = new SpeechSynthesisUtterance(text);
    utterance.voice = voice;
    utterance.pitch = 0.33;
    utterance.rate = 0.1;
    speechSynthesis.speak(utterance);
    const recorder = new MediaRecorder(stream);
    recorder.start();
    recorder.ondataavailable = async({data}) => {
      const ac = new AudioContext();
      const sourceBuffer = new AudioBufferSourceNode(ac, {buffer: await ac.decodeAudioData(await data.arrayBuffer())});
      sourceBuffer.connect(ac.destination);
      sourceBuffer.start(0);
      sourceBuffer.onended = e => console.log(e);
    }
    utterance.onend = e => (recorder.stop(), stream.getAudioTracks()[0].stop());
  }
  speechSynthesis.onvoiceschanged = handleVoicesChanged;
  speechSynthesis.getVoices();
})
.catch(console.error);

Since that is the only erroneous reason provided for closing the issue the issue should be immediately reopened.

[padenot]

This code is wrong, it's just recording the microphone and has nothing to do with the issue we're discussing here.

[guest271314]

@padenot

This code is wrong, it's just recording the microphone and has nothing to do with the issue we're discussing here.

Perhaps you did not follow the instructions properly. Plug in headphones so that the audio output does not emit through speakers. Enumerate and select the appropriate device.

In any event that is only only approach. Another approach would be to use meSpeak.js which outputs a data URL.

If you have a different working approach post the code.

[padenot]

I did, and the resulting buffer only contains the result of the getUserMedia, passed to MediaRecorder, decoded via decodeAudioData, and played back using the Web Audio API, as expected, because this is what the code does.

The speech synthesis is not connected to any of this, it's always playing back directly without using MediaStream.

Case in point, if you comment sourceBuffer.start(0), it doesn't change anything.

[guest271314]

Just to be clear, Web Speech API has a number of issues (in brief see this answer https://stackoverflow.com/a/48504311 at How can I make my web browser speak programmatically?). If you really want to connect to the underlying TTS engine you can use speechd (very helpful maintainers), spd-conf (python3-speechd), espeak-ng and bypass Web Speech API altogether, e.g., from How to programmatically send a unix socket command to a system server autospawned by browser or convert JavaScript to C++ souce code for Chromium?

// JavaScript
async function SSMLStream({ssml="", options=""}) {
  const fd = new FormData();
  fd.append("ssml", ssml);
  fd.append("options", options);

  const request = await fetch("speak.php", {method:"POST", body:fd});
  const response = await request.arrayBuffer();
  return response;
}

let ssml = `<speak version="1.0" xml:lang="en-US"> 
             Here are <say-as interpret-as="characters">SSML</say-as> samples. 
             Hello universe, how are you today? 
             Try a date: <say-as interpret-as="date" format="dmy" detail="1">10-9-1960</say-as> 
             This is a <break time="2500ms" /> 2.5 second pause. 
             This is a <break /> sentence break</prosody> <break />
             <voice name="us-en+f3" rate="x-slow" pitch="0.25">espeak using</voice> 
             PHP and <voice name="en-us+f2"> <sub alias="JavaScript">JS</sub></voice>
           </speak>`;

SSMLStream({ssml, options:"-v en-us+f1"})
.then(async(data) => {

    let context = new AudioContext();
    let source = context.createBufferSource();
    source.buffer = await context.decodeAudioData(data);
    source.connect(context.destination);
    source.start()

})
// PHP
<?php 
  if(isset($_POST["ssml"])) {
    header("Content-Type: audio/x-wav");
    $options = $_POST["options"];
    echo shell_exec("espeak -m --stdout " . $options . " '" . $_POST["ssml"] . "'");
  };

[guest271314]

@padenot When you enumerate devices and select "Monitor of Built-in " that is monitoring the speech output. Meaning the "speakers" are not being recorded.

[guest271314]

@padenot The code has everything to do with what you are discussing here. Meaning the code outputs the expected result. It would not be impossible to establish a direct connection to the speechd socket which can run even when the browser is closed. Meaning Web Speech API is broken and can be entirely removed from the equation. Connect Web Audio API directly to speechd or espeak-ng if that is the approach required. Not only is that possible _right now_ it is merely a the matter of making the decision to do so _right now_. For STT you can use https://github.com/cmusphinx/pocketsphinx.

[guest271314]

@padenot When you enumerate devices and select "Monitor of Built-in " that is monitoring the speech output. Meaning the "speakers" are not being recorded.

Neither is "microphone" being recorded in that pattern. You are recording "auidiooutput". If you have a different solution to achieve the expected result post the code.

[guest271314]

You should find this code using meSpeak.js if you actually read all of the links posted above. To make sure the code is viewable and usable re this topic see https://stackoverflow.com/questions/38727696/generate-audio-file-with-w3c-web-speech-api.

[guest271314]

FWIW a basic concept for utilizing AudioWorklet as sink (and Native Messaging to execute espeak-ng in a shell and output WAV file) to output audio result of local speech synthesis engine https://github.com/guest271314/native-messaging-espeak-ng/tree/bash-audioworklet.

Instead of getting the file then transferring the channel data to AudioWorklet one variation could be to pipe stdout from the local binary to the audio worklet or a programmatically created device (https://github.com/w3c/mediacapture-main/issues/654) for getUserMedia() to recognize and get the MediaStream of output, either as a single speech synthesis result then stop() the MediaStreamTrack or persist the stream enabling and disabling the track (suspend(), resume()) when the local binary is excuted and output piped to through the device to the media stream.

[guest271314]

@cyrta

I assume it would be almost impossible to convince people in webrtc to add speakers as audio input devices in mediaDevices.enumerateDevices()

Yes. Tried, still trying. No success thus far. Proof-of-concept implementation https://github.com/guest271314/captureSystemAudio