Discord.js: Voice channel stream separation of audio and video data

[healzer]

Hi team,

I have been using DiscordJS for many months now for my voice-to-text framework.
In particular the feature of listening to multiple users' audio, which can then be transcribed into text.
A lot of our users are hearing impaired / deaf people who can now better communicate with their friends.

We currently have the issue that when someone shares their webcam, the Opus stream for that user becomes very long, and isn't cut using the silence method for audio. Meaning the received Opus data keeps flowing until the webcam is turned off.

This is how we are currently creating a stream per user:
const audioStream = voice_Connection.receiver.createStream(user, { mode: 'pcm' })

Is it possible to separate the stream into audio and video streams at a lower level? I'm not sure if this is a bug or a feature request.

Full code is available here: https://github.com/healzer/DiscordEarsBot/

[amishshah]

Hey! Firstly, it sounds like you're working on a really cool project and I'm sure that your users really appreciate the work you are doing :heart:

An update on this issue - I've now isolated the issue. While a user's camera is on, we still receive an audio stream from them - it's just that this audio stream contains the Opus silence packet f8 ff fe. I'll comment back once I've thought of the best way to implement this fix.

[amishshah]

@healzer this should now be fixed and it is released in v12.5.1!

[healzer]

Many thanks @amishshah !!! <3

[healzer]

@amishshah I have to bother you again my friend 😅

The user who had this issue, keeps getting it even without his webcam. Its strange because none of our other users have ever encountered such a thing before. Maybe you have a better clue?

/var/www/DiscordEarsBot/node_modules/prism-media/src/opus/Opus.js:64
    return this.encoder.decode(buffer, Opus.name === 'opusscript' ? null : this._options.frameSize);
                        ^

TypeError: The compressed data passed is corrupted
    at Decoder._decode (/var/www/DiscordEarsBot/node_modules/prism-media/src/opus/Opus.js:64:25)
    at Decoder._transform (/var/www/DiscordEarsBot/node_modules/prism-media/src/opus/Opus.js:189:20)
    at Decoder.Transform._read (_stream_transform.js:191:10)
    at Decoder.Transform._write (_stream_transform.js:179:12)
    at doWrite (_stream_writable.js:403:12)
    at writeOrBuffer (_stream_writable.js:387:5)
    at Decoder.Writable.write (_stream_writable.js:318:11)
    at Readable.ondata (_stream_readable.js:718:22)
    at Readable.emit (events.js:314:20)
    at addChunk (_stream_readable.js:297:12)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-11-27T06_11_49_616Z-debug.log

[amishshah]

Hi @healzer so just to confirm, this is only happening for one of your users?

If so, could you provide some more information about whether they're using Discord on his phone or computer when it occurs?

[healzer]

@amishshah okay so I was able to get more information. The error happens only for one specific user whenever he/she tries to speak, using discord on PC. It's not much... Could it be due to an outdated discord version? Or maybe they have some special settings?

Unfortunately that's all I can get at the moment, that discord bot is hosted by one of our users who reports this.

[healzer]

@amishshah great news! we have found more details.
The bot crashes with that error when using Discord through a web browser on Chrome. It works just fine on Edge and FireFox.
As soon as the person tries to speak it crashes immediately.
I hope this gives you enough information to figure out the cause? :)