Twilio-video.js: Access to raw streams

[rlancer]

javascript participant.media.attach(document.body);

Works great if you want to just display the users video, but for my app I'm doing processing against the stream and need access to it.

[markandrus]

Hi @rlancer,

You can get access to the underlying MediaStream and MediaStreamTracks by query the Tracks directly. Take a look at the mediaStream and mediaStreamTrack properties here. You can also listen for the "trackAdded" and "trackRemoved" events on Media, in addition to iterating over tracks directly.

Does this help?

Best,
Mark

[rlancer]

@markandrus thanks for your reply!

After many attempts to grab the raw stream the only thing that worked was doing.

participant.media.on('trackAdded', function trackAdded(track) {
      console.log('Track added', track.mediaStream);
});

Maybe update the docs to include the reasons for this?

[markandrus]

@rlancer good to hear, and will do. In short, applications need to listen to the "trackAdded" and "trackRemoved" events because a remote Participant is free to add or remove Tracks at any time. So the MediaStreams need to be queried when they are added/removed.

[rissem]

I am only finding a MediaStreamTrack available on the the Twilio track object. Is there a way to access the stream that would be usable w/ methods such as this one?

https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/createMediaStreamSource

[markandrus]

@rissem yes, you can construct a MediaStream at any time with

const stream = new MediaStream()

and then add MediaStreamTracks to it, like

stream.addTrack(track.mediaStreamTrack)

Please let me know if that works.

Best,
Mark

[rissem]

Thanks! That's just what I needed 😄

[embirico]

@markandrus: Based on your recommendation here, could you confirm that it's ok for an application to skip using attach() and detach() altogether, and instead render tracks/streams directly?

(From a glance at the code, looks like attach() creates a new MediaStream and sets it as the srcObject of the HTMLMediaElement, which is what we'd like to do ourselves. However, not sure if there's any magic tracking done on Twilio's end that's important to preserve.)

[markandrus]

@embirico yes, you can definitely do this! In fact, we considered removing and/or changing the attach/detach APIs to return MediaStreams instead of HTMLMediaElements; however, we're afraid that would be too drastic an API change. To be clear:

• Many APIs take MediaStreams (srcObject, Web Audio, etc.)—you can always construct a MediaStream with

  new MediaStream()
  

• All AudioTracks and VideoTracks—be they local or remote—have a public API property, mediaStreamTrack, that's constant for the lifetime of the Audio- or Video-Track. So passing track.mediaStreamTrack to a MediaStream is totally valid.

However, not sure if there's any magic tracking done on Twilio's end that's important to preserve.

No, there shouldn't be any magic. The one quirk I recall is that some browsers (Chrome and Safari, I think), require resetting the srcObject on the HTMLMediaElement whenever the MediaStream's underlying MediaStreamTracks change (via addTrack/removeTrack on the MediaStream); although I think this should be considered a bug in those browsers and will eventually be fixed.

[embirico]

Thanks this is great news!

I might suggest mentioning somewhere in the docs that this is ok to do. Our current implementation jumps through some hoops in order to be able to use attach and detach.

For the benefit of anyone reading this issue in the future, I believe the Chromium bugs Mark is referring to are:

• https://bugs.chromium.org/p/chromium/issues/detail?id=720258
• https://bugs.chromium.org/p/chromium/issues/detail?id=749928