Twilio-video.js: Passing a mediastream directly to the track creator?

[randallb]

I call getusermedia way in advance of creating tracks for a bunch of reasons. In the new beta5 release, all the methods for passing mediastreams directly in to create tracks are gone, and now we're left passing constraints.

This is sub optimal for our usecase. I'm writing some work arounds right now but the previous way of adding a mediastream to localmedia was much preferred.

My ideal API right now would be to create the tracks by calling

https://media.twiliocdn.com/sdk/js/video/releases/1.0.0-beta5/docs/global.html#createLocalTracks

and instead of passing in an options object, I'd like to pass the mediastream directly. Beta4 had this, and in my opinion it was a huge api help.

[stueynet]

Tend to agree. Also wondering when does the CDN start pointing to beta5?

[kmahlqvist]

+1 We also create a stream in advance and in beta4 we used the existing one.

[markandrus]

Hi @randallb,

My ideal API right now would be to create the tracks by calling

https://media.twiliocdn.com/sdk/js/video/releases/1.0.0-beta5/docs/global.html#createLocalTracks

and instead of passing in an options object, I'd like to pass the mediastream directly.

Thanks, that is good feedback. In the short-term, you can do this

const { LocalAudioTrack, LocalVideoTrack, connect } = require('twilio-video');

const audioTracks = stream.getAudioTracks().map(track => new LocalAudioTrack(track));
const videoTracks = stream.getVideoTracks().map(track => new LocalVideoTrack(track));
const tracks = audioTracks.concat(videoTracks);

connect(token, { tracks }).then(room =>
  // ...
});

It's more verbose, but hopefully sufficient for now? Thanks, too, @stueynet and @kmahlqvist. We may add an override then, e.g.

/**
 * @param {MediaStream} stream
 * @returns {Array<LocalTrack>} tracks
 */
createLocalTracks = /* ... */

[manjeshbhargav]

@randallb , @stueynet , @kmahlqvist ,

In 1.0.0-beta5, we have stopped handling MediaStreams directly. However, there is a way to pass in previously acquired MediaStreamTracks by using the LocalAudioTrack and LocalVideoTrack constructors:

const { LocalAudioTrack, LocalVideoTrack, connect } = require('twilio-video');
const mediaStream = await navigator.mediaDevices.getUserMedia(constraints);
const tracks = mediaStream.getTracks().map(track => track.kind === 'audio'
  ? new LocalAudioTrack(track) : new LocalVideoTrack(track));
const room = await connect('token', { tracks });

Thanks,

Manjesh Malavalli

[markandrus]

@stueynet

Also wondering when does the CDN start pointing to beta5?

You can get 1.0.0-beta5 from the following URLs:

• https://media.twiliocdn.com/sdk/js/video/releases/1.0.0-beta5/docs
• https://media.twiliocdn.com/sdk/js/video/releases/1.0.0-beta5/twilio-video.js
• https://media.twiliocdn.com/sdk/js/video/releases/1.0.0-beta5/twilio-video.min.js

Are you asking about the v1 URLs? e.g., https://media.twiliocdn.com/sdk/js/video/v1/docs? Those point to 1.0.0-beta4 currently, although we really shouldn't have updated them. As y'all have seen, there are breaking API changes between beta releases right now. We really shouldn't pin that URL until 1.0.0 is out (at which point we are committed to compatibility). So once 1.0.0 is released (available soon after beta6), we will update that URL to the 1.0.0 release.

[stueynet]

So it's staying at beta4 then? Or should I change our links to hardcore to it.

[markandrus]

@stueynet the v1 URL will remain beta4 until we release 1.0.0 proper, at which point we will update it to point to the actual 1.0.0 release. So I recommend you update your links to the https://media.twiliocdn.com/sdk/js/video/releases/1.0.0-beta5/twilio-video.min.js URL, or, if you can, use package.json to pin to a particular version (depends on your build setup).

[randallb]

OK-- in my case, and probably most people's cases who already have the mediastream, we'll be adding the tracks directly to the localParticipant instead of when we join... but the basic concept is the same. I'll be implementing directly now.

[markandrus]

@randallb ah, ok--so you may prefer an addStream on the LocalParticipant? Related: one feature we are discussing for a future release, is the ability to add a name when you publish a Track, e.g.

localParticipant.addTrack(track, { name: 'my-camera' });

(although we could also imagine the name as a property of the LocalTrack object, rather than passed via addTrack). How would you want to specify names for each Track generated by a MediaStream object, e.g. if we allowed createLocalTracks to accept a MediaStream or if we added an addStream to LocalParticipant? Should we leave it null, choose one at random?

[randallb]

addStream on LocalParticipant would be amazing. (!!!one1!)

I end up using your IDs for everything right now. If I added a name to createLocalTracks I'd probably do it just like the previous example.

createLocalTracks(mediaStream, {name: 'scene1'})

I would love foreign keys on all my mediastream tracks. Would be amazing for a lot of reasons.

[randallb]

@manjeshbhargav FYI that code errors out right now b/c it requires an options object, which is clearly a bug.

https://github.com/twilio/twilio-video.js/pull/83

I made a silly pull request to fix it, but the point was more to highlight the bug and have whomever fix it who knows the twilio style. :)

[manjeshbhargav]

@randallb ,

Thanks for bringing this to our notice. We've scheduled the fix for this to be released with 1.0.0-beta6.

Thanks,

Manjesh Malavalli

[manjeshbhargav]

Hi @randallb ,

We're debating how to incorporate MediaStreams to our API. Our intention is to retain our Track-based design while still accepting MediaStreams. Do you think you would need a way to remove a MediaStream from a LocalParticipant sometime after you've added it?

We're thinking of the providing a method called addTracksFromStream(mediaStream) on the LocalParticipant. I guess this would fulfill your use-case of just passing a MediaStream to the LocalParticipant.

So, the question is, do you feel strongly that you need a corresponding remove(mediaStream) method as well? Our initial thinking is that addTracksFromStream is just a convenience method, and there should be no expectation that the MediaStream should be maintained by the SDK for possible later removal.

Any feedback you have is greatly appreciated.

Thanks,

Manjesh Malavalli

[randallb]

I would definitely like a remove, though I don't absolutely need one. The usecase I'm thinking about is screensharing... once a user wants to remove one, they'd just need to keep track of the tracks a bit better instead of just removing the mediastream, which is ok I think.

In summary, nice to have not need to have.

[markandrus]

Hi @randallb,

In 1.0.0-beta6 we

• Augmented the addTrack/removeTrack APIs to accept bare MediaStreamTracks,
• Augmented the tracks property of ConnectOptions to accept an Array of MediaStreamTracks in addition to LocalTracks, and we also

• Added two methods: addTracks and removeTracks. These do not accept MediaStreams directly, but it's straightforward to pass the MediaStream's underlying MediaStreamTracks as follows:

  localParticipant.addTracks(stream.getTracks());
  
  localParticipant.removeTracks(stream.getTracks());
  

We were a little worried about treating MediaStreams as glorified "containers" for MediaStreamTracks and allowing calls like

localParticipant.addTracks(stream);

localParticipant.removeTracks(stream);

or even

connect(token, { tracks: stream });

outright, simply because we were concerned that users might assume future changes to the MediaStream might percolate up to the Room. But we are willing to revisit this change in a future release (FWIW, there is some precedent by way of the MediaStream constructor itself, which accepts either an Array of MediaStreamTracks _or_ another MediaStream directly).

We hope you can take a look when you get a chance!

Thanks,
Mark

[sboudouk]

How is the evolution of this ticket ? Is it now possible to attach MediaStream obtained by getUserMedia directly to Twilio ?

[manjeshbhargav]

@sboudouk ,

Yes. You can pass in the MediaStreamTracks from getUserMedia to connect() and LocalParticipant.publishTrack():

getUserMedia(...).then(stream => {
  return Twilio.Video.connect(token, { tracks: stream.getTracks() });
}).then(room => {
  return getUserMedia(...).then(stream => {
    return room.localParticipant.publishTracks(stream.getTracks());
  });
});

Thanks,

Manjesh Malavalli
JSDK Team

[sboudouk]

Thanks for your answer and the time you took for this snippet.

Keep up the good work.

[sboudouk]

I still get the issue, I'm using twilio-video 1.19. Do I need to update to a more recent version ?

Here is my code:

    navigator.getUserMedia({ audio: true, video: true }, (stream): void => {
      console.log('i got stream:');
      console.log(stream);
      connect(token.token, {
        name: consultation.uniqueName,
        tracks: stream.getTracks(),
      }).then(roomJoined).catch((error): void => {
        console.log('in error');
        console.log(error);
      });
    }, (error): void => {
      console.log('in error');
      console.log(error);
    });

Also, I'm using cordova-plugin-iosrtc to give me access to getUserMedia.