Twilio-video.js: Share pictures and annotation

[ImAliveAndThatIsNice]

Hello,

Is it possible to share a picture and add annotation on it using twilio? If so Is there any simple example? And is it possible to change between sharing a picture and camera?

I have tried to start with "Local Data Tracks" by using the following code, but without success:

var dataTrack = new Twilio.Video.LocalDataTrack();
var token = 'Token';
Twilio.Video.connect(token, { name: 'RoomName', tracks: [dataTrack]
});
dataTrack.send(""Hello World!"");"

Best regards,

[markandrus]

Hi @ImAliveAndThatIsNice,

Yes, you could do something like this! We don't have code readily available for taking snapshots or sharing text annotations, but we do have a quickstart dedicating do "drawing" with DataTracks. Will you take a look at this project, https://github.com/twilio/draw-with-twilio, and let me know if it helps?

Thanks,
Mark

[marianopicco]

Hello Mark,

We're working on a project where there can be multiple people inside a chat room. When you click a button, you can snap a screenshot of someone's face (this is working already and pushes the current video fame into a canvas element), then we wanna be able to edit it (say draw something on it), and then send that image back so everyone can see it, for instance, substituting our own video track for a DataTrack with the image as data.

As far as I can tell, in the example, the DataTrack runs "parallel" to the video and audio tracks, can it substitute one of those? Ideally, users would choose to share either their live video, or an image.

[manjeshbhargav]

Hi @marianopicco ,

You can achieve this by using the HTML5 Canvas's captureStream API as follows:

function addSnapShotTrack(room, snapShotCanvas) {
  const stream = snapShotCanvas.captureStream(/* frameRate */);
  const track = new Twilio.Video.LocalVideoTrack(stream.getTracks()[0]);
  room.localParticipant.publishTrack(track);
}

With this, you are publishing a "video" Track which captures the contents of the Canvas
and shares it with the Room's Participants. Whenever you draw something on the Canvas,
the other Participants can see it in real time.

Caveat: Only the publisher of the Canvas "video" Track can draw on the snapshot.
Please let me know if this is the use-case you're going for.

Thanks,

Manjesh Malavalli
JSDK Team

[marianopicco]

This certainly feels like a step in the right direction, however I can't make it work with the draw-with-twilio demo (let alone my own app). I'm scouring through the code and can't figure out how to substitute the already created video track with the newly created stream.
Let's say I want this to happen when I hit the enter key. I added the following event:

  window.addEventListener('keypress', function (e) {
    var key = e.which || e.keyCode;
    if (key === 13) { // 13 is enter
      addSnapShotTrack(room, canvas);

    }
  });

The function addSnapShotTrack is very similar to the one you shown above, however it wont work. First it says Twilio is not defined (when you try to do new Twilio.Video.....), then I looked at the code and tried to copy the way it's done in other places and substituted that line with track => track.attach(stream.getTracks()[0]);

Except now track is not defined, however its used here:

 async function setupLocalAudioAndVideoTracks(video) {
  const audioAndVideoTrack = await createLocalTracks();
  audioAndVideoTrack.forEach(track => track.attach(video));
  return audioAndVideoTrack;
}

Which references audioAndVideoTrack, which is created locally from createLocalTracks(), which is nowhere to be found except in the initial declarations:

const {
  LocalDataTrack,
  connect,
  createLocalTracks
} = require('twilio-video');

I gotta say this is at the very least confusing as to how it works, it's probably my bias from Java where everything must be declared before being used, but anyway, can you help out?

Many thanks!

[marianopicco]

Just as an update, I have been trying to get this to work with the draw-with-twilio demo, and it required opening the console, unpublishing my localvideotrack first (and then even specifically the track id) :

room.localParticipant.unpublishTrack(room.localParticipant.tracks.get("4459b52f-9029-4e74-b073-347adb6af3ea"));

and then publish the "streamed track" from the canvas

var stream = canvas.captureStream(5);
var mytrack = new Twilio.Video.LocalVideoTrack(stream.getTracks()[0]);
room.localParticipant.publishTrack(mytrack.mediaStreamTrack);

Still would like to get some input as how the code to do this programatically would look like since my experiments have been less than successful.

After unpublishing the local video track (webcam) this stream seems to be deleted and I haven't been able to get it back either.

[ImAliveAndThatIsNice]

Hello all,

I am trying to change from a streaming video to a canvas. At this point what I have is the following:

// Unpublish the video track first
osTwilioVideoWeb.getConnectedRoom().localParticipant.tracks.forEach(function(track)
{
if(track.kind === 'video') {
osTwilioVideoWeb.getConnectedRoom().localParticipant.unpublishTrack(track);
}
});
// Publish

var canvasToSend = document.querySelector('canvas');
var stream = canvasToSend.captureStream(60);
var mytrack = new Twilio.Video.LocalVideoTrack(stream.getTracks()[0]);
var room = osTwilioVideoWeb.getConnectedRoom();

room.localParticipant.publishTrack(mytrack.mediaStreamTrack);
mytrack.attach();

The video is removed from the participants side but the new added track(from the canvas) does not show. I need to manually (from the console or from a button) make some change to the canvas so that it appears, ex. executing:
context.drawImage(canvas, 0, 0, w, h);

I already tried to call the line above, from the code, after publishing the track or call without success.

[dan-engineering]

Hi everyone,

Will this method

function addSnapShotTrack(room, snapShotCanvas) {
const stream = snapShotCanvas.captureStream(/* frameRate */);
const track = new Twilio.Video.LocalVideoTrack(stream.getTracks()[0]);
room.localParticipant.publishTrack(track);
}

be able to stream @30fps or more?

[JamieCorkhill]

@markandrus @manjeshbhargav Can either of you provide any estimations on the number of DataTrack messages that would be sent for the purposes of screen annotation? Suppose I want to permit users to draw on top of the video stream - this might be useful for remote help, for example. Do you think that could end up sending over 1 Million messages?

I realize that's not a lot of information, but any rough estimates would be appreciated. ]

Thanks.

[manjeshbhargav]

Hi @Mcdane ,

Will this method

function addSnapShotTrack(room, snapShotCanvas) {
const stream = snapShotCanvas.captureStream(/* frameRate */);
const track = new Twilio.Video.LocalVideoTrack(stream.getTracks()[0]);
room.localParticipant.publishTrack(track);
}

be able to stream @30FPS or more?

Yes, it will.

@JamieCorkhill ,

I recommend that you batch some of the mouse co-ordinates in a single message. That may help in reducing the number of messages that you may need to send. You can in theory send any number of messages, as long as the internal buffer does not overflow.

Thanks,

Manjesh Malavalli
JSDK Team