Twilio-video.js: Track Enable/Disable event not firing on manually published tracks

[qqkstar]

• [x ] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x ] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [x ] I verified that the Quickstart application works in my environment.
• [ x] I am not sharing any Personally Identifiable Information (PII)
  or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Code to reproduce the issue:

Case 1 when connecting to a room using the tracks connect option:

Twilio.Video.connect(twilioToken, {
  name: roomName,
  tracks: [LocalTrack|MediaStreamTrack] // The tracks array is constructed by converting from a MediaStream object containing 1 audio and 1 video track
})

Case 2 when publishing a manually construct LocalAudioTrack or LocalVideoTrack

const newLocalAudioTrack = new Twilio.Video.LocalAudioTrack(newAudioTrack); //newAudioTrack is a MediaStreamTrack object
localParticipant.publishTrack(newLocalAudioTrack); // track gets successfully published

// Trying to disable track after track has been published and registered by the remoteParticipant
localParticipant.audioTracks.forEach((publication) => { publication.track.disable(); }); // This will not work and remote Participant will continue to receive audio, no disable event is registered by any remoteParticipants

Expected behavior:

These tracks get published successfully to remoteParticipants in the room and enabling/disabling those individual tracks will fire off trackEnabled/trackDisabled events to remote participants.

Actual behavior:
The tracks get successfully published and then subscribed by remote participants. However, disabling those tracks by the localParticipant does nothing, no disable events fired and audio/video will continue to be streamed. Similarly, enabling those tracks will similarly do nothing.

Software versions:

• [x] Browser(s): Firefox85, Chrome88
• [x] Operating System: MacOS, Windows
• [x] twilio-video.js: 2.12.0
• [x] Third-party libraries (e.g., Angular, React, etc.): Node 12.18.3

[makarandp0]

Hello @qqkstar, Thanks you for writing with your question and sorry for late reply.

Tracks when published using publishTrack get published async. I noticed that in the code above, you are not waiting for the track publish promise to resolve. In that case the subsequent lines would have no effect. Can you try updating the code to

const newLocalAudioTrack = new Twilio.Video.LocalAudioTrack(newAudioTrack); //newAudioTrack is a MediaStreamTrack object

// wait for publisheTrack to resolve, before trying to disable track.
await localParticipant.publishTrack(newLocalAudioTrack); // track gets successfully published

// Trying to disable track after track has been published and registered by the remoteParticipant
localParticipant.audioTracks.forEach((publication) => { publication.track.disable(); }); // This will not work and remote Participant will continue to receive audio, no disable event is registered by any remoteParticipants

Let me know if that does not address the issue,
Thanks,
Makarand

[qqkstar]

Hey @makarandp0 ,

Thanks for getting back to me. I tried that and it still doesn't fire off enable/disable events.

I can confirm that the tracks get published successfully regardless of the await because from a remote participant's end I can see the newly published tracks via both the debug logs and also by actually seeing the new video stream. However, enabling and disabling those tracks do not fire off any events. Any time I create the tracks using new Twilio.Video.LocalAudioTrack() or new Twilio.Video.LocalVideoTrack() and then publish it, the enable/disable events do not fire off to other participants in the room. This behavior should be easily reproducible on your end and does not match the expected behavior from the documentation.

I'm convinced it's a bug, please try to see if you can reproduce it.

Thanks!

[makarandp0]

@qqkstar, you are right that await does not stop tracks from getting published. RemoteParticipant will see the tracks once they get published. But my point was that on publisher side you are trying to disable tracks too quickly, before they are actually published.

Please try waiting for sometime after publishing the tracks to enabling/disabling them. Perhaps by moving the logic of track.disable(), track.enable() on a UI action. You can check out this quickstart sample that demonstrates the usage of these APIS. Specifically this part enable/disables tracks on locally, and this part watches for enable/disable events on the remote side.

Thanks,
Makarand

[qqkstar]

Hi @makarandp0,

I do have the track enable/disable tied to an UI action and have deliberately waited for the tracks to finish publishing before trying to disable any of the tracks, but the disable events still won't fire.

Can you try reproducing the issue like this:

const myStream = await navigator.mediaDevices.getUserMedia();

 Twilio.Video.connect(twilioToken, {
    name: currentUser.channelId,
    tracks:streamToTwilioTracks(myStream),
  })

const streamToTwilioTracks =(stream) => {
  const twilioLocalTracks = [];
  const audioMediaStreamTracks = stream.getAudioTracks();
  if (audioMediaStreamTracks && audioMediaStreamTracks[0]) {
    twilioLocalTracks.push(
        new Twilio.Video.LocalAudioTrack(stream.getAudioTracks()[0]));
  }
  const videoMediaStreamTracks = stream.getVideoTracks();
  if (videoMediaStreamTracks && videoMediaStreamTracks[0]) {
    twilioLocalTracks.push(
        new Twilio.Video.LocalVideoTrack(stream.getVideoTracks()[0]));
  }
  return twilioLocalTracks;
};

If you initialize Twilio Video Connect by providing your own tracks, the enable/disable events will not fire for those tracks.

Keeping all other logic the same, enable/disable events fire correctly if I initialize Twilio Video Connect this way:

Twilio.Video.connect(twilioToken, {
  name: roomName,
  video: true,
  audio: true,
})

Please let me know if you are able to reproduce the issue.

Thanks!

[makarandp0]

Hey @qqkstar, Thanks for the update.

I tried your code I had to update

const myStream = await navigator.mediaDevices.getUserMedia();

to

const myStream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true });

but I was not able to reproduce the issue. Enabling/Disabling audio tracks is very common scenario, and we are not aware of any issues with track enabled/disabled events.

Since you mentioned that audio can be heard on the remote side after disabling the track - Looks like the track is not getting disabled for some reason. To debug this further, please share console logs from the publisher side and any room/track sids. If you can share your code that would help too.

Thanks,
Makarand

[qqkstar]

@makarandp0 For sure, to give you some context as to what I'm doing, I'm building this feature that allows the user to switch their inputs on the fly, for example switching from camera 1 to camera 2 dynamically without reloading the page. I filtered the code down to the most basic form so hopefully it's easily understood.

async function replaceStream(stream) { // stream is passed in via navigator.mediaDevices.getUserMedia();
  // Replace individual tracks from old stream with tracks from new stream
  const newAudioTracks = stream.getAudioTracks();
  const newVideoTracks = stream.getVideoTracks();

  await replaceTwilioTracks(newAudioTracks[0], newVideoTracks[0]);
}

async function replaceTwilioTracks(newAudioTrack, newVideoTrack) {
  const localParticipant = currentUser.twilioRoom.localParticipant;
  if (newAudioTrack) {
    const newLocalAudioTrack = new Twilio.Video.LocalAudioTrack(newAudioTrack);
    // Stop and unplublish existing audio tracks
    localParticipant.audioTracks.forEach((publication) => {
      publication.track.stop();
      publication.unpublish();
    });
    await localParticipant.publishTrack(newLocalAudioTrack);
  }

  if (newVideoTrack) {
    const newLocalVideoTrack = new Twilio.Video.LocalVideoTrack(newVideoTrack);
    localParticipant.videoTracks.forEach((publication) => {
        //  Stop and unplublish existing video tracks
        publication.track.stop();
        publication.unpublish();
    });
    await localParticipant.publishTrack(newLocalVideoTrack);
  }
  console.log('Replaced and republished MediaStream with fresh MediaTracks.');
}

// Mute/unmute logic
function updateCurrentUserAudioStatus(currentUser) {
  if (currentUser.audio === 'off') {
    console.log('muting');
    // Mute
    currentUser.twilioRoom.localParticipant.audioTracks
        .forEach((publication) => {
          publication.track.disable();
        });
  } else {
    console.log('unmuting');
    // Unmute
    currentUser.twilioRoom.localParticipant.audioTracks
        .forEach((publication) => {
          publication.track.enable();
        });
  }
}

Using the above logic, I can successfully toggle mute/unmute as long as I initialize twilio connect like this:

Twilio.Video.connect(twilioToken, {
  name: roomName,
  video: true,
  audio: true,
})

When I toggle mute/unmute, I see this being printed correctly and the audio gets muted expectedly on the remote participant's end:

canvas_utils.js:293 muting
15:49:22.575 twilio-video.min.js:52 2021-03-30T21:49:22.575Z info [LocalAudioTrack #1: bccbd83e-94da-4cae-ab35-0352c9f29a29] Disabling
15:49:23.150 canvas_utils.js:307 unmuting
15:49:23.151 twilio-video.min.js:52 2021-03-30T21:49:23.151Z info [LocalAudioTrack #1: bccbd83e-94da-4cae-ab35-0352c9f29a29] Enabling

But as soon as I replace the stream using the tracks I manually initialized, the same mute/unmute logic stops working. Here's the console debug to show you that I replaced and republished new tracks successfully, but using the same enable/disable logic, I don't see the same 'enabling' 'disabling' debug statements like right above:

2021-03-30T21:31:30.335Z info [LocalAudioTrack #1: 3fec784d-e9ae-43e4-bc8e-cc525cb6d27e] Stopping
15:31:30.337 twilio-video.min.js:52 2021-03-30T21:31:30.337Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Removed a AudioTrack: 3fec784d-e9ae-43e4-bc8e-cc525cb6d27e
15:31:30.337 twilio-video.min.js:52 2021-03-30T21:31:30.337Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Removed a LocalAudioTrack: 3fec784d-e9ae-43e4-bc8e-cc525cb6d27e
15:31:30.338 twilio-video.min.js:52 2021-03-30T21:31:30.338Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Removed a AudioTrackPublication: MT9e351a2de80c3b34b6f615c5e7b86e1f
15:31:30.338 twilio-video.min.js:52 2021-03-30T21:31:30.338Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Added a new AudioTrack: dbdb32c4-ec9f-4992-899c-ce1312ceb021
15:31:30.339 twilio-video.min.js:52 2021-03-30T21:31:30.339Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Added a new LocalAudioTrack: dbdb32c4-ec9f-4992-899c-ce1312ceb021
15:31:30.483 twilio-video.min.js:52 2021-03-30T21:31:30.483Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Added a new AudioTrackPublication: MT22d5744f19f70867238d452b3b6f4a7c
15:31:30.484 twilio-video.min.js:52 2021-03-30T21:31:30.484Z info [LocalVideoTrack #2: a714b7b6-20da-403e-8a91-716f064bfd63] Stopping
15:31:30.485 twilio-video.min.js:52 2021-03-30T21:31:30.485Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Removed a VideoTrack: a714b7b6-20da-403e-8a91-716f064bfd63
15:31:30.485 twilio-video.min.js:52 2021-03-30T21:31:30.485Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Removed a LocalVideoTrack: a714b7b6-20da-403e-8a91-716f064bfd63
15:31:30.485 twilio-video.min.js:52 2021-03-30T21:31:30.485Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Removed a VideoTrackPublication: MTdf6c8d9961ac7b20dffd07e5d6e1a68c
15:31:30.486 twilio-video.min.js:52 2021-03-30T21:31:30.486Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Added a new VideoTrack: 22d0cabc-db15-4205-a4e2-21d6b35a81d2
15:31:30.486 twilio-video.min.js:52 2021-03-30T21:31:30.486Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Added a new LocalVideoTrack: 22d0cabc-db15-4205-a4e2-21d6b35a81d2
15:31:30.679 twilio-video.min.js:52 2021-03-30T21:31:30.679Z info [LocalParticipant #1: PA77f2a8a8d669e835fec76a3c7f17d156] Added a new VideoTrackPublication: MT9759feefdd43005e735e72205d960edd
15:31:30.679 av_configuration.js:180 Replaced and republished MediaStream with fresh MediaTracks.
15:50:48.137 canvas_utils.js:293 muting
15:50:49.627 canvas_utils.js:307 unmuting

I also confirmed that the newly published publicationId and trackName match with the what's inside the localParticipant object.

Looking forward to hearing your thoughts!

[makarandp0]

Thank you for the details @qqkstar,
I looked at the room you posted about (RM063ae65deb26067997d524a0baefd63d) and I do not see the audiotrack (MT22d5744f19f70867238d452b3b6f4a7c) getting disabled.
As you noticed for successful disable you see log something like:

15:49:22.575 twilio-video.min.js:52 2021-03-30T21:49:22.575Z info [LocalAudioTrack #1: bccbd83e-94da-4cae-ab35-0352c9f29a29] Disabling

This is logged synchronously @ https://github.com/twilio/twilio-video.js/blob/master/lib/media/track/localmediatrack.js#L216
Since you do not see the line on console, I suspect your forEach loop isn't finding any tracks. Can you add a console log just before and after you call publication.track.disable(); and ensure that it does get called?

Hope this helps,
Makarand

[qqkstar]

@makarandp0 here you go.

As you can see, enabling disabling works when I first connect to the room.
Once I replace and republish the tracks, it stops working.

The forEach loop is picking up the newly published trackID from localParticipant just that somehow disabling/enabling it doesn't trigger the events.

Successfully joined room: [Room #1: RM7772de1b86545a8dc47dfd6864668f6d]
13:38:34.249 canvas_utils.js:293 muting
13:38:34.250 canvas_utils.js:303 Disabling track[LocalAudioTrack #1: 54f0a461-10de-4d2b-9b34-f4e15635612d]
13:38:34.250 twilio-video.min.js:52 2021-03-31T19:38:34.250Z info [LocalAudioTrack #1: 54f0a461-10de-4d2b-9b34-f4e15635612d] Disabling
13:38:34.251 canvas_utils.js:305 Finished disabling track
13:38:41.176 canvas_utils.js:309 unmuting
13:38:41.177 canvas_utils.js:320 enabling track[LocalAudioTrack #1: 54f0a461-10de-4d2b-9b34-f4e15635612d]
13:38:41.177 twilio-video.min.js:52 2021-03-31T19:38:41.177Z info [LocalAudioTrack #1: 54f0a461-10de-4d2b-9b34-f4e15635612d] Enabling
13:38:41.178 canvas_utils.js:323 Finished enabling track
13:38:47.085 twilio-video.min.js:52 2021-03-31T19:38:47.085Z info [LocalAudioTrack #1: 54f0a461-10de-4d2b-9b34-f4e15635612d] Stopping
13:38:47.086 twilio-video.min.js:52 2021-03-31T19:38:47.086Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Removed a AudioTrack: 54f0a461-10de-4d2b-9b34-f4e15635612d
13:38:47.087 twilio-video.min.js:52 2021-03-31T19:38:47.087Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Removed a LocalAudioTrack: 54f0a461-10de-4d2b-9b34-f4e15635612d
13:38:47.088 twilio-video.min.js:52 2021-03-31T19:38:47.088Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Removed a AudioTrackPublication: MT5e53b5c6df9e4da7cb0ce402da61f533
13:38:47.088 twilio-video.min.js:52 2021-03-31T19:38:47.088Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Added a new AudioTrack: 8a5e3664-1f62-4ea0-b279-4eb1b4575258
13:38:47.089 twilio-video.min.js:52 2021-03-31T19:38:47.089Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Added a new LocalAudioTrack: 8a5e3664-1f62-4ea0-b279-4eb1b4575258
13:38:47.264 twilio-video.min.js:52 2021-03-31T19:38:47.264Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Added a new AudioTrackPublication: MT40f816fb6ace41c79fd938ca7a40beba
13:38:47.265 twilio-video.min.js:52 2021-03-31T19:38:47.265Z info [LocalVideoTrack #2: 21423151-325e-454a-855a-6f620d77f80e] Stopping
13:38:47.266 twilio-video.min.js:52 2021-03-31T19:38:47.266Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Removed a VideoTrack: 21423151-325e-454a-855a-6f620d77f80e
13:38:47.266 twilio-video.min.js:52 2021-03-31T19:38:47.266Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Removed a LocalVideoTrack: 21423151-325e-454a-855a-6f620d77f80e
13:38:47.266 twilio-video.min.js:52 2021-03-31T19:38:47.266Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Removed a VideoTrackPublication: MTe2d9a92c1ae4d92d80a1707d8606971b
13:38:47.267 twilio-video.min.js:52 2021-03-31T19:38:47.267Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Added a new VideoTrack: 592fa941-1e14-41f8-b4ab-79cf2728e96b
13:38:47.267 twilio-video.min.js:52 2021-03-31T19:38:47.267Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Added a new LocalVideoTrack: 592fa941-1e14-41f8-b4ab-79cf2728e96b
13:38:47.428 twilio-video.min.js:52 2021-03-31T19:38:47.428Z info [LocalParticipant #1: PA8928f2b5e3691b22af840507ac4296df] Added a new VideoTrackPublication: MTb20d47d574983233bbb869f602210b65
13:38:47.428 av_configuration.js:180 Replaced and republished MediaStream with fresh MediaTracks.
13:38:51.426 canvas_utils.js:293 muting
13:38:51.430 canvas_utils.js:303 Disabling track[LocalAudioTrack #3: 8a5e3664-1f62-4ea0-b279-4eb1b4575258]
13:38:51.430 canvas_utils.js:305 Finished disabling track
13:38:55.034 canvas_utils.js:309 unmuting
13:38:55.035 canvas_utils.js:320 enabling track[LocalAudioTrack #3: 8a5e3664-1f62-4ea0-b279-4eb1b4575258]
13:38:55.035 canvas_utils.js:323 Finished enabling track

[makarandp0]

Interesting, I see your log but not the log we expect from https://github.com/twilio/twilio-video.js/blob/master/lib/media/track/localmediatrack.js#L216
Suggesting that maybe it never reached the code path - Could you print value of track.isEnabled along with your log? I wonder if the track thinks it's already disabled? Do you have this application hosted somewhere that I can take a look at ?

[qqkstar]

Hey @makarandp0 ,

😳 Yep, you were right on the money. Turns out in my mute/unmute function I had a separate variable that was referencing the same track and I was enabling/disabling that track before calling on localParticipant track's enable/disable. As a result, twilio thought the track was already enabled/disabled and was therefore not emitting the appropriate event.

All I had to do to fix it was to call localParticipant tracks enable/disable first before managing my other track variable reference.

Thank you for all the help!