Twilio-video.js: Missing Audio and Video tracks in publications

[quentinhayot]

Hi,

We are trying to migrate from 1.x to 2.x.
We have issues with Audio and Video publications having their track set to null and isSubscribed to false when using Chrome.

• Data tracks are NOT affected
• Audio and video tracks are affected

✅Firefox <-> Firefox
✅Firefox <-> Safari
✅Safari <-> Safari
🔴Firefox <-> Chrome : (Firefox gets all remote tracks correctly but chrome only gets data)
🔴Chrome <-> Safari : (Safari gets all remote tracks correctly but chrome only gets data)
🔴Chrome <-> Chrome : (Only data tracks are working)

Bonus:
✅Android (sdk 2) <-> Safari
🔴Android (sdk 2) <-> Chrome : (Android gets audio and video, chrome gets nothing, data not tested)
🔴Android (sdk 2) <-> Firefox : (Firefox get audio and video, Android gets nothing data not tested)

In all case all publications are there but in the 🔴cases, audio and video publications will have "track" set to "null" and isSubscribed to "false".

Hope this helps !

[manjeshbhargav]

Hi @quentinhayot ,

Thanks for writing in with your issue.

For the browser scenarios, can you share how you are using the 2.x Video SDK (code snippets would be ideal)? If you haven't already, please make sure your code refactor is consistent with this migration guide.

Regarding Android SDK v2 issues, I would suggest that you upgrade to the latest Android SDK, since v2 uses a really old version of WebRTC whose DataChannel implementation is incompatible with the later versions of WebRTC.

Thanks,

Manjesh Malavalli
JSDK Team

[quentinhayot]

Sure !
We use Angular 6.
I omitted disconnect / unpublish events and irrelevant code.

import * as TV from 'twilio-video';

export class VisioComponent implements OnInit {
    private localTracks = null;
    [...]

    ngOnInit() {
        [...]
        this.dataTrack = new TV.LocalDataTrack({
            maxPacketLifeTime: null,
            maxRetransmits: null,
            ordered: true,
            reliable: true,
        });
        TV.createLocalTracks({
            audio: {
                workaroundWebKitBug180748: true
            },
            video: true,
        }).then(localTracks => {
            localTracks[1].attach('#localVideo');
            this.localTracks = localTracks;
            this.localTracks.push(this.dataTrack);
            this.startVisio('ROOM ID', 'TOKEN');
        });
    }

    startVisio(roomId: string, token: string) {
        TV.connect(token, {
            name: roomId,
            tracks: this.localTracks
        }).then(room => {

            room.participants.forEach((participant) => {
                this.participantConnected(participant);
            });

            room.on('participantConnected', (participant) => {
                this.participantConnected(participant);
            });
       });

    trackPublished(publication, participant) {
        console.log(`RemoteParticipant ${participant.identity} published a RemoteTrack: ${publication}`);

        publication.on('subscribed', track => {
          console.log(`LocalParticipant subscribed to a RemoteTrack: ${track}`);
          this.addTrack(track);
        });

        publication.on('unsubscribed', track => {
            console.log(`LocalParticipant unsubscribed from a RemoteTrack: ${track}`);
        });
     }

    participantConnected(participant) {
        participant.tracks.forEach(publication => {
          this.trackPublished(publication, participant);
        });

        participant.on('trackPublished', publication => {
          this.trackPublished(publication, participant);
        });

        participant.on('trackUnpublished', publication => {
            console.log(`RemoteParticipant ${participant.identity} unpublished a RemoteTrack: ${publication}`);
        });
    }

    addTrack(track) {
        if (track.kind === 'audio' || track.kind === 'video') {
          track.attach('#remoteVideo');
        } else if (track.kind === 'data') {
          track.on('message', data => {
            console.log(data);
          });
      }
    }
}

On Chrome this would output:

11:47:50.078 visio.component.ts:225 RemoteParticipant Foo published a RemoteTrack: [RemoteAudioTrackPublication #3: MT3e05a28416a06aa0d5b302d48cc11567]
11:47:50.079 visio.component.ts:225 RemoteParticipant Foo published a RemoteTrack: [RemoteVideoTrackPublication #4: MT572365b9c204abe5e11e914d545d7f00]
11:47:50.079 visio.component.ts:225 RemoteParticipant Foo published a RemoteTrack: [RemoteDataTrackPublication #5: MTb2b566cd1bd3a8961e49fb8b48136b89]
11:47:50.233 visio.component.ts:228 LocalParticipant subscribed to a RemoteTrack:  RemoteDataTrack {kind: "data", name: "1d576d9d-81f1-4947-b381-04a92d054dac", isEnabled: true, maxPacketLifeTime: null, maxRetransmits: null, …}

As you can see, only the data track is subscribed to.

[quentinhayot]

Just saw #505
Could it be related ?

[manjeshbhargav]

Hi @quentinhayot ,

In the this.trackPublished() method, sometimes when the RemoteTrackPublication is already subscribed to, you will not get the "subscribed" event. I suggest that you call this.addTrack() on already subscribed RemoteTrackPublications like this:

trackPublished(publication, participant) {
  if (publication.isSubscribed) {
    this.addTrack(publication.track);
  }
  publication.on('subscribed', track => {...});
  publication.on('unsubscribed', track =>{...});
}

Please let me know if this helps.

Thanks,

Manjesh Malavalli
JSDK Team

[zbagley]

@manjeshbhargav

Not sure if this is just an Angular/TypeScript thing, but I hit the same problem (and it's taken me over an hour to isolate the exact issue enough to find this thread). The problem is that publications aren't being subscribed to (publication.isSubscribed returns false) until _after_ the participantConnected event was handled.

The documents should likely be updated to reflect handling the publication.on('subscribed') event to ensure that the publications are subscribed to prior to handling the attach().

Note: Everything was set up as listed in the getting started documentation, except for the necessary Typings.

Relevant code:

  connectVideo() {
    let token;
    this.http.get<any>('/api/token/demo', { params: { id: this.roomIdInput } }).toPromise().then((res) => {
      token = res.token;
      Video.connect(token).then(room => {
        this.room = room; // unable to use @types/twilio-video for room due to EventEmitter conflicts
        this.room.participants.forEach(participant => {
          this.participantConnected(participant);
          console.log(`Participant "${participant.identity}" is connected to the Room`);
        });
        this.room.once('participantConnected', participant => this.participantConnected(participant));
        this.room.once('participantDisconnected', participant => this.participantDisconnected(participant));
      });
    });
  }

  participantConnected(participant) {
    participant.tracks.forEach(publication => {
      if (publication.isSubscribed) { // will always return false
        this.handleTrack(publication.track); // will never get called
        console.log(publication);
        console.log(publication.track);
        console.log(publication.track.attach());
      } else {
        console.log('not subscribed to: ', publication.trackName);
      }
      publication.on('subscribed', track => this.handleTrack(track)); // will resolve the issue
      publication.on('unsubscribed', track => console.log('unsubscribed: ', track));
  });
    console.log(`Participant "${participant.identity}" has connected to the Room`);
  }

  handleTrack(track) {
    console.log(track);
    switch (track.kind) {
      case 'audio':
        this.videoView.nativeElement.appendChild(track.attach());
        break;

      case 'video':
        if (track.name.split('__')[1] === 'self') {
          this.videoView.nativeElement.appendChild(track.attach());
        }
        break;

      case 'data':

        break;
    }
  }

[Frisbeetarian]

@zbagley Thanks for the solution, was stuck on the same issue and this solves it.

[Awem]

@zbagley Thanks, for sharing your solution. However, I think it does not solve the original issue posted by @quentinhayot. I have the same issue with similar browser configurations as described in the original post. None of the suggestions here solve it: if e.g. the Firefox/Chrome browser combination is used, no subscribe event is fired for the tracks/publications.

@manjeshbhargav is this still on the agenda? I see a lot of similar issues:
https://github.com/twilio/twilio-video.js/issues/505
https://github.com/twilio/twilio-video.js/issues/520
https://github.com/twilio/twilio-video.js/issues/553
https://github.com/twilio/twilio-video.js/issues/600
https://github.com/twilio/twilio-video.js/issues/670

This seems to be a major problem. Have you been able to reproduce the issue for the browsers described in the first post? (https://github.com/twilio/twilio-video.js/issues/535#issue-408201725)

This would definitely be a go-live-blocker for our Twilio App :disappointed:

[rpdobson123]

We're also experiencing issues with publications having track = null.

Occasionally it happens chrome-to-chrome, but it's consistently happening when trying to handle remote tracks from Safari in Chrome's perspective.

[tarr11]

I've also experienced this - unlike in @zbagley 's experience, i've found that subscribed never gets called if the participant is already in the room:

 publication.on('subscribed', track => this.handleTrack(track)); 

. However, the publication eventually changes to isSubscribed on its own, but no event is fired.

[tarr11]

FWIW - I've noticed this only happens in peer-to-peer rooms

[manjeshbhargav]

Hi @tarr11 ,

You will most likely have already subscribed to Tracks from existing Participants as soon as you join the Room. So, we recommend in our quick start app and code snippets to handle already subscribed Tracks by cycling through the .tracks collection of the RemoteParticipant and checking the .isSubscribed property of each RemoteTrackPublication:

function handleParticipant(participant) {
  participant.tracks.forEach(publication => {
    if (publication.isSubscribed) {
      handleTrack(publication.track);
    } else {
      publication.on('subscribed', handleTrack);
    }
  });
}

[quentinhayot]

FWIW - I've noticed this only happens in peer-to-peer rooms

I tried with group and group-small : same issue

Hi @tarr11 ,

You will most likely have already subscribed to Tracks from existing Participants as soon as you join the Room. So, we recommend in our quick start app and code snippets to handle already subscribed Tracks by cycling through the .tracks collection of the RemoteParticipant and checking the .isSubscribed property of each RemoteTrackPublication:

function handleParticipant(participant) {
  participant.tracks.forEach(publication => {
    if (publication.isSubscribed) {
      handleTrack(publication.track);
    } else {
      publication.on('subscribed', handleTrack);
    }
  });
}

I've been doing this since the beginning. It does not work !
I can see existing and newly published tracks, but only DataTracks will subscribe...
...and it works with the same code for other browsers (see first post).

[quentinhayot]

I logged everything and everywhere I could, here what's going on: audio and video publication are never subscribed. isSuscribed stays FALSE and "subscribed "events are never triggered.
Only data tracks are correctly subscribed (and their "subscribed" event triggered).

Note that the "subscriptionFailed" event is not triggered either.

[quentinhayot]

The "track" property is always null for audio/video publications. Either when events are fired or 30 seconds later...
Those tracks have a name/sid but the track itself is null.

What could make audio/video tracks to be NULL but data tracks to work correctly ?

[quentinhayot]

Ok, I think I found the issue. It seems to work OK now on all browsers.

In my code, I had the workaround to make Twilio Video work with Safari described here: https://github.com/twilio/twilio-video.js/issues/156#issuecomment-329912344

This code messes with the event listener. If you have this hack in your code, make sure that you are running on Safari first. I use the following code in my Angular application:

private isSafari = /Safari/.test(navigator.userAgent) && /Apple Computer/.test(navigator.vendor);

Then before the hack:

if (this.isSafari) {
      const originalAddEventListener = RTCPeerConnection.prototype.addEventListener;
      [...]
}

This will make sure to set everything up for Safari and don't mess with other browsers.

@tarr11 any chance you are using the same workaround in your code ?

I'll do a few more tests on different platforms and close this issue if everything turns ok.

[Awem]

@quentinhayot Thank you for your investigation. The issue indeed seems to be caused by incompatible event implementations. I am not using the (horrible) workaround you described and still have the issue. However, I am experiencing it on the Crome-Firefox combination (not testing Safari). I have to check, whether something else is interfering here. It is also needed to investigate how the Twilio library utilizes the event implementations.

[quentinhayot]

I'm closing this issue since the problem from my initial post is fixed.

TL;DR; Make sure to check that the browser is Safari if you use the workaround described here: https://github.com/twilio/twilio-video.js/issues/156#issuecomment-329912344

If you don't use this workaround, your problem is different and you should probably open/find another issue.

[manjeshbhargav]

Hi @quentinhayot ,

Yes, the workaround you've linked to is meant to be applied only in Safari with Angular. I'm glad that you were able to get to the bottom of the issue.

Thanks,

Manjesh Malavalli
JSDK Team