Twilio-video.js: Cannot Access Remote Participant Tracks from Room v1.0.0

[andrewhl]

• [x] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [ ] I verified that the Quickstart application works in my environment.

Code to reproduce the issue:

We've got a React Flux application. We're connecting to the room in one component (a Preflight.jsx view), and trying to attach the remote participant tracks in the componentDidMount of our VideoRoom.jsx component. The room is stored in Flux state and passed down to the VideoRoom via props. We've tried various approaches, but the participant.tracks is empty. In some cases, it has tracks, but the (documented method for adding them)[https://www.twilio.com/docs/api/video/getting-started#use-participant-media] doesn't appear to work. In some debug contexts, like if I log the room object directly, the number of participant tracks for the remote is 2 (after connecting to video), so this may be an async issue.

The localParticipant.tracks are appearing just fine.

    componentDidMount() {
        const room = this.props.room;
        const remoteDiv = document.getElementById('remote-video');
        const localDiv = document.getElementById('local-video');

        room.localParticipant.tracks.forEach(track => {
            localDiv.appendChild(track.attach());
        });

        room.on('participantConnected', function(participant) {
            participant.tracks.forEach(track => {
                document.getElementById('remote-video').appendChild(track.attach());
            });
        });

        room.participants.forEach(participant => {
            participant.tracks.forEach(track => {
                remoteDiv.appendChild(track.attach());
            });
        });
    }

Preflight.jsx

    enterVideoRoom() {
        let videoToken = this.props.videoToken;
        const uuid = this.props.uuid;

        connect(videoToken.token, {
            name: uuid
        }).then(room => {
            VideoActionCreators.enterVideoRoom(room);

            window.onbeforeunload = () => {
                room.disconnect();
            };

            room.on('participantConnected', participant => {
                VideoActionCreators.beginVideoChat();
            });

            room.on('participantDisconnected', (participant) => {
                participant.tracks.forEach(track => {
                    let attachedElements = track.detach();
                    attachedElements.forEach(element => element.remove());
                });
                room.disconnect();
            });

            room.on('disconnected', room => {
                // Detach the local media elements
                room.localParticipant.tracks.forEach(track => {
                    let attachedElements = track.detach();
                    attachedElements.forEach(element => element.remove());
                });
            });

        }, error => {
            console.log('Failed to connect to the room', error);
        });
    }

Expected behavior:

First and second chat participants should see the tracks. The participantTracks size should be 2.

Actual behavior:

The participant remote video is not displaying for the first or second user. participant.tracks doesn't iterate through the forEach, because the size is 0. This is true for participant in the callback to room.on('participantConnected') and room.participants.forEach().

Software versions:

• [x] Browser(s): Chrome and Chrome Incognito
• [x] Operating System: OSX 10.11.16
• [x] twilio-video.js: 1.0.0
• [x] Third-party libraries (e.g., Angular, React, etc.): React

[markandrus]

Hi @andrewhl,

The Tracks will be added asynchronously as WebRTC negotiation succeeds. As you mention, the tracks Maps can initially be empty, after which "trackAdded" events (and "trackRemoved" events) will fire as Tracks are added (or removed). In your example I do not see any "trackAdded" event listeners. Can you try adding these?

Mark

[andrewhl]

@markandrus Hello Mark. Thanks for your response. I added the following code in the componentDidMount of the VideoRoom.jsx component:

        room.on('participantConnected', (participant) => {
            participant.on('trackAdded', track => {
                remoteDiv.appendChild(track.attach());
            });
        });

This successfully displays the remote video to the first participant in the room. When the second participant enters, however, they do not receive the video. I've tried this, also in componentDidMount:

        room.participants.forEach(participant => {
            // participant.tracks.size === 2
            participant.tracks.forEach(track => {
                // this code is never executed
                remoteDiv.appendChild(track.attach());
            });
        });

The inner forEach loop, for whatever reason, isn't executed. If I log participant.tracks in the outer forEach, both tracks are there. Is this the right approach? Am I missing something? My understanding is that the 'trackAdded' event will not fire because the first participant has already entered the room. Thanks!

[markandrus]

Hi @andrewhl,

Glad you were able to make some progress adding the "trackAdded" event. It's a little hard for me to assess your issue with the individual code fragments. In general, you must

1. Handle any Participants already present in the participants Map.
2. Handle any Participant who may connect via the "participantConnected" event.
3. Handle any Tracks already present in a Participant's tracks Map.
4. Handle any Tracks added after the fact via the "trackAdded" event.

In the first code sample of your most recent comment, I see that you handle cases 2 and 4. In the second code sample, I see you handle cases 1 and 3. However, I am not sure if you've tried both, e.g.

// 1. Handle any Participants already present in the `participants` Map.
room.participants.forEach(participant => {

  // 3. Handle any Tracks already present in a Participant's `tracks` Map.
  participant.tracks.forEach(track => {
    remoteDiv.appendChild(track.attach());
  });

  // 4. Handle any Tracks added after the fact via the "trackAdded" event.
  participant.on('trackAdded', track => {
    remoteDiv.appendChild(track.attach());
  });
});

// 2. Handle any Participant who may connect via the "participantConnected" event.
room.on('participantConnected', participant => {

  // 3. Handle any Tracks already present in a Participant's `tracks` Map.
  participant.tracks.forEach(track => {
    remoteDiv.appendChild(track.attach());
  });

  // 4. Handle any Tracks added after the fact via the "trackAdded" event.
  participant.on('trackAdded', track => {
    remoteDiv.appendChild(track.attach());
  });
});

However, if you take this approach, you end up with duplication since you must handle cases 3 and 4 in both the initially-connected (1) and subsequently-connected Participants (3) cases. This is why we factor out a participantConnected function in our README.md's Usage example. Can you please verify that you are handling all 4 of these cases?

[andrewhl]

@markandrus Fantastic. Very helpful. Placing all four of those scenarios in the same file (VideoRoom.jsx) resolved the video mounting issue. Thank you so much.

[jhovad]

Yep, very helpful! :)) Thx @markandrus ! ... It should to be part of quickstart :)