Twilio-video.js: (Video|Audio)Track.isStarted

[Westbrook]

I'm trying to get a better understanding of the isStarted property of a video or audio track and the associated started event and was wondering if there was better documentation than The VideoTrack started. This means there is enough video data to begin playback. available for review in this area.

Particularly, I've run into instances where a track is listed as isStarted=false and I'm:

1. unsure why
2. desiring to start() the track
3. interested in whether I need to manage a more complex API surface to correct this

I've had no trouble adding a screen share track in cases where the main video remained as isStarted=false, so possibly I should do some sort of listening for the started event and then replace after some interval? Is there some better way to manage tracks that don't flip into isStarted=true.

For instance, in RM4a1e8d3d1ca28f834452532112735433, I had two participants connected with audio and video tracks and one (PA882a18e1cd2459ba86f66a79dd28317d) delivering a shared screen for a time. However, while the initial video track for PA882a18e1cd2459ba86f66a79dd28317d was listed in getVideoTracks() of participants, it never changed to isStarted=true and now there is no reference to that track in the recordings. I'm also interested if there would be somewhere in my dashboard that I could see that track, even though data for is doesn't exist. It made it to the other participant, so it seems that it would be logged somewhere, right?

Thanks in advance for your assistance

[markandrus]

Hi @Westbrook,

If isStarted remains false on a RemoteTrack, there's typically an error (unless you intentionally disconnect from a Room before a RemoteTrack has a chance to start). The most common error that will cause isStarted to remain false is a media connection (ICE connection) failure. So, if you receive a RemoteTrack, e.g., in the "trackPublished" (or "trackAdded") event, but that RemoteTrack never raises a "started" event, then something is wrong.

I've had no trouble adding a screen share track in cases where the main video remained as isStarted=false, so possibly I should do some sort of listening for the started event and then replace after some interval? Is there some better way to manage tracks that don't flip into isStarted=true.

Sorry, I'm not sure I understand: you're saying that, although a "main" (e.g., webcam) RemoteVideoTrack remained not started, you were able to add a screen share (and it did start?). That would indicate some other error than an ICE connection, and we'd appreciate a bug report with a Room SID we could look into.

Regarding whether or not you should do something to manage RemoteTrack, our goal would be that you would not have to and to instead raise, e.g., a "reconnecting" event if we detect your media connection failed and needs to be repaired (something we're working on). You could, though, set a timeout and wait, e.g., up to 10 seconds to see if a RemoteTrack ever does start. If not, it would indicate a problem you may want to raise in the UI. The getStats API may also provide information here (e.g., if you see bytesReceived is zero or constant, there again may be a media connection error).

We will look into RM4a1e8d3d1ca28f834452532112735433 on the backend.

Thanks,
Mark

[Westbrook]

Thanks for the response @markandrus

Yes, for the call RM4a1e8d3d1ca28f834452532112735433 on participant was delivering both audio and video 100% of their connection and the other was delivering one active audio track and one isStarted=false video track from the beginning of the call. Later in the call, that second participant added a shared screen that transmitted fine while the initial video track never emitted a started event.

This is something that I'd been told about experientially, but have only recently been able to confirm during an active connection, so it's not a singular problem, though I'm only 80% sure it's been the same on these other calls. This SID seems to have experienced this same issue if it is helpful in your research: RMfdf8c1876acaa0af0a6f4cf705c04c1f

[markandrus]

@Westbrook thanks for sharing the additional SID. In Room RM4a1e8d3d1ca28f834452532112735433, it appears that the VideoTrack that failed to send had SID MTb59dbce153e29adf4e2c83673cc92cde with ID 776b937a-9714-480f-b27c-4e1e1012e6cf and SSRC 2711352439. According to our internal stats reports,

• The LocalParticipant publishing this track reported all zeroes for bytesSent, dimensions.width, dimensions.height, frame_rate (very weird)
• Our media server reported zero bytes received (makes sense, if bytesSent is zero).
• RemoteParticipants reported zero byes received (also makes sense, if bytesSent is zero).

I guess there is no recording for MTb59dbce153e29adf4e2c83673cc92cde because no frames were ever received. Still, you should have some kind of indication that this is the reason. I am going to file a feature request for this with our Video Recordings Team. (EDIT: That feature request is VRS-639.)

Next, we need to figure out why bytesSent remained at zero. It's suspicious that the subsequently added VideoTrack works (and the original AudioTrack worked). So that rules out a total ICE/media connection failure. I think there might be either 1. an application bug (e.g., that stops the LocalTrack or its underlying MediaStreamTrack before publishing), or 2. a Chrome bug in renegotiation. For 1., do you know what type of VideoTrack MTb59dbce153e29adf4e2c83673cc92cde was? e.g., a screen share or a webcam? For 2., I will try to create a JSFiddle that reproduces the negotiation in Chrome.

[Westbrook]

@markandrus Thanks for the extensive response.

1) The track in question would have be delivered from a webcam. It would have originated from Chrome 65.0.3325.181 via OSX Intel 10.13, if that's any help.

2) Look forward to your Fiddle findings. Let me know if there is any further information that I can supply you in this area.

Thanks!

[markandrus]

Here's the JSFiddle I created. I was suspicious about the m=video section initially being inactive (if you were to look at the SDPs in the console, you'd see a=inactive, then, once video is added, a=sendrecv); however, this doesn't seem to cause issues. I was also suspicious of the apparent codec reordering (I guess you're using preferredVideoCodecs: ['H264']?), so I also added some SDP munging to filter down to H.264-only. Still, I can't reproduce the case I saw in the logs (video works, and even if I stop it, I don't see frameRateInput above zero).

One other thing I should have mentioned: in both RM4a1e8d3d1ca28f834452532112735433 and RMfdf8c1876acaa0af0a6f4cf705c04c1f I see VideoTracks with bytesSent at zero, dimensions at zero, etc.; however, in both, I do see frameRateInputs above zero (~30 and ~15, respectively).

[Westbrook]

We actually use the VP8 & H.264 if that makes any difference.

In both of those rooms noted, if the frameRateInputs>0 track the same as the bytesSent=0 track? Could that then be an issue with the webcam returning itself as part of the getUserMedia constraints but not being fully connected/accessible somehow? What would be the best ways to debug this on our end if we run into this again in a testable situation? I'd love to say we could get to a fully repeatable situation, but in that users that have reported this has reported full connectivity on other occasions with the same remote participants, I'm not bullish on doing so.

[markandrus]

In both of those rooms noted, if the frameRateInputs>0 track the same as the bytesSent=0 track?

Yes.

Could that then be an issue with the webcam returning itself as part of the getUserMedia constraints but not being fully connected/accessible somehow?

That's what I wonder. For customers experiencing this issue, do you know if they can see themselves or not in the app? Do you show local video and/or ask the user if they can see themselves? If local video is not playing, that would at least rule out any weirdness due to WebRTC.

[Westbrook]

We do show the local video locally, so maybe that rules that out. :/

I should be back to looking at this part of our code tomorrow and the next day, so hopefully I can find some added insight with your notes to date.

[scitech]

Hello there, I believe we're experiencing a similar issue, and would like to offer room SID RM3ab1cc0bf566afc622ba72e9f7bb6663 if it would help you debug.

Our symptoms:

• two users successfully subscribing to each other's cam/microphone
• one user publishes a screen track, and can see the video locally
• the second user can't see the screen track, which is enabled and subscribed, but isStarted=false.

@markandrus , can you determine from internal stats if this is indeed the same issue?

In our experience, we've only seen the issue when a Chrome user is publishing their screen stream to a Firefox user, but I'm not 100% confident that this isn't a coincidence.

[Westbrook]

Upon further review, I'm able to get into this state more frequently with 3+ users on a call, particularly when the 3rd+ user joins a bit delayed. That is to say, the second user usually has much less issue getting trackStarted to fire, but the third user is more likely to have issues. This seems exacerbated in FireFox, but that feedback is purely experiential right now. It is almost as if the trackStarted even once triggered in one browser is having trouble getting triggered again in a second one, but I have no real data on how that would be. I'm working right now on some report functionality so my users and I can stash a more complete export of the Twilio data state/browser data when this is occurring.

@markandrus Hopefully this will give you more actionable data to follow up on this.

[scitech]

@Westbrook we determined (w/help from Twilio support) that our issue was related to a bug in the way Firefox negotiates incoming data from hybrid codec rooms. Switching to either VP8- or H264-only rooms in the Twilio dashboard resolved the problem for us. May or may not be feasible depending on your browser support requirements, but thought it might be helpful to somebody to share.

[Westbrook]

Thanks @tylergould that's very interesting. It's been too long since I checked, but in the web while there are some quality/performance implications, H264 is supported pretty well in FF/Chrome/Safari, right? I'll see where we can get with that.

I am also running into this sometime with no FF participant, but maybe I'll be able to get the data needed to hunt it down when using a single codec.

[markandrus]

@tylergould thanks for the extra context. To re-iterate:

If you're having trouble with Firefox in Group Rooms using both VP8 _and_ H.264, please use only H.264 _or_ VP8. This will work around Firefox Bug 1333879 - Receiving multiple codecs in a SDP answer does not work until we have a proper fix in our media server.

@Westbrook

It's been too long since I checked, but in the web while there are some quality/performance implications, H264 is supported pretty well in FF/Chrome/Safari, right? I'll see where we can get with that.

Yes, it should be pretty well-supported. On mobile or more constrained devices, you sometimes want to be more thoughtful in the codec you choose, based on the availability of hardware encoders. Please let me know if you see an improvement.

I am also running into this sometime with no FF participant, but maybe I'll be able to get the data needed to hunt it down when using a single codec.

An example of this (no FF participant) would be useful, too. The original Room SIDs you shared have expired, so I can't re-evaluate whether or not those were due to Firefox Bug 1333879. If you encounter this with Chrome-only, then it may even be a separate issue. But more Room SIDs will be helpful.

[Westbrook]

Thanks for the update @markandrus. I'll be putting together the some quick reporting for us on this mid-to-late week, so that I should have more actionable data for you then.

[Westbrook]

In the context of a H264 only room, I should be able to transmit all tracks between all users? Regardless of their browser?

This SID RM3cf72003c6fd065a54a8a490e08047a5 is not experiencing this. There were even cases when a Firefox originated users' tracks had isStarted and isEnabled listed as true but the data was not being delivered to other (Chrome) users. Locally the Firefox user had issues capturing all tracks. Trying to work out the right way to support conversations x-browser while the issue of multiple codecs continues.