Twilio-video.js: Safari 11 to Firefox on Win 10 frequently failing to stream video.

[MilllerTime]

• [x] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [x] I verified that the Quickstart application works in my environment.
• [x] I am not sharing any Personally Identifiable Information (PII)
  or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Expected behavior:

Video tracks should play between Firefox and Safari, using P2P.

Actual behavior:

Obtaining video tracks between Firefox and Safari over P2P is very unreliable. I'm using Firefox 61 on Windows 10, and both Safari on iOS 11.4.1 and Safari 11.1.2 on Mac OS 10.13.6. Both iOS and Mac versions of Safari seem to exhibit the same issues.

I've attempted over 30 connections, and only 12 attempts worked decently, with video being received and played by both Safari and Firefox. I say decently because each of those 12 times, Firefox displayed either no video or only a freeze frame from Safari for several seconds after connecting to the room, before live video began streaming. But, it eventually worked.

25% of the time, neither participant received video. FF did show a freeze frame, but live video never came through. Safari did not appear to receive a video track at all. Disabling and re-enabling the video track sent from from Safari consistently unfroze the video to FF, but nothing seemed to recover the stream to Safari.

The remainder of the time video streamed fine from FF to Safari, but FF only displayed a freeze frame from Safari. Like I mentioned above, disabling and re-enabling the video from Safari unfroze the video to FF. That's an easy workaround, but it doesn't solve the problem when Safari fails to receive a track from FF.

There was also one oddball case where Safari streamed to FF, but FF didn't stream to Safari. Weirder still, Safari's local video track rendered only black frames, despite appearing fine on the FF client.

Other observations:

• Audio seemed to work consistently, I believe the issue only lies with the video tracks.
• No errors shown in the console, for FF or Safari.
• A connection from Safari to FF on Mac OS seemed to work flawlessly. The above issues only appear when running FF on Windows 10.
• Running the Video Quickstart app, FF would receive a video track from Safari, but Safari always failed to receive a video track from FF. So, it appears the issue isn't only with our site. This is hopefully an easy way to repro on your end as well.

I'm unsure if this is a Twilio issue, or if one of the browsers is misbehaving, but this combination not working is a problem.

Logs:

I captured debug logs from a session on our site where FF showed only a freeze frame from Safari, but the video from FF to Safari worked well. For all I know, the logs will look normal and FF was simply failing to render anything but the first frame. I can try to capture logs of the situation where both participants have issues if needed, though running the quickstart yourself should also show the problem.

iOS.log
Firefox.log

Software versions:

Browser(s): Safari 11.1.2, Firefox 61.0.2
Operating System: iOS 11.4.1, Mac OS 10.13.6, Windows 10
twilio-video.js: 1.13.1
Third-party libraries: React

[manjeshbhargav]

Hi @MilllerTime ,

Thank you for writing to us with your issue. While we check out the logs, can you re-try your tests with [email protected] and [email protected]?

Thanks,

Manjesh Malavalli
JSDK Team

[wesleykendall]

@manjeshbhargav I can verify that I've been testing Safari / Firefox with little to no success. It seems that it is completely random when I can get video to show up, and this is usually after refreshing the connection repeatedly.

I have no problems with Chrome to Chrome and Chrome to Safari. I have some issues with Chrome to Firefox, but nowhere near as pronounced as Safari to Firefox.

I tried updating my 1.14 version to 2.0.0-beta and had loads of other problems pop up. It seems like now there is a newer object that gets passed to the trackPublished event. I fiddled around with that for a while but could not get my video application to work.

Are there any docs on the new changes or example code? All I see is the release notes. I'd like to try to get to the bottom of this too as it seriously affects our product

[manjeshbhargav]

Hi @wesleykendall ,

Here are some resources to help you migrate to Twilio Video 2.x APIs:

• https://media.twiliocdn.com/sdk/js/video/releases/2.0.0-beta1/docs/
• https://www.twilio.com/docs/video/migrating-1x-2x

Thanks,

Manjesh Malavalli
JSDK Team

[MilllerTime]

@manjeshbhargav Thanks for sharing the migration guide, it looks helpful. I've been trying to get the 2.0.0-beta to work as well, with a couple issues I haven't tracked down yet.

I can report that v1.14.0 still has serious issues between Firefox and Safari. I'll let you know how things shake out with the v2 beta asap.

[MilllerTime]

@manjeshbhargav I'm now running 2.0.0-beta1 and experiencing the same kinds of issues between Firefox and Safari in a P2P room. Actually it seems to be worse - the majority of the time I'm seeing no video on either end. Audio still goes through however, and I can even disable & re-enable the video tracks, with the corresponding events being dispatched for all parties (but still no actual video appearing on screen).

I logged the room stats object from Firefox on Windows 10, and this is what the remote tracks from Safari look like:

The codec property is null for both audio and video. The audio track's audioLevel is null, and the video track's dimensions property null while the framerate is 0. Things don't seem right, but surprisingly the audio is playing fine while video is failing to come through.

This is occuring with the following software versions:

Firefox 62.0.2 on an up-to-date Windows 10.
Safari on iOS 11.4.1

I'll report back when I update to iOS 12 (currently having issues with the beta).

I did try using Safari 12 on MacOS (with the same version of Firefox above) and encountered a different outcome. Video streamed fine from Firefox to Safari. However, the video stream from Safari to Firefox only showed a few frames, then froze. If I disabled then re-enabled the video track from Safari, a few more frames would play on Firefox and then freeze again.

Obviously we really want to see Safari <--> Firefox working, but thanks for the ongoing effort on v2!

[wesleykendall]

Do you have any suggestions on how to reliably detect when this state occurs @MilllerTime or @manjeshbhargav? We've had a few users report this issue to us and we don't know how to detect it at the moment.

[MilllerTime]

@wesleykendall I was looking into this myself, but haven't found a reliable method. I was attempting to query the room stats several seconds after a remote video track was subscribed to, in order to check its dimensions and framerate. Turns out this information isn't reliable across browsers. The stats feature works great in Chrome, but not as well in FF and not at all in Safari. It's unfortunate that the devices with issues also can't reliably detect them.

So, I'd also love to know if there's a way to detect this until the root problem is resolved.

[vincentwoo]

Would appreciate a twilio rep commenting on the current state of affairs here. Is this a known unreliability issue?

[MilllerTime]

I've been testing Firefox 63.0.1 on Windows 10 with Safari on iOS 12.0.1. The same types of issues described above are still occuring the majority of the time.

Would also appreciate an official comment on this, whether anything is known or not. I hate to be a bother, but it's been a little while now. @markandrus or @rfbrazier, do either of you have any info about this issue?

[ZelStudio]

Same issue ... Mac OS Safari or firefox and iOS 11.
Many of our clients are unhappy ! Twilio, we pay for it !

[ahey]

We have similar issues here, the common theme seems to be Firefox. Whenever Firefox is used for one end of the call, we experience the video freezing. For now we have dropped support for Firefox entirely until it's fixed.

[ZelStudio]

iOS Phones too ... 11.4.1 and may be other too

[anna-vasilko]

@MilllerTime @ahey @ZelStudio Thank you for the reports! We have not seen overall Firefox <-> Safari incompatibility in our previous testing. But we are digging more. The fact that for you it happens only in Firefox and affects only video while audio is fine, makes us think it's some sort of performance issue in Firefox. It would really help if you can:

• share few fresh room sids and browser logs
• check how much CPU is used by FF on the machine when the issue happens
• does it happen only when Firefox is used on Windows for all of you, or Mac as well?

[ZelStudio]

Hi, @anna-vasilko , not only firefox but also iOS device (iPhone7 ios 11.4.1) and chrome on MacOS. Switching Camera also went to black. Those issues are new since September, all was fine before.

[anna-vasilko]

@MilllerTime We have identified a couple of interoperability issues between Firefox and other browsers related to the fact that Firefox is on "unified" plan and Chrome and Safari are on "plan b". Those are treated as a top priority by the team. Would still help to receive a few fresh room sids from you to be able to confirm if your issue is no different.

[anna-vasilko]

@ZelStudio This thread is tracking issues specifically between Firefox and Safari. Your comment makes me think that your issue is different if there is no Firefox in the picture. It is impossible to diagnose without having a clear description of the use case, room sids and logs. Can you please provide those?
So far we have not heard of any issues in cases that do not involve Firefox (like Chrome to Chrome or Safari to Safari), that is considering thousands of customers using this product. Most probably you are doing something different and we would like to understand what. Please provide:

• detailed description of the use case
• what browsers and OS are used on each side when the issue occurs
• a couple of room sids for the affected video rooms
• browser logs

[MilllerTime]

@anna-vasilko I ran a few tests, all of which exhibited the problems I've described above. Details and room SIDs below.

Twilio 1.14.1: iOS 12.1; Safari <==> Win 10; Firefox 63.0.3
RMda5b8426bbc49371e16ed251bcd08d1f

Twilio 2.0.0-beta2: iOS 12.1; Safari <==> Win 10; Firefox 63.0.3
RMff96b33a45ae8bfa101a7278ee0ecd1b

Twilio 2.0.0-beta2: MacOS 10.13.6; Safari 12 <==> Win 10; Firefox 63.0.3
RMfad13e8b3d89d459a9cfdbcebaaffe40

[anna-vasilko]

@MilllerTime Thank you for the roomsids! Super helpful. We are checking right now if you are hitting the same issue that we have in progress.

[anna-vasilko]

@MilllerTime Fixes for the Firefox - Safari issue are planned to be released this week. We'll keep you posted.

[syerrapragada]

@MilllerTime We've released 2.0.0-beta4 which includes the fix for the Firefox - Safari issue. Would you please test it out and let us know if it fixes the issue?

Also, we are currently working on 1.x patch release. We'll keep you posted.

Thanks,
Sreeram

[MilllerTime]

@syerrapragada The 2.0.0-beta4 release resolves the issues for us! Thanks for getting this resolved!

Would definitely appreciate a 1.x patch as well 👍

[syerrapragada]

@MilllerTime We've released 1.14.2 which includes the fix for Firefox - Safari interop issue.

[MilllerTime]

The 1.x patch seems to be working well. Thanks again!