Twilio-video.js: Safari A MediaStreamTrack ended due to a capture failure

[phoomin2012]

• [x] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [ ] I verified that the Quickstart application works in my environment.
• [x] I am not sharing any Personally Identifiable Information (PII)
  or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Code to reproduce the issue:
https://gist.github.com/phoomin2012/4b93c2d86d882bccfdf1ffd7d738fe0d

Actual behavior:

User didn't do anything. When the time pass, the localVideoTrack will blank by itself.

After error A MediaStreamTrack ended due to a capture failure produce, the video that attach will be black and localVideoTrack restart by itself and can't handle this error. When localVideoTrack started, the video still black.

Software versions:

• [x] Browser(s): Safari Version 14.0.1 (16610.2.11.51.8)
• [x] Operating System: macOS Big Sur 11.0.1 and iOS 14
• [x] twilio-video.js: 2.7.3
• [x] Third-party libraries (e.g., Angular, React, etc.): Next.JS, Preact

[wpccolorblind]

Seeing this as well, looks like an issue with latest Big Sur / Safari.

[aldovincenti]

I'm not using this library, but I'm experiencing the same issue with latest Safari.

[aldovincenti]

Additional information:

1. This issue is not reproducible in a local environment (I read it here and tested it personally, everything works fine in a local environment).
2. When it happens, the "onended" event on the track is not even fired.

At this point, I think it's definitely a Safari bug. If anyone finds a workaround in the meantime, it would be helpful to share it here. There is a thread in the Apple forum related to this issue, but if anyone knows how to directly contact Apple to report the issue, please do it.

[manjeshbhargav]

Hi @phoomin2012 , @aldovincenti , @wpccolorblind ,

Thanks for reporting this issue. I will investigate on my end and file a WebKit bug if warranted. I'll also look for a workaround.

Thanks,

Manjesh

[t2k2]

having the same issue, hope there's a workaround

[t2k2]

I was able to fix the issue on my end by styling a

[aldovincenti]

@t2k2 thanks for sharing your case, but I guess it might be a different bug. In my case, I'm not using a webGL texture and the error happens a few seconds after the other peer is connected.

[t2k2]

@aldovincenti
No sweat! yea our cases are different but I thought sharing my solution might offer some insight into your issue.

It seems that Apple disabled offscreen video playback in Safari ios14 / Big Sur and this is likely related to your problem.

I'm not sure what technique Twilio uses, but if it renders the camera input to an offscreen (hidden) video element before using it this is likely whats causing the issue.

[wottpal]

Hi @manjeshbhargav, did you already file the WebKit-Bug? We are experiencing the same issue with Safari 14.

[makarandp0]

Hello @aldovincenti @phoomin2012,
Question about the repro steps - Does this issue happen only for MacOS Big Sur video tracks or does it happens for iOS video tracks as well? Which version iOS did you see this on ? I have not been able to reproduce this on my IPad running iOS 14.2 so far.

Also does it repro consistently or once in few attempts? Can you please share any room-sids (and participant sid) that experienced this issue.
Thanks,
Makarand

[aldovincenti]

@makarandp0 it happens on MacOS Big Sur and iOS and it's not possible to reproduce it consistently. I was able to fix it on macOS Big Sur forcing VP8 (so I suppose it's a bug related to H264). I still haven't found a solution for iOS.

I don't think it's related to Twilio, I rather believe it's a webkit bug.

Also the NHS (the public National Health Service of UK) is experiencing the same issue on iOS 14: https://www.liverpoolft.nhs.uk/media/9473/ios-14-and-attend-anywhere-intermittent-compatibility-issues.pdf / https://status.england.nhs.attendanywhere.com/incidents/8yjy9j08q8z0

Great job Apple.

[zachrattner]

For folks running into this on iOS, the latest iOS 14.3 beta does not seem to have this problem. Same code fails with this error on 14.2 and works fine on 14.3.

[mrlika]

It looks for me like a bug in iPadOS / Safari (it has happened to me on iPadOS 14.1). I was able to fix it by changing flow of calls.

One of the way I was able to fix it is by disabling in Twilio code (via hard-coding) creation of dummy HTML video element _dummyEl for each stream.

It happens for me under some circumstances when you create/stop/re-create/attach/detach/re-attach camera streams and HTML video elements. It is not reproducible for me in a standalone environment, only inside a production app. It was reproducible in about 80% of cases (not always).

Investigation shows that camera stream gets ended event from outside. Twilio detects it and tries to re-create the stream. Sometimes it succeeds but sometimes it leads to endless re-creation loop because the stream gets ended event from outside each time after a few seconds of playback.

[konikowska]

We have the same issue on Safari 14.0.1 either on Big Sur or different OS version (e.g. 10.15.6 - Catalina) with the latest Safari (14.0.1).

[boraikizoglu]

I can confirm that I have the same issue on Safari 14.2(iOS)

[mrlika]

I managed to fix the issue by creating camera stream only once.

The flow that has issues on iPadOS:
join page -> create camera stream for preview -> press join -> stop camera stream -> video conference page -> create camera stream again

The flow that has no issues:
join page -> create camera stream for preview -> press join -> pass camera stream -> video conference page

[wottpal]

Still getting the same error on Safari 14.0.2 on macOS 11.1. Also reproducible on AppRTC-demo online. Is there already a public issue / ticket on Safari or WebKit?

[neave]

This issue seems to be fixed in iOS 14.3, at least for me

[dinesh547]

@makarandp0 it happens on MacOS Big Sur and iOS and it's not possible to reproduce it consistently. I was able to fix it on macOS Big Sur forcing VP8 (so I suppose it's a bug related to H264). I still haven't found a solution for iOS.

I don't think it's related to Twilio, I rather believe it's a webkit bug.

Also the NHS (the public National Health Service of UK) is experiencing the same issue on iOS 14: https://www.liverpoolft.nhs.uk/media/9473/ios-14-and-attend-anywhere-intermittent-compatibility-issues.pdf / https://status.england.nhs.attendanywhere.com/incidents/8yjy9j08q8z0

Great job Apple.

Same issue with me as well.
I am facing crash on ios 14.3 device and all ios simulators on Mac big sur. same was working fine with earlier os version

[makarandp0]

Sorry for the delay on this - I recently updated my macbook to Big Sur, and trying to repro this issue - I am running MacOS Big Sur 11.2.1 with Safari Version 14.0.3

If you are still seeing this - can you please post steps to repro this issue for Safari on MacOS?
Thanks,
Makarand

[makarandp0]

related webkit Bug: https://bugs.webkit.org/show_bug.cgi?id=221192

[makarandp0]

Update: we got a response from apple, that this issue is fixed. If you still encounter this after upgrading to iOS 14.3 and MacOS 11.1 or newer, please open another issue.

[dinesh547]

on Big Sur 11.2 i am not seeing the issue on xcode 12.4 but same issue is
still there with xCode11.3. if someone is on Xcoe11.3 & Big sur. can see
the issue on simulators with 3.8.0 version.

On Tue, Mar 23, 2021 at 11:16 PM Makarand Patwardhan <
@.*> wrote:

Update: we got a response from apple, that this issue is fixed. If you
still encounter this after upgrading to iOS 14.3 and MacOS 11.1 or newer,
please open another issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/twilio/twilio-video.js/issues/1281#issuecomment-805102443,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7PCIO3V2H3NYYONNDMBOTTFDHWTANCNFSM4T22F3VQ
.

[bnjkukunuri]

I am facing "Mediastream track ended due to capture failure" error while HTML5 video element has mediaStream attached and when playback has begun. onloadmetadata event call back didnot execute and console shows error "Mediastream track ended due to capture failure". I am facing this on iPhone 14.4.2 version of safari and all other browsers on this iOS.
Details on implementation - the HTML5 video element is rendered and then stream is attached with the result of navigator.mediaDevices.getUserMedia(MediaConstraints), the stream has track enabled true and status as live but upon calling play(), it resulting into error and then call back MediaStreamTrack: ended event is fired after 30secs. When error happens it does not get into reject promise of play callback, play().catch() didnt execute.
It actually first works while capturing pictures using iPhone camera for first 5 pictures and the ramdomly fails there after 5th or 6th photo is capture, video element and canvas is rendered using react component after every photo capture.

Please help!

[dinesh547]

Sorry, I have not worked on twilio video for web/safari. I am more into
native iOS

On Fri, 9 Apr 2021 at 7:07 PM, bnjkukunuri @.*> wrote:

I am facing "Mediastream track ended due to capture failure" error while
HTML5 video element has mediaStream attached and when playback has begun.
onloadmetadata event call back didnot execute and console shows error
"Mediastream track ended due to capture failure".
Details on implementation - the HTML5 video element is rendered and then
stream is attached with the result of
navigator.mediaDevices.getUserMedia(MediaConstraints), the stream has track
enabled true and status as live but upon calling play(), it resulting into
error and then call back MediaStreamTrack: ended event is fired after
30secs. It actually first works while capturing pictures using iPhone
camera for first 5 pictures and the ramdomly fails there after 5th or 6th
photo is capture, video element and canvas is rendered using react
component after every photo capture.

Please help!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/twilio/twilio-video.js/issues/1281#issuecomment-816687924,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7PCILITQLM5DBL2SNQIW3TH37KXANCNFSM4T22F3VQ
.

[bnjkukunuri]

Issue mentioned above from me has been resolved. I made changes to clone the stream object before assigning to video element srcObject and have the MediaStream track stopped end of capture of picture. The stream object is saved in react state and reused on re-render of video element.

[mrlika]

The stream object is saved in react state and reused on re-render of video element

Yes, it is what I noticed - in Safari it is better to not recreate media streams but reuse them.

[bnjkukunuri]

We were re-using the stream earlier too, the code seems working all fine before latest upgrade. The change thats is seems to the fix now is to stop the track right after the each photo capture.

[nitinmadelyn]

This bug can be easily reproduce using following steps:

1. Connect any bluetooth device with your macOS
2. Start a call
3. Disconnect your bluetooth in between call
4. And you can see that video gets disconnected and you see the error in console

@mrlika
You can thank me later, but before that please give some solution to this.