Cordova-plugin-iosrtc: Black Bars around video stream and video stream is stuck when scrolling

Created on 22 Sep 2020 · 10Comments · Source: cordova-rtc/cordova-plugin-iosrtc

[x] I have used Google with the error message or bug in association with the library and Cordova words to make sure the issue I'm reporting is only related to iOSRTC.
[x] I have provided steps to reproduce (e.g. sample code or updated extra/renderer-and-libwebrtc-tests.js file).
[x] I have provided third party library name and version, ios, Xcode and plugin version and adapter.js version if used.

Versions affected

Cordova version (e.g 7.1.0): 10.0.0
Cordova iOS version (e.g 5.1.0): 6.1.1
Plugin version (e.g 6.0.12): 6.0.13
iOS version (e.g 10.2): 13.7
Xcode version (e.g 11.1 - 11A1027): 12.2 Beta
WebRTC-adapter version (e.g. 7.4.0):
WebRTC Framework version (e.g. JSSip 3.1.2):

Description

I'm not able to get a 4/3 video atm. It's also stuck to the screen as seen in the video below. I tested the same javascript code with on a hosted environment and accessed it with Safari. It worked perfectly fine. Below is a short video of what I encountered.

https://youtu.be/KiO-VVzpV-w

Steps to reproduce

If you run this code, you should be able to reproduce. I've tried numerous combinations of the constraints in getUserMedia using back and front cameras alongside different resolutions to no avail.

 cordova.plugins.iosrtc.registerGlobals();
 var appContainer = document.body;
  var localVideoEl;
  navigator.mediaDevices
    .getUserMedia({
      audio: false,
      video: {
        width: 640, 
        height: 480,
        aspectRatio: 4 / 3,
      },
    })
    .then(function (stream) {
      localVideoEl = document.createElement("video");
      localVideoEl.style.width = "640";
      localVideoEl.style.height = "480";
      localVideoEl.setAttribute("autoplay", "autoplay");
      localVideoEl.setAttribute("playsinline", "playsinline");
      localVideoEl.srcObject = stream;
      appContainer.appendChild(localVideoEl);

})

Expected results

A 4/3 video that is not stuck to the screen. I also fed the video in a image recognition model and the confidence score was super low. I think it's because it's feeding in those black bars as well since I assume its part of the video. I tested the same javascript code on a hosted environment and accessed it with Safari. It worked perfectly fine.

Actual results

See video below:
https://youtu.be/KiO-VVzpV-w

help wanted

Source

aspdev91

All 10 comments

I did some further digging and came across this on xCode when the video appears:

2020-09-22 20:55:49.290496-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x282466120 h=--& v=--& UIView:0x10635d6a0.minY == 0   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSLayoutConstraint:0x282465ea0 UIView:0x10635d6a0.top == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.top   (active)>",
    "<NSLayoutConstraint:0x2824648c0 'UIViewSafeAreaLayoutGuide-top' V:|-(44)-[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide']   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x282465ea0 UIView:0x10635d6a0.top == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.top   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.291470-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x282466a80 h=--& v=--& UIView:0x10635d6a0.width == 0   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x282463c50 h=-&- v=-&- WKWebView:0x10700a400'React App'.minX == 0   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x2824639d0 h=-&- v=-&- H:[WKWebView:0x10700a400'React App']-(0)-|   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSLayoutConstraint:0x282465db0 UIView:0x10635d6a0.left == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.left   (active)>",
    "<NSLayoutConstraint:0x282466210 UIView:0x10635d6a0.right == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.right   (active)>",
    "<NSLayoutConstraint:0x282462da0 'UIView-Encapsulated-Layout-Width' UIView:0x106306fc0.width == 375   (active)>",
    "<NSLayoutConstraint:0x282464730 'UIViewSafeAreaLayoutGuide-left' H:|-(0)-[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'](LTR)   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSLayoutConstraint:0x282465270 'UIViewSafeAreaLayoutGuide-right' H:[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide']-(0)-|(LTR)   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x282466210 UIView:0x10635d6a0.right == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.right   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.292211-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x282466120 h=--& v=--& UIView:0x10635d6a0.minY == 0   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSAutoresizingMaskLayoutConstraint:0x282467f20 h=--& v=--& UIView:0x10635d6a0.height == 0   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x282463e30 h=-&- v=-&- WKWebView:0x10700a400'React App'.minY == 0   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x282462e90 h=-&- v=-&- V:[WKWebView:0x10700a400'React App']-(0)-|   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSLayoutConstraint:0x282465e50 UIView:0x10635d6a0.bottom == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.bottom   (active)>",
    "<NSLayoutConstraint:0x2824628a0 'UIView-Encapsulated-Layout-Height' UIView:0x106306fc0.height == 812   (active)>",
    "<NSLayoutConstraint:0x282464910 'UIViewSafeAreaLayoutGuide-bottom' V:[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide']-(34)-|   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x282465e50 UIView:0x10635d6a0.bottom == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.bottom   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.293183-0400 Revbod[4460:887487] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-09-22 20:55:49.293296-0400 Revbod[4460:887487] PluginMediaStreamRenderer#refresh() [elementLeft:8.0, elementTop:562.0, elementWidth:300.0, elementHeight:150.0, videoViewWidth:300.0, videoViewHeight:150.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2020-09-22 20:55:49.293481-0400 Revbod[4460:887487] iosrtcPlugin#MediaStreamRenderer_render()
2020-09-22 20:55:49.293532-0400 Revbod[4460:887487] PluginMediaStreamRenderer#render()
2020-09-22 20:55:49.293849-0400 Revbod[4460:887487] iosrtcPlugin#MediaStreamRenderer_close()
2020-09-22 20:55:49.293897-0400 Revbod[4460:887487] PluginMediaStreamRenderer#close()
2020-09-22 20:55:49.293930-0400 Revbod[4460:887487] PluginMediaStreamRenderer#reset()
2020-09-22 20:55:49.294259-0400 Revbod[4460:887487] iosrtcPlugin#new_MediaStreamRenderer()
2020-09-22 20:55:49.294313-0400 Revbod[4460:887487] PluginMediaStreamRenderer#init()
2020-09-22 20:55:49.296542-0400 Revbod[4460:887487] PluginMediaStreamRenderer#run()
2020-09-22 20:55:49.296725-0400 Revbod[4460:887487] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-09-22 20:55:49.296812-0400 Revbod[4460:887487] PluginMediaStreamRenderer#refresh() [elementLeft:8.0, elementTop:562.0, elementWidth:300.0, elementHeight:150.0, videoViewWidth:300.0, videoViewHeight:150.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2020-09-22 20:55:49.298488-0400 Revbod[4460:887487] iosrtcPlugin#MediaStreamRenderer_render()
2020-09-22 20:55:49.298548-0400 Revbod[4460:887487] PluginMediaStreamRenderer#render()
2020-09-22 20:55:49.299335-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x28241aa80 h=--& v=--& UIView:0x10625bf60.minY == 562   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSAutoresizingMaskLayoutConstraint:0x282463e30 h=-&- v=-&- WKWebView:0x10700a400'React App'.minY == 0   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x282462e90 h=-&- v=-&- V:[WKWebView:0x10700a400'React App']-(0)-|   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x28241ada0 h=--& v=--& UIView:0x10625bf60.height == 150   (active)>",
    "<NSLayoutConstraint:0x28241aad0 UIView:0x10625bf60.bottom == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.bottom   (active)>",
    "<NSLayoutConstraint:0x2824628a0 'UIView-Encapsulated-Layout-Height' UIView:0x106306fc0.height == 812   (active)>",
    "<NSLayoutConstraint:0x282464910 'UIViewSafeAreaLayoutGuide-bottom' V:[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide']-(34)-|   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x28241aad0 UIView:0x10625bf60.bottom == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.bottom   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.300108-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x28241aa80 h=--& v=--& UIView:0x10625bf60.minY == 562   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSLayoutConstraint:0x28241ae90 UIView:0x10625bf60.top == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.top   (active)>",
    "<NSLayoutConstraint:0x2824648c0 'UIViewSafeAreaLayoutGuide-top' V:|-(44)-[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide']   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x28241ae90 UIView:0x10625bf60.top == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.top   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.301595-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x282418c80 h=--& v=--& UIView:0x10625bf60.minX == 8   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSLayoutConstraint:0x28241abc0 UIView:0x10625bf60.left == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.left   (active)>",
    "<NSLayoutConstraint:0x282464730 'UIViewSafeAreaLayoutGuide-left' H:|-(0)-[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'](LTR)   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x28241abc0 UIView:0x10625bf60.left == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.left   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.302511-0400 Revbod[4460:887487] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x282418c80 h=--& v=--& UIView:0x10625bf60.minX == 8   (active, names: '|':WKWebView:0x10700a400'React App' )>",
    "<NSAutoresizingMaskLayoutConstraint:0x28241ae40 h=--& v=--& UIView:0x10625bf60.width == 300   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x282463c50 h=-&- v=-&- WKWebView:0x10700a400'React App'.minX == 0   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x2824639d0 h=-&- v=-&- H:[WKWebView:0x10700a400'React App']-(0)-|   (active, names: '|':UIView:0x106306fc0 )>",
    "<NSLayoutConstraint:0x28241ac10 UIView:0x10625bf60.right == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.right   (active)>",
    "<NSLayoutConstraint:0x282462da0 'UIView-Encapsulated-Layout-Width' UIView:0x106306fc0.width == 375   (active)>",
    "<NSLayoutConstraint:0x282465270 'UIViewSafeAreaLayoutGuide-right' H:[UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide']-(0)-|(LTR)   (active, names: '|':WKWebView:0x10700a400'React App' )>"
)

Will attempt to recover by breaking constraint 
<NSLayoutConstraint:0x28241ac10 UIView:0x10625bf60.right == UILayoutGuide:0x283e49f80'UIViewSafeAreaLayoutGuide'.right   (active)>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-09-22 20:55:49.361612-0400 Revbod[4460:887487] PluginMediaStreamRenderer | video size changed [width:480.0, height:640.0]
2020-09-22 20:55:49.363154-0400 Revbod[4460:887487] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-09-22 20:55:49.363259-0400 Revbod[4460:887487] PluginMediaStreamRenderer#refresh() [elementLeft:8.0, elementTop:562.0, elementWidth:300.0, elementHeight:150.0, videoViewWidth:113.0, videoViewHeight:150.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]

The dimensions in that last part seems a bit off..

aspdev91 on 23 Sep 2020

@aspdev91 The layout constraints warning have nothing to do with your issue see https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/422 that why i have hide your comment above.

If you want to update video location on scrool you need to call cordova.plugins.iosrtc.refreshVideos.

Here is an example to do that automaticly. We may add that by default later.

// Scan every 500ms for refreshing video tag position and on various user events.
var scanVideoTagTimer,
    scanVideoTagInterval = 500;

function scanVideoTag() {
    clearTimeout(scanVideoTagTimer);
    var shouldRefreshingVideos = $window.document.querySelectorAll('video').length > 0;
    if (shouldRefreshingVideos) {
        $window.cordova.plugins.iosrtc.refreshVideos();
    }
    scanVideoTagTimer = setTimeout(scanVideoTag, scanVideoTagInterval);
};

// Start scanVideoTag
scanVideoTagTimer = setTimeout(scanVideoTag, scanVideoTagInterval);

window.onorientationchange = scanVideoTag;
window.addEventListener('touchstart', scanVideoTag);
window.addEventListener('click', scanVideoTag);
window.addEventListener('touchmove', scanVideoTag);
window.addEventListener('touchend', scanVideoTag);

hthetiot on 23 Sep 2020

@aspdev91 The black bar issue you have on capture can be related to this bug

https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/570

I do recomand you try master until 6.0.14 is released and this but with it:

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#master --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

hthetiot on 23 Sep 2020

Let me kow if this 2 thing works for you and i will close the issue.

hthetiot on 23 Sep 2020

Notice that to get Image to canvas you need to use special iosrtc code see example here:

https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/extra/renderer-and-libwebrtc-tests.js#L405

hthetiot on 23 Sep 2020

@hthetiot Thank you for the help on this.. This was a blocker for us.

The scrolling issue is resolved with the code you provided.

For the image canvas, I tried to run the test code and got this error in Safari debug tools:

Unhandled Promise Rejection: TypeError: undefined is not an object (evaluating 'localVideoEl.render.save')

For the black bar, I was on latest master when I made this post and I just pulled master again using your command. Unfortunately, the black bars are still there.

aspdev91 on 23 Sep 2020

For the image canvas, I tried to run the test code and got this error in Safari debug tools:

localVideoEl need to be your element and you need to propably do that in a setTimeout or after the video canplay event.

For the black bar, I was on latest master when I made this post and I just pulled master again using your command. Unfortunately, the black bars are still there.

Please create separate issue that would help to keep issue scope.

hthetiot on 24 Sep 2020

Black Bars around video stream

Did you use object-fit also it will depend the size of your video tag

hthetiot on 24 Sep 2020

@hthetiot The object fix resolved the black bars! Thanks!

After some tinkering, the canvas is working as well. Thanks @hthetiot

I have a follow up question on alternatives for the canvas as this adds performance overhead. I'll post it a separate issue.

aspdev91 on 24 Sep 2020

🎉1

@aspdev91 glad it is solved for you, I plan to add SHIM to emulate Original Stream to Canvas in the future.

For object-fit CSS here is quick reference for people landing on this issue:
https://css-tricks.com/almanac/properties/o/object-fit/

hthetiot on 25 Sep 2020

❤1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Failed to get current SOC Generation string and crash on device with audio hardware issue

sboudouk · 12Comments

Calls not constantly working - working/not working loop

outEduardoSilva · 11Comments

Getting : TypeError: undefined is not an object (evaluating 'stream.id') when removing a shared screen

magestican · 11Comments

Webrtc-adapter mediaConstraints.video.facingMode does not select correct camera

smuddy · 15Comments

Not implemented Error for MediaStreamTrack.getConstraints|applyConstraints

j4hangir · 10Comments