Cordova-plugin-iosrtc: iOS 14.3 getUserMedia fails

[Gregadeaux]

Please read first!

Please use Public Google Group (mailing list) for general technical discussions and questions.

• [x] I have used Google with the error message or bug in association with the library and Cordova words to make sure the issue I'm reporting is only related to iOSRTC.
• [x] I have provided steps to reproduce (e.g. sample code or updated extra/renderer-and-libwebrtc-tests.js file).
• [x] I have provided third party library name and version, ios, Xcode and plugin version and adapter.js version if used.

Note: If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue.

Versions affected

• Cordova version 10.0.0
• Cordova iOS version 6.1.1
• Plugin version 6.0.16
• iOS 14.3
• twilio-video plugin 2.7.3

Description

On iOS 14.3, when I try to call any method on twilio-video.js that would call getUserMedia (like createLocalTracks, createLocalVideoTrack, or createLocalAudioTrack), I receive an error. This reads like a permissions error, but permissions are accepted when they pop up, and the error occurs after the last permission prompt is accepted.

WARN in [createLocalTracks #2]: Call to getUserMedia failed: NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

This does not occur on older versions of iOS.

Steps to reproduce

Try and create a localTrack on iOS 14.3

Expected results

I would expect it to call getUserMedia in the plugin

Actual results

It does not appear to even be reaching iosrtc's getUserMediaPlugin file, it just fails immediately.

[Gregadeaux]

Addition: Does not seem to be related to twilio-video at all. If you do navigator.mediaDevices.getUserMedia you get the error.

[Gregadeaux]

navigator.webkitGetUserMedia does successfully execute inside this plugin though.

[hthetiot]

please provide "I have provided steps to reproduce", you need to make sure you call "cordova.plugins.iosrtc.registerGlobals();" before loading twilio client javascript.

See registerGlobals:
https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/blob/master/www/js/common.js#L639

Then Inject twilio Client.
https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/blob/master/www/js/index-twilio.js#L7

Unless you provide exact steps we cannot make sure its not setup issue, that is required to inject twilio afterward bacause twilio keep internal reference to WebRTC global instead of refer to them again.

try to make https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/ work locally, try make sure your setup is correct, you need to edit www/index.html to use index-twilio.js instead of index-local.js and set twilio token in index-twilio.js.

We are not doing twilio support, if you have issue with twilio refer to twilio as customer and request support.

[hthetiot]

Just tested iosrtc sample with index-local.js on 14.3, no issue the issue is from bad twilio+iosrtc usage or new twilio+iosrtc issue.

[Gregadeaux]

I will try and run the sample, but as stated in a comment, I have removed twilio from the test and am seeing the same error. I do not receive that error on lower versions of iOS with the same app. So it is unrelated to twilio.

This is what is in my app.component.ts constructor

```
this.platform.ready().then(() => {
this.statusBar.styleDefault();
this.splashScreen.hide();
this.screenOrientation.lock('landscape').catch(e => e);

  if (this.platform.is('ios')) {
    if (window["cordova"] && window["cordova"]["plugins"] && window["cordova"]["plugins"]["iosrtc"]) {
      cordova.plugins.iosrtc.registerGlobals();
      cordova.plugins.iosrtc.debug.enable('*', true);


      // // load adapter.js
      // const adapterVersion = 'latest';
      // const script = document.createElement('script');
      // script.type = 'text/javascript';
      // script.src = 'https://webrtc.github.io/adapter/adapter-' + adapterVersion + '.js';
      // script.async = false;
      // document.getElementsByTagName('head')[0].appendChild(script);
    }
  }```

[hthetiot]

Let me recheck

[hthetiot]

iOS 14.3 does bring getUserMedia support to WKWebview see https://bugs.webkit.org/show_bug.cgi?id=208667

We may need to perform better feature detection but over time you should not need this plugin to call registerGlobals, plugin can be useful for audio output and some other reasons such as support for iOS 10 to 14.2.

You need to request the permission properly, must likely using dedicated iosRTC api may work see request permission api see https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/docs/iosrtc.md#iosrtcrequestpermission

[hthetiot]

Tested Sylaps App on iOS 14.3 and works without changes from the store.

I think you just need to use the permissions API I mentioned prior.

Officials Apple note for 14.3 https://webkit.org/blog/11353/mediarecorder-api/

[hthetiot]

Permissions capture on Sylaps ios and working on iOSRTC 6.0.16

[hthetiot]

I will try and run the sample

@Gregadeaux , please confirm the issue with the sample, i confirmed using my app aka Sylaps for ios and it works see capture above.

[SzymonWS]

I'm facing the same issue on iOS 14.3. My Ionic App uses 6.0.16 version of plugin. When I call navigator.mediaDevices.getUserMedia({video: false, audio: true}); I get an error:

NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission

What's interesting - when before getUserMedia I call requestPermisson function, I get info, that permission is approved.

What's also interesting - when I run my app on --debug, there is no problem. Problem occurs, on the --prod build.

[Qvadis]

Facing the same problem here since the last software upgrade (iOS 14.3).

We are not using twillio, but we are getting the getUserMedia error as well.

[samgabriel]

Same here

[samgabriel]

@hthetiot you need to try to stop sharing and then start the sharing again that is when it happens. On the second call to getUserMedia

[samgabriel]

@hthetiot this is not an iosrtc issue. I think apple screwed something up. even if we completely disable registerGlobals the problem still happens with the new builtin getUserMedia

[pethessa]

We are seeing the same issue and see this on the device console (despite it saying in JS that permissions are granted). Has anyone filed a bug with apple, if so what is the tracker#?

[samgabriel]

@pethessa no we haven't yet. Can you and post it here pls

[IgorVNC]

Same issue

Going to try to request the permissions as explained here https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/618#issuecomment-747729234

[SzymG]

I solved problem in my case. I've done everything as @hthetiot said, including _requestPermission_ etc, but it wasn't still working. My problem was, that I called registerGlobals() in app.component.ts, when app started. When I added _registerGlobals()_ also to the service, where i called _getUserMedia_ everything worked as I expected. App asked me for additional permission, but problem was solved. Hope I helped.

[IgorVNC]

Hmm,
tried to use permissions request but did not help - getting same as described here https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/618#issuecomment-748033871

[IgorVNC]

As a side note (not a solutiuon to the issue):

I have rechecked the https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/requesting_authorization_for_media_capture_on_ios

'Verify and Request Authorization for Capture' section

There is a proper implementation of 'notDetermined' case:

 case .notDetermined: // The user has not yet been asked for camera access.
        AVCaptureDevice.requestAccess(for: .video) { granted in
            if granted {
                self.setupCaptureSession()
            }
        }

which I think should be implemented in 'cordova-plugin-iosrtc' as well
@hthetiot

[IgorVNC]

I think the issue is related to secure 'https' restrictions for getUserMedia,
as described here https://bugs.webkit.org/show_bug.cgi?id=208667#c63

and here https://github.com/geckse/ionic-capacitor-webrtc-camera-example#here-are-my-key-learnings-for-generell-working-with-the-webrtc-api-on-ios

so it's not mainly related to permissions, which is properly granted, but still the error occures

[AleksandarTokarev]

I am getting this error here on getUserMedia call.

NotReadableError: The I/O read operation failed.

We are using meteorjs + cordova.
@IgorVNC i am not sure if the https stuff could be related to our issue, but we get this prompt

After Allow, we get the error above.

[IgorVNC]

Ok, I think I found a solution

and it's basically related to my above comment with Cordova secure access to 'getUserMedia' and to @SzymG comment https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/618#issuecomment-748620749

While debuggin, I realized that the plugin's getUserMedia implementaton is not used https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/src/iosrtcPlugin.swift#L1023
Instead, seems a WKWebView implementation was used which produced the above error.

And I have a 'registerGlobals' call which brought more questions...

What I tried then is to call 'registerGlobals' right when a user press a 'call' button and it helped (!).

This leaded me to some explanation of the issue - it maybe related to 'adaper.js' mangling with 'getUserMedia' or so, e.g. a 'registerGlobals' was called first, and then it was overrided with standard WKWebView implementation of 'getUserMedia' by 'adaper.js'. So some kind of a race condition.

Mooving 'registerGlobals' call to a later stage resolved the issue.

[hthetiot]

If you get permission issue it's most likely you are not using getUserMedia from iosRTC but from WKWebview, you need to make sure you call
registerGlobals prior using getUserMedia or directly use cordovaplugins.iosrtc methods in the plugins namespace.

See how he resolved his issue by changing registerGlobals call:
https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/618#issuecomment-748620749

[IgorVNC]

@hthetiot you are right,
just explained the same in https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/618#issuecomment-748915355

[hthetiot]

This leaded me to some explanation of the issue - it maybe related to 'adaper.js' mangling with 'getUserMedia' or so, e.g. a 'registerGlobals' was called first, and then it was overrided with standard WKWebView implementation of 'getUserMedia' by 'adaper.js'. So some kind of a race condition.

"Mooving 'registerGlobals' call to a later stage resolved the issue."

Yes could be related to adapter.js check sample we inject adapter and library after registration of SHIM via registerGlobals.

See sample boostrap here: https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/blob/master/www/js/common.js#L630

[hthetiot]

@Gregadeaux it has been confirmed that you need to make sure to call registerGlobals to expose SHIM at the right time.

Note on people that think this plug-in is now useless: you need to be on an https page, not the case with localhost unless debug. Meaning this plug-in is still useful.

[numerized]

I've successfully managed it by following @IgorVNC advice:
What I tried then is to call 'registerGlobals' right when a user press a 'call' button and it helped (!).

To be more precise, I've added registerGlobals() right before getUserMedia is required.

[numerized]

Oh and make sure it is not called nowhere before, for example, I use RTCMultiConnection.js, the thing it that they added their own registerGlobals inside their lib, and therefore I had to remove the whole thing where they want to declare cordova.iosrtc for us.

[hthetiot]

Good to know @numerized

[hthetiot]

Related https://github.com/muaz-khan/RTCMultiConnection/issues/712#issuecomment-451124390

[hthetiot]

Related https://github.com/apizee/ApiRTC-mobile/issues/15#issuecomment-750828422

[hthetiot]

Finally found the issue, it look like webrtc-adater is overiding navigator.mediaDevices object even after registerGlobals, causing to use native getUserMedia instead of SHIM. Solution is to freeze navigator.mediaDevices to prevent webrtc-adater to overide.

    // Prevent WebRTC-adapter to overide navigator.mediaDevices after shim is applied
    Object.freeze(navigator.mediaDevices)

[hthetiot]

Fixed on master, please try master to confirm:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/8d45a81123beb1ecc1d5e4c6e4e4d8d58c10ba11

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#master --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

[hthetiot]

Please @samgabriel @numerized @IgorVNC you may test master to confirm the fix our add line from 8d45a81 to see if that fix the issue.

Once confirmed i will release 6.0.17 https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/CHANGELOG.md#version-6017

[hthetiot]

The fix is not perfect, will continue to test and improve.

[neeact]

Same problem in our app on iOS 14.3.
Solved it by calling registerGlobals() right before requesting the getUserMedia.

@hthetiot should we also update the iosRTC stack to the latest release as you mention previously?
(just asking because it's working now with the above fix...)

[neeact]

Same problem in our app on iOS 14.3.
Solved it by calling registerGlobals() right before requesting the getUserMedia.

@hthetiot should we also update the iosRTC stack to the latest release as you mention previously?
(just asking because it's working now with the above fix...)

Just tested build 6.1.17 without calling registerGlobals() before getUserMedia and the problem persists...
When calling registerGlobals() before getUserMedia problem is solved.

[hthetiot]

Same problem in our app on iOS 14.3.
Solved it by calling registerGlobals() right before requesting the getUserMedia.
@hthetiot should we also update the iosRTC stack to the latest release as you mention previously?
(just asking because it's working now with the above fix...)

Just tested build 6.1.17 without calling registerGlobals() before getUserMedia and the problem persists...
When calling registerGlobals() before getUserMedia problem is solved.

Thank you @neeact

I will get this bitch, i will keep you posted here.

[hthetiot]

@hthetiot should we also update the iosRTC stack to the latest release as you mention previously?

I want to move on, but i also want keep thing stable before updating webRTC.framework

6.1 will have some changes and I want to go to 7 and 8 right after. Check milestones for details.

[neeact]

I will get this bitch, i will keep you posted here.

Thank you @hthetiot !

[numerized]

Ok I'm now starting the https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/8d45a81123beb1ecc1d5e4c6e4e4d8d58c10ba11 test

[hthetiot]

alright, after much debugging, i conclude that webrtc-adapter override navigator.mediaDevices even if the webrtc-adapter is injected before or after calling registerGlobals, the initial workaround is to call every time cordova.plugins.iosrtc.registerGlobals every time before you call navigator.mediadevices.getUserMedia.

Here is the fix i tested for iosRTC

// Prevent WebRTC-adapter to overide navigator.mediaDevices after shim is applied since ios 14.3
Object.defineProperty(
  navigator,
  'mediaDevices',
  {
    value: new cordova.plugins.iosrtc.MediaDevices(),
    writable: false
  },
  {
    enumerable: false,
    configurable: false,
    writable: false,
    value: 'static'
  }
);

see: https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/a2d3d0e63ad0fa92e6b57d8e4b539fc96bc229e6

To test that fix, you can try master:

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#master --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

or simply call following methods right after cordova.plugins.iosrtc.registerGlobals:

// Prevent WebRTC-adapter to overide navigator.mediaDevices after shim is applied since ios 14.3
Object.defineProperty(
  navigator,
  'mediaDevices',
  {
    value: new cordova.plugins.iosrtc.MediaDevices(),
    writable: false
  },
  {
    enumerable: false,
    configurable: false,
    writable: false,
    value: 'static'
  }
);

[hthetiot]

Once the fix is confirmed fixed on master by at least 2 users, i will release 6.0.17 the last 6.0.x, will follow 6.1 with Typescript, 7.0.0 with WebRTC 79 then 8.0.0 with WebRTC 84+, the goal is to keep 6.0.x as LTS then once 8.0 is stable push for migration. Look at milestones for details of the planned releases.

https://github.com/cordova-rtc/cordova-plugin-iosrtc/milestones

[hthetiot]

Please @Gregadeaux can you test master, that you issue and lot of people helped, may be you can help confirm that fix :)

Cc @samgabriel please test, if confirmed your PRs will land on master aka 6.1 after 6.0.17 release with that fix.

[neeact]

Tested the below fix right after registerGlobals() and it works like a charm:

// Prevent WebRTC-adapter to overide navigator.mediaDevices after shim is applied since ios 14.3
Object.defineProperty(
  navigator,
  'mediaDevices',
  {
    value: new cordova.plugins.iosrtc.MediaDevices(),
    writable: false
  },
  {
    enumerable: false,
    configurable: false,
    writable: false,
    value: 'static'
  }
);

All the best,
-ct-

[hthetiot]

Thank you @neeact

I also got a report by email that the fix is working, that enough for me I will release.

[hthetiot]

Released on npm and git tag as 6.0.17 see CHANGELOGS.md for details:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/CHANGELOG.md#version-6017

Next release is 6.1.0 see details:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/milestone/21

[numerized]

Hello there,

I'm on the latest version 6.0.17 and when I have a loss of wifi or network switch or simply I disconnect / reconnect the same call without relaunching the app I get this

Attempting to change value of a readonly property.

I works well in the happy path but not in this special scenario

What can we do ?

[hthetiot]

What can we do ?

@numerized , before anything you should understand your usage of iosRTC, and here is simply appears you call registerGlobals multiple time... I did not need to debug to understand that, it make sense, please try do a minimum of research before coming here and report a problem.