Cordova-plugin-iosrtc: getUserMedia compatiblity with Twilio Video

[sboudouk]

Expected behavior

Since twilio-video uses getUserMedia to retrieve user's Video & Audio tracks, twilio-video should work as expected.

Observed behavior

TypeError: track must be a LocalAudioTrack, LocalVideoTrack, LocalDataTrack, or MediaStreamTrack

Steps to reproduce the problem

  let globalRegistered = 0;

  const setGlobal = (): void => {
  if (device.platform === 'iOS'){
      if (globalRegistered === 0) {
        console.log('successfully registered global');
        // eslint-disable-next-line
        (cordova as any).plugins.iosrtc.registerGlobals();
        globalRegistered = 1;
      }
    }
  };


  document.addEventListener('deviceready', (): void => {
    setGlobal();
  }, false);

  const handleConnectToRoom = async (): Promise<void> => {
    navigator.getUserMedia({ audio: true, video: true }, (stream): void => {
      console.log('I got a Stream:');
      console.log(stream);
      connect(token.token, {
        name: 'roomname',
        tracks: stream.getTracks(),
      }).then(roomJoined).catch((error): void => {
        console.log('Error while connecting to TwilioRoom: ');
        console.log(error);
      });
    }, (error): void => {
      console.log('Error in getUserMedia ');
      console.log(error);
    });
  };

Platform information

• Cordova version: 9.0.0 ([email protected])
• Plugin version: cordova-plugin-iosrtc 6.0.10 "iosrtc"
• iOS version: 13.4

• Xcode version: 11.4.1 (11E503a)

• twilio-video version: 2.3.0 (same issues with 1.XX tested with 1.19)

It would be great if that the MediaStreamTrack cake would match with twilio-video since It's a very popular package.

Thanks & keep up the good work.

[hthetiot]

Thank you @sboudouk for this detailed issue. I will see if I can solve the problem.

[savgiannis]

Yeah twilio video is much needed I would appreciate if it is fixed!

[hthetiot]

I dont want to spent to much Twilio credits, if you want me to debug more I will need that someone give me access to a twilio accoundSID and videoToken and videoSecret.

[sboudouk]

I got you @hthetiot I'll give you mine by mail.

[hthetiot]

Thank you @sboudouk

[hthetiot]

I need to release 6.0.12 before but I have found some way to make it works.
(6.0.12 https://github.com/cordova-rtc/cordova-plugin-iosrtc/milestone/25)

This may be on 6.0.1 because I had to SHAM MediaStreamTrack better and also override existing MediaStreamTrack and MediaStream prototype (I know it's dirty) to allow loading Twlio before iosrtc and still use iosRTC shim, this may cause issue with webrtc-adapter i need more testing.

[Maxouhell]

Hello !
We are currently facing the same problem.

Do you have any news on that ?

Thanks !

[hthetiot]

No update yet, the fix will be a combination of https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-619563313 and #508

Also IF there were update you would see update here. If you want you can also contribute that free software, therefore it's done only on my spare time, I'm not your employee or at your service.

[Maxouhell]

Seriously ? Oo
Calm down, I'm just asking if you found a solution while i'm searching for it...

I would be very happy to contribute if i find the solution... ><'

[hthetiot]

I'm just asking if you found a solution while i'm searching for it...

-> https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-619563313

[cj]

@hthetiot

First off thank you for this amazing plugin!

I just tried the fix you suggest but I got getUserMedia is not supported:

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-05-18 19:23:32.188210-0500 ***[2292:686887] [Snapshotting] Snapshotting a view (0x10457b560, _UIReplicantView) that has not been rendered at least once requires afterScreenUpdates:YES.
2020-05-18 19:23:33.074137-0500 *[2292:686887] WARN: 2020-05-19 00:23:33.071Z | WARN in [createLocalTracks #1]: Call to getUserMedia failed: Error: getUserMedia is not supported
2020-05-18 19:23:33.079292-0500 *[2292:686887] WARN: 2020-05-19 00:23:33.074Z | WARN in [createLocalTracks #2]: Call to getUserMedia failed: Error: getUserMedia is not supported
2020-05-18 19:23:33.079442-0500 *[2292:686887] ERROR: Unable to connect to Room: getUserMedia is not supported

I can give you access to my Twilio account accoundSID and videoToken and videoSecret just shoot me an email cjlazell at gmail.com. If you add a Github sponsor button, I'd love to sponsor this project too! https://github.com/sponsors

Cheers!

[cj]

@hthetiot wups the getUserMedia is not supported was just because I forgot to initialize it (sorry long day). It's now loading, but there seems to be another issue, sound is fine, but the video is just black.

[cj]

ok - figured out the black video, this was the cause https://github.com/twilio/video-quickstart-js/issues/116#issuecomment-614872845. Now the local participant shows the video (the cordova app running on ios)!

There is another issue. If i now open a browser on a computer, I can see and hear the iOS video, but on the device itself I can't see any audio or video from the computer. If I try it on iOS using the safari browser instead of the cordova app, everything works just fine. I see the iOS local video and the connected computers audio and video.

[cj]

iOS cordova is attaching the audio (I can hear it) of the computer, but just not the video.

[cj]

So I think I may have found the reason why the computer video does not show up in iOS cordova when someone joins. https://gist.github.com/cj/3226ecd8a4acdf616d66360565967dd3#file-gistfile1-txt-L33

ERROR: pluginMediaStream with id=default_63D585DA-B0D2-44FF-A486-058AB001694C already exist

Seems like https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/src/PluginMediaStream.swift#L24 might not be working correctly.

Doing something hacky like self.id = rtcMediaStream.streamId + "_" + UUID().uuidString + String(rtcMediaStream.streamId.count); gets ride of the error, but the video still never gets displayed.

[hthetiot]

Thank you @cj im working on it in #508 keep looking you might find something else. I will also look at your workaround for MediaStreamId.

[hthetiot]

508 is now ready and should fix the issue with Twilio.

@cj @sboudouk give it a try using the sample here https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-619563313 and the branch bugs/ontrack (see install instruction on #508 description).

[cj]

@hthetiot no problem - I Just gave https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/508 a try and it worked perfectly, even with the Twilio NPM package! Thank you!

You should 100% add a sponsor badge, and I'm sure I'm not the only one that would use it to support your fantastic plugin and hard work!

[hthetiot]

Thank you @cj, I have the sponsor I need to finalize the GitHub request.

@sboudouk your fix is ready see last comment.

[hthetiot]

@cj I made some change to try to handle #505 issue can you test #508 again with you have a moment:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/508/commits/dff0bdb888ce26e57f5ecaa97561684d1dd9b31e

Long story short, I was creating a new MediaStream everytime with possibly one track instead of 2 and it cause issue for Library that still use event.streams[0] on track event instead of creating there own MediaStream like that should. See https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/505#issuecomment-631402350

Thank you for your help.

[cj]

@hthetiot that latest version did not work, the computer video never got added to iOS.

2020-05-20 10:58:52.372471-0500 ***[3248:973618] iosrtc:MediaStreamRenderer refresh() +0ms
2020-05-20 10:58:52.372771-0500 ***[3248:973618] iosrtc:MediaStreamRenderer refresh() | no video track yet +0ms
2020-05-20 10:58:52.373227-0500 ***[3248:973618] iosrtc:MediaStreamRenderer refresh() | [data:{"elementLeft":0,"elementTop":268,"elementWidth":400,"elementHeight":200,"videoViewWidth":400,"videoViewHeight":200,"visible":true,"opacity":1,"zIndex":0,"mirrored":false,"objectFit":"contain","clip":true,"borderRadius":0}] +0ms
2020-05-20 10:58:52.373496-0500 ***[3248:973618] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-05-20 10:58:52.373634-0500 ***[3248:973618] PluginMediaStreamRenderer#refresh() [elementLeft:0.0, elementTop:268.0, elementWidth:400.0, elementHeight:200.0, videoViewWidth:400.0, videoViewHeight:200.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2020-05-20 10:58:52.373946-0500 ***[3248:973618] iosrtc:MediaStreamRenderer render() [stream:{"_listeners":{},"_id":"2013180324-2645161682-1254368535-3708531460","_active":true,"connected":true,"_audioTracks":{},"_videoTracks":{"690220b0-c3b5-455a-9a71-2b6ac0224faf":{"_listeners":{"ended":[null,null,null,null]},"id":"690220b0-c3b5-455a-9a71-2b6ac0224faf","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_blobId":"MediaStream_2013180324-2645161682-1254368535-3708531460"}] +0ms
2020-05-20 10:58:52.374165-0500 ***[3248:973618] iosrtcPlugin#MediaStreamRenderer_render()
2020-05-20 10:58:52.374331-0500 ***[3248:973618] PluginMediaStreamRenderer#render()
2020-05-20 10:58:52.374835-0500 ***[3248:973618] iosrtc:RTCPeerConnection onEvent() | [type:addstream, data:{"stream":{"id":"default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1","audioTracks":{"75471f2e-27b6-48e5-a392-529d3d8c1603":{"readyState":"ended","id":"75471f2e-27b6-48e5-a392-529d3d8c1603","kind":"audio","enabled":true,"trackId":"75471f2e-27b6-48e5-a392-529d3d8c1603"}},"videoTracks":{}},"type":"addstream","streamId":"default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1"}] +38ms
2020-05-20 10:58:52.374947-0500 ***[3248:973618] iosrtc:MediaStream create() | [dataFromEvent:{"id":"default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1","audioTracks":{"75471f2e-27b6-48e5-a392-529d3d8c1603":{"readyState":"ended","id":"75471f2e-27b6-48e5-a392-529d3d8c1603","kind":"audio","enabled":true,"trackId":"75471f2e-27b6-48e5-a392-529d3d8c1603"}},"videoTracks":{}}] +31ms
2020-05-20 10:58:52.375105-0500 ***[3248:973618] iosrtc:MediaStream new MediaStream(arg) | [arg:[]] +0ms
2020-05-20 10:58:52.375423-0500 ***[3248:973618] iosrtcPlugin#MediaStream_init()
2020-05-20 10:58:52.375478-0500 ***[3248:973618] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1 already exist
2020-05-20 10:58:52.375661-0500 ***[3248:973618] iosrtcPlugin#MediaStream_setListener()
2020-05-20 10:58:52.375780-0500 ***[3248:973618] iosrtc:MediaStreamTrack new() | [dataFromEvent:{"readyState":"ended","id":"75471f2e-27b6-48e5-a392-529d3d8c1603","kind":"audio","enabled":true,"trackId":"75471f2e-27b6-48e5-a392-529d3d8c1603"}] +38ms
2020-05-20 10:58:52.375809-0500 ***[3248:973786] PluginMediaStream#setListener()
2020-05-20 10:58:52.375878-0500 ***[3248:973618] iosrtcPlugin#MediaStreamTrack_setListener()
2020-05-20 10:58:52.375968-0500 ***[3248:973785] PluginMediaStreamTrack#setListener() [kind:audio, id:75471f2e-27b6-48e5-a392-529d3d8c1603]
2020-05-20 10:58:52.375981-0500 ***[3248:973618] Error in Success callbackId: iosrtcPlugin947657497 : TypeError: undefined is not an object (evaluating 'stream.emitConnected')
2020-05-20 10:58:52.376675-0500 ***[3248:973618] onEvent@ionic://app/plugins/cordova-plugin-iosrtc/dist/cordova-plugin-iosrtc.js:2428:11

[cj]

@hthetiot I just re-installed this commit aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600 and it does indeed work. So something in this new one is breaking it.

[hthetiot]

2020-05-20 10:58:52.375981-0500 *[3248:973618] Error in Success callbackId: iosrtcPlugin947657497 : TypeError: undefined is not an object (evaluating 'stream.emitConnected')

[hthetiot]

Sorry about that the lint failed to catch the issue.
Can you try 44e5b8b, thank you again for your help.
I did not have this issue in the sample I think because I used ontrack only.

[hthetiot]

@cj How did you use the package because you need to load twilio after iosrtc we just d'like to know me and @sboudouk

[hthetiot]

@cj if you have a working Twilio-video.js we would be curious to see how.
See https://github.com/twilio/twilio-video.js/issues/963#issuecomment-634203022 details.

[hthetiot]

Updated Twilio possible issue with this commit:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/403284a0909cbc62edd9e47689fd2dadccc1a636

[cj]

@hthetiot @sboudouk This was just me playing around, but it works https://gist.github.com/cj/2beabb496fde34da009c946775be6021 you basically just require the twilio package after you have registered iosrtc.

p.s. I was using "twilio-video": "^2.4.0"

[sboudouk]

@cj Using this code, did you manage to get remoteParticipant track to display on iOS device ?

Thanks for sharing.

[cj]

@sboudouk yes - it was working perfectly in iOS!

[hthetiot]

Thank you @cj

Closing the issue for now then.

[matijagrcic]

Are above changes available in 6.0.12 https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/425#issuecomment-635225234 ? Only asking as I'm getting same error as
https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-630579714

ERROR: pluginMediaStream with id=default_63D585DA-B0D2-44FF-A486-058AB001694C already exist

Thanks for this whole thread, helped greatly so far.

[sboudouk]

@matijagrcic try to use https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600 commit, it works for me, when 6.0.12 don't, you should give it a try.

[matijagrcic]

Thanks, how would i install this this commit?

[sboudouk]

```
cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600 --save
cordova platform remove ios --no-save
cordova platform add ios --no-save

[matijagrcic]

Thanks, wasn't sure do i just reference some branch like mentioned here https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/508 or the commit like you did.

[sboudouk]

@matijagrcic try to use aee2b39 _commit_

[matijagrcic]

Yep, all good, already trying that now.

[matijagrcic]

I'm still seeing the following

2020-06-03 23:10:38.614782+0100 App[43033:442187] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist

full log below:

2020-06-03 23:10:38.606955+0100 App[43033:442187] iosrtc registerGlobals() +0ms
2020-06-03 23:10:38.607696+0100 App[43033:442187] iosrtc restoreCallbacksSupport() +0ms
2020-06-03 23:10:38.608673+0100 App[43033:442187] iosrtc:getUserMedia [original constraints:{"audio":false,"video":{"facingMode":"user"}}] +1m
2020-06-03 23:10:38.609096+0100 App[43033:442187] iosrtc:getUserMedia [computed constraints:{"video":{"facingMode":{"exact":"user"}}}] +0ms
2020-06-03 23:10:38.611019+0100 App[43033:442187] iosrtcPlugin#getUserMedia()
2020-06-03 23:10:38.611164+0100 App[43033:442187] PluginGetUserMedia#call()
2020-06-03 23:10:38.611266+0100 App[43033:442187] PluginGetUserMedia#call() | video authorization: authorized
2020-06-03 23:10:38.611432+0100 App[43033:442187] PluginGetUserMedia#call() | video requested
2020-06-03 23:10:38.611609+0100 App[43033:442187] PluginGetUserMedia#call() | chosen video constraints: {
    facingMode =     {
        exact = user;
    };
}
2020-06-03 23:10:38.612396+0100 App[43033:442187] PluginMediaStream#init()
2020-06-03 23:10:38.612649+0100 App[43033:442187] PluginMediaStreamTrack#init()
2020-06-03 23:10:38.612805+0100 App[43033:442187] PluginMediaStreamTrack#run() [kind:video, id:48E7AD91-13D7-4E37-82D5-3136A8D0A06F]
2020-06-03 23:10:38.612883+0100 App[43033:442187] PluginMediaStream#run()
2020-06-03 23:10:38.613912+0100 App[43033:442187] iosrtc:getUserMedia getUserMedia() | success +7ms
2020-06-03 23:10:38.614085+0100 App[43033:442187] iosrtc:MediaStream create() | [dataFromEvent:{"id":"13975956-A8BA-4D7D-ACB6-2FF25624A479","audioTracks":{},"videoTracks":{"48E7AD91-13D7-4E37-82D5-3136A8D0A06F":{"readyState":"live","id":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F","kind":"video","enabled":true,"trackId":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F"}}}] +1m
2020-06-03 23:10:38.614224+0100 App[43033:442187] iosrtc:MediaStream new MediaStream(arg) | [arg:[]] +0ms
2020-06-03 23:10:38.614673+0100 App[43033:442187] iosrtcPlugin#MediaStream_init()
2020-06-03 23:10:38.614782+0100 App[43033:442187] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist
2020-06-03 23:10:38.614914+0100 App[43033:442187] iosrtcPlugin#MediaStream_setListener()
2020-06-03 23:10:38.615018+0100 App[43033:443387] PluginMediaStream#setListener()
2020-06-03 23:10:38.615068+0100 App[43033:442187] iosrtc:MediaStreamTrack new() | [dataFromEvent:{"readyState":"live","id":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F","kind":"video","enabled":true,"trackId":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F"}] +1m
2020-06-03 23:10:38.615697+0100 App[43033:442187] iosrtcPlugin#MediaStreamTrack_setListener()
2020-06-03 23:10:38.615828+0100 App[43033:443849] PluginMediaStreamTrack#setListener() [kind:video, id:48E7AD91-13D7-4E37-82D5-3136A8D0A06F]
2020-06-03 23:10:38.615876+0100 App[43033:442187] iosrtc:MediaStream emitConnected() +1ms
2020-06-03 23:10:38.615998+0100 App[43033:442187] iosrtc:MediaStream getAudioTracks() +0ms
2020-06-03 23:10:38.616121+0100 App[43033:442187] iosrtc:MediaStream getVideoTracks() +0ms

// require Twilio after we have registered iosrtc
const Video = require("twilio-video");

Video.createLocalVideoTrack({ facingMode: "user" }).then((track) => {
  this.localPreviewVideoElement.nativeElement.appendChild(track.attach());
});

Any pointers as it seems to get the video properly but then fails?

[sboudouk]

@matijagrcic can you provide full code please ? With Twilio and iOS version ?

[matijagrcic]

All info below, anything else needed please let me know.

❯ npm view twilio-video version
2.5.0

iOS version: 12+

export enum CameraDirection {
  Front,
  Rear,
}
type VideoConfiguration = {
  [key in CameraDirection]: string;
};

export const VideoConfigurations: VideoConfiguration = {
  [CameraDirection.Front]: 'user',
  [CameraDirection.Rear]: 'environment',
};

function registerWebRTCForiOS() {
  var cordova = window.cordova;
  if (cordova && cordova.plugins && cordova.plugins.iosrtc) {
    cordova.plugins.iosrtc.registerGlobals();

    // patch MediaStreamTrack with clone
    MediaStreamTrack.prototype.clone = function () {
      return this;
    };
    cordova.plugins.iosrtc.debug.enable('*', true);
  }
}

function showLocalVideoPreview(cameraDirection: CameraDirection) {
  registerWebRTCForiOS();

  // we need to require Twilio after we have registered iosrtc
  const Video = require('twilio-video');

  Video.createLocalVideoTrack({
    facingMode: VideoConfigurations[cameraDirection],
  }).then((track) => {
    this.localPreviewVideoElement.nativeElement.appendChild(track.attach());
  });
}

showLocalVideoPreview(CameraDirection.Front);

[sboudouk]

You should make sure that it's iosrtc MediaStreamTrack that is used before calling Video, maybe in the React's useEffect function:

 useEffect(() => {
    const cordova = window.cordova

    if (cordova && cordova.plugins && cordova.plugins.iosrtc) {

      // Expose WebRTC and GetUserMedia SHIM as Globals (Optional)
      // Alternatively WebRTC API will be inside cordova.plugins.iosrtc namespace
      cordova.plugins.iosrtc.registerGlobals()

      // Patch MediaStreamTrack with clone
      MediaStreamTrack.prototype.clone = function () {
        return this
      }

      // Enable iosrtc debug (Optional)
      cordova.plugins.iosrtc.debug.enable('*', false)
    }
    MediaStreamTrack.prototype.clone = function () {
      return this
    }
    connect(name, room)
  // eslint-disable-next-line
  }, [])

Also, while running on device, put a breakpoint before calling "Video.createLocalVideoTrack" and in Safari's console, you can try to log MediaStreamTrack.

Even if I think that's it's not the issue, because you would get a different error, so it might be an issue with iOS 12. Can you test this on iOS 13+ ?

[matijagrcic]

Thank you @sboudouk. Yeah it's strange, as per the output log all seems fine (stream is acquired) but then it errors out and so the output isn't displayed. Will try running it under iOS 13+ and will follow up with the result but this might not be feasible from the business perspective.

[matijagrcic]

I've managed to get it working on iOS 12+ (no getting ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist anymore) by changing the code slightly but now it's complaining about the 2020-06-05 13:09:44.361364+0100 App[12577:99329] [LayoutConstraints] Unable to simultaneously satisfy constraints..

I've added full log below, based on searching thru this repo I've tried setting width/height but I've also tried without them.

Reference: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/375#issuecomment-546917118

Would you have any clue of what can be wrong here?

020-06-05 13:09:44.344110+0100 App[12577:99329] iosrtc:getUserMedia [original constraints:{"audio":false,"video":{"facingMode":"user","width":640,"height":480}}] +20s
2020-06-05 13:09:44.344261+0100 App[12577:99329] iosrtc:getUserMedia [computed constraints:{"video":{"width":{"exact":640},"height":{"exact":480},"facingMode":{"exact":"user"}}}] +0ms
2020-06-05 13:09:44.344456+0100 App[12577:99329] iosrtcPlugin#getUserMedia()
2020-06-05 13:09:44.344547+0100 App[12577:99329] PluginGetUserMedia#call()
2020-06-05 13:09:44.344655+0100 App[12577:99329] PluginGetUserMedia#call() | video authorization: authorized
2020-06-05 13:09:44.344872+0100 App[12577:99329] PluginGetUserMedia#call() | video requested
2020-06-05 13:09:44.345765+0100 App[12577:99329] PluginGetUserMedia#call() | chosen video constraints: {
    facingMode =     {
        exact = user;
    };
    height =     {
        exact = 480;
    };
    width =     {
        exact = 640;
    };
}
2020-06-05 13:09:44.345961+0100 App[12577:99329] PluginMediaStream#init()
2020-06-05 13:09:44.346278+0100 App[12577:99329] PluginMediaStreamTrack#init()
2020-06-05 13:09:44.346433+0100 App[12577:99329] PluginMediaStreamTrack#run() [kind:video, id:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.346516+0100 App[12577:99329] PluginMediaStream#run()
2020-06-05 13:09:44.347468+0100 App[12577:99329] iosrtc:getUserMedia getUserMedia() | success +5ms
2020-06-05 13:09:44.347642+0100 App[12577:99329] iosrtc:MediaStream create() | [dataFromEvent:{"id":"C390DCA1-2882-45D2-A07F-C8F397F769A2","audioTracks":{},"videoTracks":{"DD36969E-8111-4070-811F-FCF5CFC7A1B5":{"readyState":"live","id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","enabled":true,"trackId":"DD36969E-8111-4070-811F-FCF5CFC7A1B5"}}}] +19s
2020-06-05 13:09:44.347922+0100 App[12577:99329] iosrtc:MediaStream new MediaStream(arg) | [arg:[]] +0ms
2020-06-05 13:09:44.348081+0100 App[12577:99329] iosrtcPlugin#MediaStream_init()
2020-06-05 13:09:44.348179+0100 App[12577:99329] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=C390DCA1-2882-45D2-A07F-C8F397F769A2 already exist
2020-06-05 13:09:44.348311+0100 App[12577:99329] iosrtcPlugin#MediaStream_setListener()
2020-06-05 13:09:44.348415+0100 App[12577:100344] PluginMediaStream#setListener()
2020-06-05 13:09:44.348994+0100 App[12577:99329] iosrtc:MediaStreamTrack new() | [dataFromEvent:{"readyState":"live","id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","enabled":true,"trackId":"DD36969E-8111-4070-811F-FCF5CFC7A1B5"}] +19s
2020-06-05 13:09:44.349178+0100 App[12577:99329] iosrtcPlugin#MediaStreamTrack_setListener()
2020-06-05 13:09:44.349303+0100 App[12577:100344] PluginMediaStreamTrack#setListener() [kind:video, id:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.349340+0100 App[12577:99329] iosrtc:MediaStream emitConnected() +1ms
2020-06-05 13:09:44.350208+0100 App[12577:99329] iosrtc:MediaStream getAudioTracks() +0ms
2020-06-05 13:09:44.350381+0100 App[12577:99329] iosrtc:MediaStream getVideoTracks() +2ms
2020-06-05 13:09:44.352594+0100 App[12577:99329] iosrtc:MediaStream new MediaStream(arg) | [arg:] +2ms
2020-06-05 13:09:44.352838+0100 App[12577:99329] iosrtcPlugin#MediaStream_init()
2020-06-05 13:09:44.353378+0100 App[12577:99329] PluginMediaStream#init()
2020-06-05 13:09:44.353612+0100 App[12577:99329] PluginMediaStream#run()
2020-06-05 13:09:44.353787+0100 App[12577:99329] iosrtcPlugin#MediaStream_setListener()
2020-06-05 13:09:44.353906+0100 App[12577:100344] PluginMediaStream#setListener()
2020-06-05 13:09:44.353945+0100 App[12577:99329] iosrtc:MediaStream getVideoTracks() +0ms
2020-06-05 13:09:44.354068+0100 App[12577:99329] iosrtc:MediaStream addTrack() [track:{"_listeners":{"ended":[null,null]},"id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}] +0ms
2020-06-05 13:09:44.354189+0100 App[12577:99329] iosrtcPlugin#MediaStream_addTrack()
2020-06-05 13:09:44.354294+0100 App[12577:100344] PluginMediaStream#addTrack()
2020-06-05 13:09:44.354328+0100 App[12577:99329] iosrtc:MediaStream emitConnected() +0ms
2020-06-05 13:09:44.354602+0100 App[12577:99329] AppHealthDEBUG: createLocalVideoTrack
2020-06-05 13:09:44.354862+0100 App[12577:100344] PluginMediaStream#addTrack() | video track added
2020-06-05 13:09:44.355081+0100 App[12577:99329] iosrtc:MediaStream new MediaStream(arg) | [arg:] +1ms
2020-06-05 13:09:44.355187+0100 App[12577:100344] PluginMediaStream | OnAddTrack [label:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.355516+0100 App[12577:100344] PluginMediaStreamTrack#run() [kind:video, id:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.355691+0100 App[12577:99329] iosrtcPlugin#MediaStream_init()
2020-06-05 13:09:44.356580+0100 App[12577:99329] PluginMediaStream#init()
2020-06-05 13:09:44.357129+0100 App[12577:99329] PluginMediaStream#run()
2020-06-05 13:09:44.357344+0100 App[12577:99329] iosrtcPlugin#MediaStream_setListener()
2020-06-05 13:09:44.357507+0100 App[12577:99329] iosrtc:MediaStream getVideoTracks() +1ms
2020-06-05 13:09:44.357645+0100 App[12577:99329] iosrtc:MediaStream addTrack() [track:{"_listeners":{"ended":[null,null,null]},"id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}] +0ms
2020-06-05 13:09:44.357922+0100 App[12577:99329] iosrtcPlugin#MediaStream_addTrack()
2020-06-05 13:09:44.358157+0100 App[12577:99329] iosrtc:MediaStream emitConnected() +0ms
2020-06-05 13:09:44.358937+0100 App[12577:99329] iosrtc:videoElementsHandler new video element added +13s
2020-06-05 13:09:44.359205+0100 App[12577:99329] iosrtc:videoElementsHandler observeVideo() +0ms
2020-06-05 13:09:44.359466+0100 App[12577:99329] iosrtc:MediaStreamRenderer new() | [element:"[object HTMLVideoElement]"] +13s
2020-06-05 13:09:44.359672+0100 App[12577:99329] iosrtcPlugin#new_MediaStreamRenderer()
2020-06-05 13:09:44.359842+0100 App[12577:99329] PluginMediaStreamRenderer#init()
2020-06-05 13:09:44.360411+0100 App[12577:99329] PluginMediaStreamRenderer#run()
2020-06-05 13:09:44.360634+0100 App[12577:99329] iosrtc:MediaStreamRenderer refresh() +0ms
2020-06-05 13:09:44.360790+0100 App[12577:99329] iosrtc:MediaStreamRenderer refresh() | no video track yet +1ms
2020-06-05 13:09:44.361364+0100 App[12577:99329] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x6000013455e0 h=--& v=--& UIView:0x7fbeb2c77cc0.height == 0   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x6000013393b0 h=-&- v=-&- WKWebView:0x7fbeb4818a00' App\U00ae'.minY == 20   (active, names: '|':UIView:0x7fbeb2e11c30 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x600001339400 h=-&- v=-&- V:[WKWebView:0x7fbeb4818a00' App\U00ae']-(0)-|   (active, names: '|':UIView:0x7fbeb2e11c30 )>",
    "<NSLayoutConstraint:0x600001336a30 UIView:0x7fbeb2c77cc0.top == UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide'.top   (active)>",
    "<NSLayoutConstraint:0x6000013369e0 UIView:0x7fbeb2c77cc0.bottom == UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide'.bottom   (active)>",
    "<NSLayoutConstraint:0x6000013394a0 'UIView-Encapsulated-Layout-Height' UIView:0x7fbeb2e11c30.height == 667   (active)>",
    "<NSLayoutConstraint:0x600001345270 'UIViewSafeAreaLayoutGuide-bottom' V:[UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide']-(0)-|   (active, names: '|':WKWebView:0x7fbeb4818a00' App\U00ae' )>",
    "<NSLayoutConstraint:0x6000013451d0 'UIViewSafeAreaLayoutGuide-top' V:|-(0)-[UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide']   (active, names: '|':WKWebView:0x7fbeb4818a00' App\U00ae' )>"
)

[matijagrcic]

Actually ignore the above, it works fine, using createLocalVideoTrack_ on devices running iOS12 and iOS13.

Summary:

Ionic Angular

config.xml

<plugin name="cordova-plugin-inappbrowser" spec="^3.2.0" />
<plugin name="cordova-plugin-ionic" spec="^5.4.6">
<plugin name="cordova-plugin-ionic-webview" spec="^4.1.3" />
<plugin name="cordova-plugin-wkwebviewxhrfix" spec="https://github.com/TheMattRay/cordova-plugin-wkwebviewxhrfix" />

package.json

"cordova-plugin-ionic-webview": "^5.0.0",
"cordova-plugin-iosrtc": "git+https://github.com/cordova-rtc/cordova-plugin-iosrtc.git#aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600",
"twilio-video": "^2.5.0",

I'll follow up when the Twilio P2P streaming starts working on iOS, works flawlessly on Android.
Thank you for all the help and let me know if I can provide any more info that would be useful to you and the project.

[matijagrcic]

Both caller and participant video/audio is working on iOS in Twilio P2P room, but the video size is small. I've tried adding video constraints in terms of sizes but nothing changes the size of the video. Do you happen to know anything else I can try?

I didn't specify any height/width in the scss for the div or the video element. Is this mandatory on iOS?

On Android the video size is full height.

References: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/332

[hthetiot]

I've managed to get it working on iOS 12+ (no getting ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist anymore) by changing the code slightly but now it's complaining about the 2020-06-05 13:09:44.361364+0100 App[12577:99329] [LayoutConstraints] Unable to simultaneously satisfy constraints..

Layout warning can be ignored see https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/422

[hthetiot]

I didn't specify any height/width in the scss for the div or the video element. Is this mandatory on iOS?

On Android the video size is full height.

Try using with/height css to set video size, 100%, flexbox or other should works.

[hthetiot]

Alternatively you may want to call cordova.plugin.iosrtc.refreshVideos() to trigger size computation.

https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/docs/iosrtc.md#iosrtcrefreshvideos

[hthetiot]

Nothing to do with #332 or https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/375#issuecomment-546917118

It's the video tag size refresh most likely the issue not the video stream.

[hthetiot]

MediaStreamTrack.prototype.clone = function () {
      return this
    }

This should not be needed cause already on latest and master.
Cc @sboudouk

[matijagrcic]

Tried both https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-645623495 and https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-645624244 and video element is still small, log and scss below.

Works great in Android so not sure what's missing.

2020-06-18 16:44:43.768322+0100 App[63129:774260] iosrtc:MediaStreamRenderer refresh() | [data:{"elementLeft":0,"elementTop":119,"elementWidth":414,"elementHeight":207,"videoViewWidth":414,"videoViewHeight":207,"visible":true,"opacity":1,"zIndex":0,"mirrored":false,"objectFit":"fill","clip":true,"borderRadius":0}] +0ms
2020-06-18 16:44:43.768683+0100 App[63129:774260] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-06-18 16:44:43.768953+0100 App[63129:774260] PluginMediaStreamRenderer#refresh() [elementLeft:0.0, elementTop:119.0, elementWidth:414.0, elementHeight:207.0, videoViewWidth:414.0, videoViewHeight:207.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2020-06-18 16:44:43.769354+0100 App[63129:774260] iosrtc:MediaStreamRenderer render() [stream:{"_listeners":{},"_id":"3027821557-3741128866-2149914973-1061480553","_active":true,"connected":true,"_audioTracks":{},"_videoTracks":{"51B6C4DF-F7E4-41E8-8A75-38EF886814BC":{"_listeners":{"ended":[null,null,null,null]},"id":"51B6C4DF-F7E4-41E8-8A75-38EF886814BC","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_blobId":"MediaStream_3027821557-3741128866-2149914973-1061480553"}] +1ms

.main-video {
    display: flex;
    width: 100%;
    justify-content: center;
    align-items: center;

    video {
      height: auto;
      width: 100%;
      object-fit: fill;
    }
  }

Anything else I can try?

[hthetiot]

Try height 100% or with hardcoded size in pixel to debug, I think we need
to create a separate issue because this is not the same topic of the
original issues.

Also you must likely have a simple CSS or rendering issue that has nothing
bro do with Twillio.

I would recommend you look at the sample application we made:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample

On Thu, Jun 18, 2020, 18:03 Matija Grcic [e-mail] wrote:

Tried both #497 (comment)
https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-645623495
and #497 (comment)
https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-645624244
and video element is still small, log and scss below.

2020-06-18 16:44:43.768322+0100 App[63129:774260] iosrtc:MediaStreamRenderer refresh() | [data:{"elementLeft":0,"elementTop":119,"elementWidth":414,"elementHeight":207,"videoViewWidth":414,"videoViewHeight":207,"visible":true,"opacity":1,"zIndex":0,"mirrored":false,"objectFit":"fill","clip":true,"borderRadius":0}] +0ms
2020-06-18 16:44:43.768683+0100 App[63129:774260] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-06-18 16:44:43.768953+0100 App[63129:774260] PluginMediaStreamRenderer#refresh() [elementLeft:0.0, elementTop:119.0, elementWidth:414.0, elementHeight:207.0, videoViewWidth:414.0, videoViewHeight:207.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2020-06-18 16:44:43.769354+0100 App[63129:774260] iosrtc:MediaStreamRenderer render() [stream:{"_listeners":{},"_id":"3027821557-3741128866-2149914973-1061480553","_active":true,"connected":true,"_audioTracks":{},"_videoTracks":{"51B6C4DF-F7E4-41E8-8A75-38EF886814BC":{"_listeners":{"ended":[null,null,null,null]},"id":"51B6C4DF-F7E4-41E8-8A75-38EF886814BC","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_blobId":"MediaStream_3027821557-3741128866-2149914973-1061480553"}] +1ms

.main-video {
display: flex;
width: 100%;
justify-content: center;
align-items: center;

video {
  height: auto;
  width: 100%;
  object-fit: fill;
}

}

Anything else I can try?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-646125516,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAACDO26UWT3ARZ3LEYRD5TRXI3G5ANCNFSM4MPLPNZA
.

[matijagrcic]

Thanks, I already looked at the sample application. Don't think it's a CSS issue rather a combination of Ionic (Angular) cordova plugins and iosrtc given it works fine on Android and also on Safari just not in this case. The height never gets properly set whatever css combination I use while width is set properly, tried several approaches.
I'll follow up with the resolution.

[hthetiot]

The dimensions is calculated on JavaScript side using window.getComputedStyle(this.element)

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/js/MediaStreamRenderer.js#L161

Nothing special about it, what append when you force the size like 640x480 and don't use object-fit? Please provide capture or full example no reproduce you issue Inna separate issue, this issue should not be dedicated to getUserMedia and RTCPeerConnection Twilii support

[hthetiot]

@sboudouk send me an email if you want me to reopen the issue for me it look like you solved your initial problem.

@matijagrcic please create separate issue with issue instructions requirements.

Thank you for you understanding.

[hthetiot]

For master issue with twilio https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/541