Cordova-plugin-iosrtc: Problem with change camera during video call (replaceTrack is not a function)

[myzzib]

Expected behavior

Switching from front and retro camera during video call through the replaceTrack function, see code below

Observerd behavior

I get errors on "undefined is not an object (evaluating 'sender.track.kind')" and "replaceTrack is not a function".

Steps to reproduce the problem

switch camera through the code below

Platform information

• Cordova version: 8.1.2
• Plugin version: 6.0.9 and 6.0.10
• iOS version: 12.4.5 Iphone 6
• Xcode version:11.4.1 Swift 4.2

Thanks,
Regards Maurizio

WebRTCSession.prototype.switchMediaTracks = function(deviceIds, callback) {



    if(!navigator.mediaDevices.getUserMedia) {

        throw new Error('getUserMedia() is not supported in your browser');

    }



    var self = this,

        localStream = this.localStream;



    if (deviceIds && deviceIds.audio) {

        self.mediaParams.audio.deviceId = deviceIds.audio;

    }



    if (deviceIds && deviceIds.video) {

        self.mediaParams.video.deviceId = deviceIds.video;

    }



    localStream.getTracks().forEach(function(track) {

        track.stop();

    });



    navigator.mediaDevices.getUserMedia({

        audio: self.mediaParams.audio || false,

        video: self.mediaParams.video || false

    }).then(function(stream) {

        self._replaceTracks(stream);

        callback(null, stream);

    }).catch(function(error) {

        callback(error, null);

    });

};



WebRTCSession.prototype._replaceTracks = function(stream) {

    var peers = this.peerConnections,

        localStream = this.localStream,

        elemId = this.mediaParams.elemId,

        ops = this.mediaParams.options,

        newStreamTracks = stream.getTracks();



    this.detachMediaStream(elemId);



    newStreamTracks.forEach(function(track) {

        localStream.addTrack(track);

    });



    this.attachMediaStream(elemId, stream, ops);



    for (var userId in peers) {

        _replaceTracksForPeer(peers[userId]);

    }



    function _replaceTracksForPeer(peer) {

        peer.getSenders().map(function(sender) {

  ** error here          sender.replaceTrack(newStreamTracks.find(function(track) {

                return track.kind === sender.track.kind;  ** error here 

            }));

        });

    }

};

[hthetiot]

@myzzib Thank you for reporting, I will look into it in 72 hours max. I'm busy on something else.

[hthetiot]

related #423 #442

[Christina05]

Hi, any news or workarounds for this?

[abardik]

@hthetiot

I can confirm that this feature will tremendously improve user experience, because replaceTrack allows to switch between front/rear cameras, screen capture (which you already implemented via getDisplayMedia) or any other potential video sources, as well as between audio sources (for example, microphone and system sound) with no renegotiation at all - the replaced track just appears on the other side immediately, so remote user should not wait for renegotiation (which usualy takes 2-5 seconds) every time you switch your camera/screen/etc.

Speaking of compatibility, replaceTrack is already supported by all browsers, including Safari 13.1+:
https://caniuse.com/#search=replacetrack

And I can confirm - it really works like a charm, with no failure at all (for comparison, renegotiation used to fail often enough to start annoying users).

So if, in any case, it's possible to implement replaceTrack without spending enormous amount of time, it would be great to have this feature on iOS devices.

Thank you.

[abardik]

@Christina05

If you need a workaround for replaceTrack, you can use the following code (place it after cordova.plugins.iosrtc.registerGlobals). But it's not a native replaceTrack, so it will require a renegotiation. And it's based on outdated addStream/removeStream, so please, use it on your own risk, because with 'unified-plan' addStream/removeStream creates new tracks, but doesn't really remove old tracks, just mutes and marks them disabled.

Source: https://blog.mozilla.org/webrtc/warm-up-with-replacetrack/

  let orgRTCPeerConnection = window.RTCPeerConnection;
  window.RTCPeerConnection = function(config) {
    let pc = new orgRTCPeerConnection(config);
    pc._senders = [];
    pc.getSenders = () => pc._senders;
    pc._orgAddStream = pc.addStream;
    pc.addStream = stream => {
      pc._orgAddStream(stream);
      stream.getTracks().forEach(track => pc._senders.push({ track,
        replaceTrack: function (withTrack) {
          return new Promise(resolve => {
            stream.removeTrack(track);
            stream.addTrack(withTrack);
            let onn = pc.onnegotiationneeded;
            pc.onnegotiationneeded = null;
            pc.removeStream(stream);
            pc._orgAddStream(stream);
            if (onn) {
              Promise.resolve().then(onn);
              pc.addEventListener("signalingstatechange", function listener() {
                if (pc.signalingState != "stable") return;
                pc.removeEventListener("signalingstatechange", listener);
                pc.onnegotiationneeded = onn;
                resolve();
              });
            }
          });
        }
      }));
    };
    return pc;
  }

[hthetiot]

@abardik I will see what I can do after #508 and #494 is released

[hthetiot]

Related to #423

[abardik]

@hthetiot Thank you, it would be great.

[hthetiot]

RTCRtpSender landed so i will deal with that in 6.0.14

[hthetiot]

Draft Implemented here:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/544
  Cc @abardik @myzzib

[irimiaionut]

Just tried branch task/RTCRtpSender-replaceTrack to replaceTrack. Web and android already working. When using iosrtc I get the following error. Couldn't find error source.

Attempting to configure 'target' with descriptor '{....}' on object 'negotiationneeded' and got error, giving up: TypeError: Properties can only be defined on Objects.
Error: `event` must have a valid `type` property

[hthetiot]

@irimiaionut I will look into it.
See issue is missing type property here i think https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/544/files#diff-d60c1f113017b14d32febffb7b73e125R2646

[hthetiot]

@irimiaionut I fixed the error i think and also merged master in the branch, if you may give it a new shoot #544
Notice that renegotiation will be needed on negotiationneeded.

[hthetiot]

Sergio: I'm waiting for this fix

This kind of comment really make me not wanting to contribute anymore, im not pay for making this fixes, so may be you should say please and actually comment on the right issue for the fix.
This fix will be released once reporter have actually tested it.

[hthetiot]

Please try this PR see testing instructions:

587

Tested example:

var newTrack = localStream.getVideoTracks()[0].clone();
var sender = pc1.getSenders()[0];
sender.replaceTrack(newTrack);

[GlausLivio]

I have tested replaceTrack in my application, sorry but it was not working for me.
After replaceTrack, I can still show front camera not back camera in local video view

[hthetiot]

@GlausLivio thank you for testing.
ReplaceTrack should trigger negotiationneeded see implementation:
https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/js/RTCRtpSender.js

This should force your library to perform an ice restart and renegotiate to receive new stream.
https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/negotiationneeded_event

[hthetiot]

I have tested replaceTrack in my application, sorry but it was not working for me.
After replaceTrack, I can still show front camera not back camera in local video view

Thank you again for testing, I understand this did not work for you, but can you confirm 1. you did not get an error, 2. You did update your locale stream to switch, You got negotiationneeded.

Also want version are you using master vs 6.0.14 ?

[hthetiot]

https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/c0bfa9aa35ede70c5e5e4979250ec93dc1f8bc7d#diff-90a37269e2c73cda7b742c190542e775c6e2344c6f71e60ac1f1bf5b6511d6ca

Could be related to that change that landed on master and fix front/back due double front in the list

[GlausLivio]

I tested again, it was 6.0.14
Now, it shows local video correctly, when replaceTrack function called, local video is switched correctly.
but In remote side, video stream frozen. not working.
so 1. I did not get any Javascript side error

1. Yes, I got negtiationneeded
   sorry for late response

[hthetiot]

I tested again, it was 6.0.14
Now, it shows local video correctly, when replaceTrack function called, local video is switched correctly.
but In remote side, video stream frozen. not working.
so 1. I did not get any Javascript side error

1. Yes, I got negtiationneeded
   sorry for late response

That progress, can you trigger a renegotiate?

[DaveLomber]

Hi @hthetiot,

I have a quick clarification regarding current 'replaceTrack' implementation

Is it intended to have it work via 'negotiationneeded'?

I just checked the MDM documentation and it says the method works without additional negotiation - it just replaces a track and the receiver party started to get new track immediately, w/o additional logic

Could you please clarify it. Thanks

[hthetiot]

We may be able to deal with that more like the chrome implementation, for now the SHAM require to handle negotiationneeded to be used and yes renegotiate.

Once transceiver or ever if it take too much time it may be implemented without negotiationneeded and renegotiating that why I leave that issue open for now.

If someone can handle negotiationneeded what ever third party library used to see if SHAM work until SHIM is implemented.

[hthetiot]

implementing "negotiationneeded" is part of Perfect negotiation in WebRTC:
https://blog.mozilla.org/webrtc/perfect-negotiation-in-webrtc/

Notice that @RSATom fixed a negotiationneeded bug here and its landed on master https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/619

[RSATom]

@hthetiot don't think #619 is my merit...

[hthetiot]

Ye

@hthetiot don't think #619 is my merit...

Yeah I'm at confused with another merge/fix