Janus-gateway: VideoTrack mute / unmute delay

[leonuh]

Hi there,

The stream is delayed on the "receiver side" after disable and enable the videoTrack.
The delay has gone after page refresh (rejoin)
To test:
Change: getAudioTracks to getVideoTracks in https://github.com/meetecho/janus-gateway/blob/master/html/videoroomtest.js#L350

Any idea?

[lminiero]

Nothing we can do, that's a browser thing. When a track is muted, silence/black video is sent, and when unmuted the stream is restored: as such, something is always flowing. Nothing Janus has possibility to have effect on, since as far as Janus is concerned the media never stopped flowing.

Closing as not an issue.

PS: you may want to look into the new mute-related methods in janus.js as they make things simpler than having to manipulate tracks directly: https://github.com/meetecho/janus-gateway/commit/59d9ba8521c51140630a6a7fe3dad8aa2b73b784

[leonuh]

Thank you for your answer. I am using these methods already.

Just to avoid any misunderstanding:
The audio keeps the synchronicity. The video is not in sync with the audio after the stream was restored. It is not a problem for the audio regardless of the mute / unmute.

Do you have any suggestion / workaround to trigger a kind of reset for the stream (eg. a new stream instance after mute / unmute for the remote users)?

[lminiero]

Just checked with apprtc.appspot.com, which is peer-to-peer, and there the problem doesn't happen... just open a session among two peers, and on the JavaScript console of one of them use this command to mute/unmute the video:

document.getElementById('mini-video')._srcObject.getVideoTracks()[0].enabled = false; // mute
document.getElementById('mini-video')._srcObject.getVideoTracks()[0].enabled = true; // unmute

Apparently there there's no delay at all, which means that there's indeed something that is happening withing Janus that is confusing viewers. I'll have to check what's happening.

[leonuh]

As a workaround (until the bugfix) it works without any delay:

plugin.send({message: { request: 'configure', video: true/false }});

[lminiero]

This still has the publisher send stuff, though (even though it's discarded by the plugin).

[leonuh]

Unfortunately the previous workaround doesn't work properly either cause of the earlier bug: If the video is false then the audio doesn't work for the new feeds. It works only if I toggle the video at least for a moment from the publisher's side. It seems like the audio depends on the video data somehow.

I tried out other ideas to disable / enable camera separately but non of them works - I assume the mcu doesn't handle them yet:

With tracks:
I store the stream's videoTrack separately, then:

//Mute video
stream.removeTrack(videoTrack);
//Unmute video
stream.addTrack(videoTrack);

When I remove the track it stops the video for the subscribers, but there was no effect when I added back. Besides none of the events were triggered on the subscribers side (onaddtrack, onremovetrack)

With streams:
I created a separated stream which stores just the audioTrack:
audioStream = new window.webkitMediaStream(theOriginalStreamsAudiotrack)

//Mute video
plugin.webrtcStuff.pc.removeStream(originalStream)
plugin.webrtcStuff.pc.addStream(audioStream)
//Unmute video
plugin.webrtcStuff.pc.removeStream(audioStream)
plugin.webrtcStuff.pc.addStream(originalStream)

Unfortunately this didn't work either, very similar to the previous one (although if I just remove and add back the original stream then it works). Besides none of the events were triggered on the subscribers side (onaddstream, onremovestream)

Do you have any (temporary) solution / idea?

[lminiero]

Still no solution, sorry. I'm not even sure this is NOT happening with pure p2p solutions, for instance, as when I tried pausing video in the EchoTest demo and resuming it when the bitrate was high, the resume part was smooth, which means it might be a bitrate-related issue (e.g., happening at the lower bitrates we force in the videoroom). If so, this would be a bug in Chrome and not in Janus. Anyway, I made this test weeks ago, so it might have been occasional and not related after all. Besides, I have not tested if this issue happens with Firefox as well or not: have you?

[leonuh]

It is an issue in Firefox too. I attach 2 screenshots that can help (from chrome://webrtc-internals): Before and after mute / unmute.
Before
After

[lminiero]

The high googJitterBufferMs and googTargetDelayMs are interesting. I found this issue from about a year ago https://bugs.chromium.org/p/webrtc/issues/detail?id=4355
Not many updates or feedback though.

[lminiero]

It looks like the true "culprit" is googJitterBufferMs, as googTargetDelayMs seems to depend on it. Looking at the chrome code (not really familiar with it), apparently when you exceed the delay bounds, as would be the case of receiving a video stream again after a pause of some time, the jitter buffer is resetted, which I guess makes sense. It also returns an error, though. This means the delay should be "absorbed" after a bit. I don't know how the browser is calculating the Jitter here, and how we can help limit its impact. We don't do much RTCP right now, so not sure if that might be it.

[leonuh]

I tried it out. When I unmute the client the googJitterBufferMs and googTargetDelayMs's value were between 500 - 600ms. Then I continuously watched. In every second the values decreased a bit. After 3-4 minutes they reached an acceptable state (100ms). After that I mute / unmute again, and the process started over from 5xx ms.

[lminiero]

I did some Wireshark tests to see what was wrong, and I could verify that, RTP-wise, we're doing the right thing: the timestamp and sequence numbers are correctly overwritten, and the same inter-packet differences are preserved. What I did notice is that, after a resume, the first keyframe originated by the browser has the expected timestamp, but the packet following that doesn't: the sequence number is prevSeq+1, while the timestamp is prevTs+197640, which means "render this packet ~2s later than the previous one". This explains why the rendering is delayed: the timestamp itself is broken. It's not something Janus does, the browser creates it that way, and we simply adapt it as part of the RTP headers rewriting to keep it coherent on the viewer's side, and in fact the same exact difference is present there as well.

This makes me think it is indeed a browser bug and not a Janus one.

PS: this is on a Chrome 49.

[lminiero]

@leonuh can you check if you can still replicate the issue on the branch we have in #467? My original feeling at the time was that we were lacking something RTCP-wise: I just tried muting/unmuting the video in both EchoTest and VideoRoom and could not replicate the lagging video anymore, which means we may have fixed it there.

[leonuh]

I think I can check it tonight - thank you

[leonuh]

It is still buggy for me - be careful, if you republish the stream then it removes the delay (thats how I created a workaround)

[lminiero]

I wasn't republishing: to test, I'm getting the local video track and disabling/enabling it, e.g.,

echotest.webrtcStuff.myStream.getVideoTracks()[0].enabled=false
[... wait some seconds ...]
echotest.webrtcStuff.myStream.getVideoTracks()[0].enabled=true

for the EchoTest, and,

sfutest.webrtcStuff.myStream.getVideoTracks()[0].enabled=false
[... wait some seconds ...]
sfutest.webrtcStuff.myStream.getVideoTracks()[0].enabled=true

for the VideoRoom. In both cases, using a Janus built from the PR I mentioned, I couldn't replicate the issue, and those google*Ms values we saw increase were always low.

If you've tested the right version and it still isn't working, then I'll try to replicate this again (maybe waiting for more seconds before enabling video transmission again). Who knows, maybe the issue is somewhere else.

[lminiero]

As a side note, I've tested with Chrome 50, so not sure if anything was improved there as well with respect to that: it may be, as I've just tried with Firefox too (45 and 48) and I indeed still see the problem there. As such, it may partly be a matter of client side too.

[lminiero]

Did one more test by involving the RTP forwarding feature in the VideoRoom. I basically started a gstreamer instance that would receive VP8 frames via RTP and render them, and instructed the plugin to forward my publisher there:

sfutest.send({message:{request:"rtp_forward",secret:"adminpwd",room:1234,publisher_id:myid,host:"127.0.0.1",video_port:17004}})

Then did the same enable=false/true trick in Firefox, and while Firefox had the delayed video after resuming the transmission, the playback was perfect in gstreamer instead. This seems to reinforce the idea that it may be more an issue in the client side, rather than anything we're doing wrong (or missing) in Janus itself.

[lminiero]

Just FYI, I've posted on the Firefox group about this, as I'm starting to believe it might be a bug there (as anticipated it doesn't seem to bother Chrome anymore): https://groups.google.com/forum/#!topic/mozilla.dev.media/2Gq_QVc9Sps
Hopefully they'll have more info for us to evaluate.

[lminiero]

Linking here as a reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1256679

[lminiero]

Issue apparently solved in Firefox (worked for me in Nightly), so I'm closing the issue here as well.

[leonuh]

I am sorry but it is still an issue - latest Janus with the latest stable Chrome (Ubuntu) - check these (googCurrentDelayMs, googJitterBufferMs, googTargetDelayMs):

How to replicate: Just mute and unmute 2-4 times

[ibc]

Just wondering if you have tested with Chrome 50 (current stable) and Canary (52) and the video delay happens there or not.

We have a similar problem with our MCU (audio OK but video delayed in receiver with a very high googJitterBufferMs: 3019 in the received video) which just happens in Chrome 50, but does not happen in Chrome Canary 52 nor in Firefox 46.

[lminiero]

I remember testing with Chrome 50 at the time and it looked like it was fixed, but apparently it isn't, from what you guys say. It seems to have been fixed in Firefox according to the issue I opened instead. If it's fixed in Canary 52, this seems to confirm it was a browser issue, and so there's not much we can do about it if not wait for 52 to become the new stable.

[ibc]

@lminiero thanks for your response. Important update: it was a bug in our MCU when it comes to generate the RTCP timestamp. Fixed :)

What it's interesting is the fact that, with the same MCU, both Firefox 46 and Canary do not produce the ~4 seconds video delay. Who knows :)

[lminiero]

Ah, thanks for the hint, it might have to do with RTCP timestamps in our case as well, then (although I thought RTCP timestamps don't really have to match the RTP ones?). I'll give this a look when I get back to the office, as I'm abroad now.

[murillo128]

BTW, check the googRTT of the ssrc_send stream, that caused the value googTargetDelayMS increase in the sssrc_recv in our case.
Also in our case it was due to wrong value feed LSR/DLSR of the receiver report. Not sure if it would be applicable to you.

[lminiero]

It's likely that's the case, as the RTCP termination stuff in Janus is pretty young and so probably flawed somewhere. I'll check as soon as I can, thank you both guys for the hints!

[leonuh]

Thank you for your hard work too!

[lminiero]

Reopening so that I don't forget about this (again).

[lminiero]

Played with this again, today, and I can't find anything wrong in how we're doing the RR and the LSR stuff. We (or at least I) never got 3s od delay, though: it's always at about 300/500ms after a mute/unmute, and to me it only happens if the bitrate is low (that is, easily reproducible at ~256kbps), and not when it's higher.

I verified, playing with the VideoRoom plugin, that if I do the mute/unmute thing, existing subscribers will get the video delayed, while new subscribers joining after that get it correct. This means that the media stream as sent by the browser and as forwarded by Janus is correct, and that the issue is on the receiving side of those who witnessed the mute/unmute. We do send coherent Sender Reports, so I don't think the issue is there either (unless Chrome expects a Report Block in there too?).

My guess is that, just as this was fixed in Firefox (works fine in Firefox 49, in fact), this needs to be fixed in Chrome as well. I'll update a post I saw from some time ago that never got any attention.

[lminiero]

I updated the issue they had opened, and they managed to replicate the issue:
https://bugs.chromium.org/p/webrtc/issues/detail?id=4355

[bizzr3]

i have the same issue,
fresh installation
chrome 58
ubuntu desktop
remote audio starts exactly starts after 2 seconds all the time, but after 2 seconds there is no delay at all, i don't have any issue with firefox all perfect in firefox.
any update about this issue ?

[bizzr3]

checked by windows machine as well, i don't have any issue with windows machine, i have deployment tomorrow morning, right after i will provide Wshark log for both windows and ubuntu.

[lminiero]

Closing as apparently not an issue anymore on recent Chrome versions. They closed the issue I opened on their tracker as well.