Janus-gateway: Problems un-muting after a long muted period

[ancorgs]

This has been already reported (quite some time ago) in the Google group and is one of the oldest open issues in Jangouts.

I'm right now reporting the results observed with Janus 0.1.2. I have now updated the server to the latest master and we plan to re-test and re-report today.

Getting to the point...

We are experiencing problems with the videoroom plugin of Janus when a participant is muted for a long time (let's say around 20 minutes). When un-muting some audio connections cannot be restored.

It's kind of a complex problem. We have detected two different kind of breakages and tested two different approaches for muting. Video and data channels are not affected because we keep them open during the whole conversation. We usually only mute the audio channel.

I will try to describe all the combinations. Let's assume this scenario in which participant A has been muted for 20 minutes and presses the "unmute" button.

        |
  A-- janus --C
        |
        D

Failure 1

This is the most common one. Janus is still receiving the audio from A, but some participants (let's say B and C) cannot hear him because the audio channel that brings that particular audio from Janus to them is broken. Other participants (let's say D) can still hear A. Of course B, C and D can hear each other, as long as none of them has performed a "long-mute". Participant A can hear everybody as well. When this problem arises, we can see A LOT of lines like this in Janus logs.

Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [371884396] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=106498560, seq=45400)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [2298667597] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=104112000, seq=42914)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [1006111317] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=73476480, seq=11002)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [16884969] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=109818240, seq=48858)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [1552633016] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=109818240, seq=48858)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [371884396] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=106499520, seq=45401)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [1006111317] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=73477440, seq=11003)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [2298667597] ... SRTP protect error... err_status_replay_old (len=105-->105, ts=104112960, seq=42915)...
Sep 19 13:55:44 li1355-15 janus[5344]: [ERR] [ice.c:janus_ice_send_thread:3411] [16884969] ... SRTP protect error... err_status_replay_old (len=117-->117, ts=109819200, seq=48859)...

Participants B and C can restart their connection to A (using a button to ignore him and then de-ignoring him again) and everything works again.

Obviously, the chance to get failure 1 increases as the number of participants in the room increases

Failure 2

This happens less often. Looks like Janus stops receiving audio from A. There is nothing relevant in the Janus logs and nobody can hear A. The only way to fix this is with participant A restarting his connections (logging out and in again).

Muting mechanism 1

Since we still want to detect locally that a user is speaking even when he is muted, the mechanism Jangouts uses to (un)mute a channel is sending a "configure" request. Like, for example

{request: "configure", audio: false }

With this mechanism, failure 1 happens relatively often. In long meetings with around 10 participants we experience it several times per meeting. Failure 2 has also been reported to happen with this mechanism, but I have never seen it.

Muting mechanism 2

In order to debug this problem, we performed some tests with an alternative (un)muting mechanism which is, in essence, the same used in janus.js. That is, just stream.getAudioTracks()[0].enabled = false.

With this mechanism, it looks like failure 1 does not happen (we would need more tests to ensure it). Failure 2 happens and I have observed it.

We'll be back

We plan to do more tests with more recent versions of Janus and to implement a hacky workaround in Jangouts if the failures (specially failure 1) are still there, but I hope the information in this report is enough to find a cure (maybe is just a matter of configuring some parameter in Janus).

[lminiero]

{request: "configure", audio: false } is probably not a good idea, and the option itself is unwise (maybe I should remove it). In fact, that flag tells the plugin to drop incoming packets, which means users don't receive any RTP at all for the time the user is muted. This can easily lead to trouble as the one you documented, as the recipient cannot know whether those packets were not received because they were not sent, or because they were lost in the network, as the signalling didn't update the SDP to say the media would be inactive for some time. This can also lead to many NACKs sent by those recipients and, I guess, in those recipients sometimes giving up on the stream.

The right way to do that is the second approach, that is locally muting the stream on the publisher side. This keeps the stream alive by sending silence, which is a very low bandwidth feed, that recipients keep on receiving making everybody happy.

As to failure 2., you should check with Wireshark/tcpdump if those packets are indeed not getting to Janus at all, or if they're being discarded for some reason. If Janus isn't receiving anything, there's not much it can do and is an issue with the sender.

[ancorgs]

Well, you added that option because I asked you to do it. :wink: Initially in this thread and more "formally" in this feature request. So please, don't remove it. It's still useful in several scenarios.

We stopped using local mute because it didn't allow us to have our "you are muted" alert, requested here. If you don't get what I mean, just log into https://jangouts.tk, mute yourself and try to speak.

[lminiero]

Can't you rely on a broadcast message originated by the user who self-mutes, to notify other participants that they just muted? With the TextRoom plugin this is really easy.

[ancorgs]

If I got you correctly, you suggest to keep the audio enabled but tell the others to ignore it in their sides. This is kind of a "security issue" if we don't disable the channel. Muting should imply not sending audio (or sending silence), not just telling the others to block their ears.

On the other hand, we already use the data channel to tell others that we muted/unmuted. So we could process that message in the listener side to also enable/disable the channel there, just to not go out of sync (right now the message is only used to reflect the change in the UI). But I fear the introduced delay - if you unmute and speak right away some listener would still not be listening.

I'll try to come-up with a list of possible solutions/workarounds both in Janus and/or Jangouts side

[lminiero]

Nope, by muting the audio locally I mean on the publisher's side, as in enabled=false. If you need other users to be aware of the fact the user just muted, a broadcast message would do the trick.

[ancorgs]

Yes, that's what we used to do (and still do). But it will break the local "you are trying to speak but you are muted" alert.

[lminiero]

As in, "you've been muted by the moderator"?

[ancorgs]

No. It detects when you are muted but trying to speak. As said, log in https://jangouts.tk, mute yourself and try to speak to see what I mean.

[cacheworks]

Couldn't you still use stream.getAudioTracks()[0].enabled = false to mute and then do your speech detection with a conditional like this: ?

if(stream.getAudioTracks()[0].enabled===false){
    //notify user when speech is detected
}

[ancorgs]

@cacheworks yes of course. The problem is that you cannot detect speech when the audio track is disabled. If the audio track is disabled then no audio is detected for the stream (at least I was not able in my previous experiments). You can see the code for speech detection here https://github.com/jangouts/jangouts/blob/master/src/app/components/feed/speak-observer.factory.js

[lminiero]

I guess you can do that with a separate MediaStream you don't attach to any
PeerConnection, but just use for that feature.

Il 20/set/2016 13:17, "Ancor Gonzalez Sosa" [email protected] ha
scritto:

@cacheworks https://github.com/cacheworks yes of course. The problem is
that you cannot detect speech when the audio track is disabled. If the
audio track is disabled then no audio is detected for the stream (at least
I was not able in my previous experiments). You can see the code for speech
detection here https://github.com/jangouts/jangouts/blob/master/src/app/
components/feed/speak-observer.factory.js

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/meetecho/janus-gateway/issues/620#issuecomment-248272894,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADg5vO7NtrU2FnNtBSMbN-pAK8nIvicxks5qr8DJgaJpZM4KBWm0
.

[cacheworks]

^ +1 Indeed, was trying to find where I saw it, it looks like this application is also just creating a separate stream for Hark to use for speech detection

[ancorgs]

That's indeed number one in my list of wild ideas. The whole list (consider it a brain dump written for myself) follows:

• (1) Use stream.enabled=false and have a second local-only audio stream (not attached to a PeerConnection) to detect activity:
  
  
  
  • Maybe also useful to implement a "test my microphone" screen
  
  
  • Can be considered too tricky?
  
  
• (2) Hijack the audio channel somehow so we detect audio locally but still send silence.
  
  
  
  • Not sure if it is possible. In my previous experiments local audio level for disabled track was always 0 (as expected).
  
  
  • Maybe "duplicating" the stream, somehow....
  
  
• (3) Instead of (or in addition to) configure(audio:false), sent a SDP with audio:recvonly or audio:inactive
  
  
  
  • More consistent with the current approach
  
  
• (4) Any way in which we can modify Janus to pass sync info (RTP) but not audio/video information if the channel is disabled via configure?
• (5) Keep sender as it is now, but also process the status change in the other side, so listeners configure their connections to match emitter status
  
  
  
  • Definitely too tricky compared to other solutions
  
  
• (6) Keep everything as it is but detect when a connection has been dropped and automatically reconnect.
  
  
  
  • Is this doable with "oniceconnectionstatechange" and "peer.iceConnectionState=='disconnected'"?
  
  
  • Useful as a general mechanism to "repair" connections
  
  
  • Can introduce a delay (while we detect and repair)
  
  
  • Also too tricky
  
  

[lminiero]

3) is basically "onhold" but can't be done right now as we don't support renegotiation yet. Not sure what you mean by 4) instead. 1) is probably the easiest approach.

[cacheworks]

I've put together a quick demo with Hark to show how (1) could be achieved
https://jsfiddle.net/p0jqedba/7/

[ancorgs]

The main problem with (1), as proved by @cacheworks's demo (thanks, btw) is that it asks for microphone authorization twice (once for every local stream). The alternative to avoid that is to clone the local stream (so it only gets initialized once).

The problem is that cloning of tracks in Firefox is only available starting at Firefox 38 and I want to support older browsers.

I also had another crazy idea to abuse the local stream, but again I found Firefox limitations (lack of MediaStream.removeTrack().

Still researching.

[cacheworks]

Ah good point, I assumed it would reuse the permission for that tab like Chrome does.
Thanks for the heads up on clone() that might come in useful.

I'll play around some more in Firefox to see if I can get it working while only getting permission once

Up to which version of Firefox would you like to support? adapter.js seems to be bumping their minimum version to 38
https://github.com/webrtc/adapter/issues/192

[ancorgs]

Well, of course Firefox 38 would be nice so we don't increment the version required for Jangouts to work.

[saghul]

FWIW, the current Firefox ESR is 45.X, 38 is basically EoL AFAICT.

[ancorgs]

I have been hitting my head against a wall for some days. I decided to take the following approach.

1) Create the local stream myself using getUserMedia. Let's call it localStream.
2) Hold that localStream to perform the speech detection and generate another stream connected to it. Let's call it transmittedStream
3) Pass transmittedStream to pluginHandle.createOffer instead of the media configuration

This way, I can mute transmittedStream while still observe localStream.

After many attempts generating transmittedStream in several ways. I found out that it was not working because of: "NotSupportedError: track in constructed stream not yet supported (see Bug 1259236)."

So browsers bugs strike back. :disappointed:

[ancorgs]

Ok, so just to keep you informed. I decided to discard the six possible solutions and go for a ugly workaround while I wait any of the solutions to be viable. To fine-tune my ugly workaround I would appreciate some advise from webRTC experts (see question at the end).

1) Discarded because browsers ask for microphone permissions twice.
2) Discarded because it cannot be done in Firefox < 48 (maybe even Firefox < 49) due to browser bugs. RTCPeerConnection.addStream fails if the stream is not the absolutely pristine stream returned by getUserMedia.
3) Discarded because Janus does not support renegotiation. @lminiero Are there plans to support it?
4) Discarded because it was probably a nonsense. :wink: Just to clarify, I was suggesting to keep sending sync information but not sending the audio itself.
5) Discarded because it's complex and could introduce delays when un-muting (the other side would need some time to configure its side of the connection).
6) Discarded because... well, was quite a crazy idea from the very beginning.

So, while I expect (2) or (3) to be viable (just a matter of time), here is my ugly workaround:

While the emission is suspended (via configure(audio: false)), I will have a timer every X seconds that will execute this:

• disable the audio channel (speech detection will stop working temporarily)
• send a configure(audio: true) to start sending silence allowing the other side to resync
• wait Y seconds to make sure the resync takes place
• send a configure(audio: false) to restore the initial situation
• enable the audio channel again (speech detection will work again)

I want to maximize X and to minimize Y. In my experience, X can be quite high (several minutes). My question to the experts is: what would be the minimum Y?

Thanks for the help and the involving. I really appreciate how helpful Janus community is (even when the problem is not exactly related to Janus itself).

[lminiero]

3) Discarded because Janus does not support renegotiation. @lminiero Are there plans to support it?

Looks like you missed this :wink:
As soon as you guys help me merge that stuff, I'll be able to focus on new things (renegotiation, multistream, etc.).

[lminiero]

As to the video|audio: false option in configure, I was thinking about what may be causing the replay errors in SRTP, and I guess it's likely related to the fact that, between when you stop relaying the medium and when you relay it again, the incoming sequence numbers and/or timestamps are close by to the last that were relayed, possibly lower than the last being sent, which may be causing the problem.

One possible way to fix this is to force a reset of those values when restarting a paused stream, using what we already have in the code to handle switching publishers. More specifically, when a subscriber switches publisher, obviously SSRC/sequence number/timestamp of the new source are completely different: in order to make sure a publisher can seamlessly keep on playing the incoming feed without disruption, we modify the RTP header so that it seems to be the natural continuation of the old publisher. We may do something like this also for when we pause one of the streams, e.g.:

1. you're sending audio, and then you configure audio:false;
2. some time passes, you configure audio:true again;
3. we set the previous SSRC of all viewers to a bogus value (e.g., 0), so that even if the SSRC didn't really change we force the switching mechanism to kick in;
4. the sequence numbers and timestamps are fixed automatically as if they were started again.

This may be worth trying, but I'm not sure it would work, as while this would do the right thing for sequence numbers, timestamps for the packet immediately after the resume will actually be a continuation of the last packet being sent who knows how long ago, instead of being the timestamp associated with the current presentation time, so not sure what the behaviour in the browser would be. Anyway, I'll try to prepare a small patch to see if this works.

[lminiero]

@ancorgs can you let me know if this patch fixes those SRTP errors for you?

[ancorgs]

@lminiero Thanks a lot!

I will try to get enough people to (not)reproduce it tomorrow https://groups.google.com/d/msg/jangouts/d5ZOePOU09M/RTi603twDQAJ

[lminiero]

@ancorgs as a side note, that patch I made is for the configure you use on publishers: it has no effect when you do the same on a specific viewer, that is to temporarily stop receiving something only on a selected user (something I think you also asked for some time ago). Not sure which of the two you're using for the muting feature.

[ancorgs]

Yes, we use the configure on publishers to implement muting, so your patch seems to address the exact problem we have.

We did a test this morning and at some point we were still able to see the err_status_replay_old errors in the Janus logs. Interestingly enough, even with the errors popping up, we notice no symptoms in Jangouts. Everybody was still able to hear everybody.

So I'm not sure what to think. I will deploy the new version in production (I observed no drawback compared to previous versions) and will keep you informed.

[ancorgs]

Since we deployed the new version of janus in the server, my team has had two quite long meetings with zero audio un-muting problems.

Curious enough, the problem still appeared but associated to video streams. We made a break and most participants "muted" both the video and the audio channels. Audio channels resumed flawlessly (as said, we experience no problems in that regard), but many video channels got "frozen" and we saw a lot of those err_status_replay_old lines in the server. It makes not much sense since the referenced commit should fix the problem for both channels, but honestly I don't care much.

Since the video problem is easy to work-around in Jangouts (we don't need that configure(video: false), since we have no reason to monitor the local video), I'm absolutely happy with the fix and closing the issue.

Thanks.

[lminiero]

Nice, thanks for the feedback!

[lminiero]

That said, I think I know why it fails for video... stupid typos! I'll fix this right away...