Janus-gateway: Consistent packet losses when getting RTSP stream

Created on 26 Apr 2018 · 24Comments · Source: meetecho/janus-gateway

Hi,
I'm using the streaming plugin to get a stream from a local RTSP server. The stream seems to run well. However, I keep getting packet losses every 30-40 seconds. This of course freezes the stream for a few seconds and reconnects.

I've used VLC to check the stream and it seems to not cause packet losses/freezes. I've also lowered the resolution, bit rate, image quality and frame rate, but it seems to still cause the same problem every 30-40 seconds.

Thanks in advance!

Source

RaymondLeow

Most helpful comment

Latching would be trivial to do, so I think I'll just do that. No intention of going towards RTSPS or interleaved mode instead :wink:

lminiero on 27 Apr 2018

🎉1 👍1

All 24 comments

You should check the traffic with something like wireshark/tcpdump to see why that happens. RTSP streaming works just fine for me so it may be specific to your setup.

lminiero on 26 Apr 2018

Hi and thanks for the quick reply!

So what I find in the Janus logs is that Janus can't find media so it tries to reconnect, according to this dump.

[WARN] [rtsp-test] 5s passed with no media, trying to reconnect the RTSP stream
* Rebuilt URL to: rtsp://192.168.1.65/
*   Trying 192.168.1.65...
* TCP_NODELAY set
* Connected to 192.168.1.65 (192.168.1.65) port 554 (#0)
> DESCRIBE rtsp://192.168.1.65 RTSP/1.0
CSeq: 1
Accept: application/sdp

However, what I find with Wireshark is that the camera keeps sending video data all the time. Then Janus suddenly sends a request to restart even though the camera sends packets within that five second window. 192.168.1.65 is the location of the camera and 192.168.1.223 is the location of Janus.

"No.","Time","Source","Destination","Protocol","Length","Info"
"1","0.000000000","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"2","0.000023519","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"3","0.040309843","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"4","0.040339015","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"5","0.040633979","192.168.1.65","192.168.1.223","UDP","1497","32769  >  54409 Len=1455"
"6","0.040645423","192.168.1.65","192.168.1.223","UDP","145","32769  >  54409 Len=103"
"7","0.080879016","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"8","0.080913028","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"9","0.121340563","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"10","0.121369457","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"11","0.161842398","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"12","0.162235726","192.168.1.65","192.168.1.223","UDP","1457","32769  >  54409 Len=1415"
"13","0.162244417","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"14","0.187213108","192.168.1.223","192.168.1.65","TCP","74","45568  >  554 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=3118466077 TSecr=0 WS=128"
"15","0.187640443","192.168.1.65","192.168.1.223","TCP","74","554  >  45568 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=136562675 TSecr=3118466077 WS=16"
"16","0.187694668","192.168.1.223","192.168.1.65","TCP","66","45568  >  554 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=3118466078 TSecr=136562675"
"17","0.187799712","192.168.1.223","192.168.1.65","RTSP","141","DESCRIBE rtsp://192.168.1.65 RTSP/1.0"
"18","0.188217724","192.168.1.65","192.168.1.223","TCP","66","554  >  45568 [ACK] Seq=1 Ack=76 Win=14480 Len=0 TSval=136562675 TSecr=3118466078"
"19","0.191466833","192.168.1.65","192.168.1.223","RTSP","237","Reply: RTSP/1.0 401 Unauthorized"
"20","0.191501881","192.168.1.223","192.168.1.65","TCP","66","45568  >  554 [ACK] Seq=76 Ack=172 Win=30336 Len=0 TSval=3118466082 TSecr=136562676"
"21","0.191806319","192.168.1.223","192.168.1.65","RTSP","300","DESCRIBE rtsp://192.168.1.65 RTSP/1.0"
"22","0.194051393","192.168.1.65","192.168.1.223","RTSP/SDP","718","Reply: RTSP/1.0 200 OK"
"23","0.194539411","192.168.1.223","192.168.1.65","RTSP","348","SETUP rtsp://192.168.1.65/trackID=0 RTSP/1.0"
"24","0.196287406","192.168.1.65","192.168.1.223","RTSP","261","Reply: RTSP/1.0 200 OK"
"25","0.196639780","192.168.1.223","192.168.1.65","RTSP","367","SETUP rtsp://192.168.1.65/trackID=1 RTSP/1.0"
"26","0.197660575","192.168.1.65","192.168.1.223","RTSP","261","Reply: RTSP/1.0 200 OK"
"27","0.197941603","192.168.1.223","192.168.1.65","RTSP","305","PLAY rtsp://192.168.1.65 RTSP/1.0"
"28","0.202229278","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"
"29","0.202253521","192.168.1.65","192.168.1.223","UDP","214","32771  >  48639 Len=172"

I'm not familiar with how to post dumps here since Pastebin gives me the dumps as is. I can post the dumps in other ways if you so prefer.

RaymondLeow on 27 Apr 2018

FYI current Janus Gatway implementation of RTSP protocol inside streaming plugin can't work if there are NAT between rtsp source and Janus Gateway Server. But don't sure if it's your case...
I got the same error when tried to use rtsp server from internet (technically it was my own server, but it doesn't matter) with Janus inside my local network behind router with NAT.

RSATom on 27 Apr 2018

@RaymondLeow are you sure the camera is sending to the right ports? You can check if the ports you see in Wireshark match those negotiated in RTSP.

lminiero on 27 Apr 2018

@RSATom that's true, but not sure what we can do to fix that. How do RTSP clients usually handle this, considering they don't do ICE AFAIK? Do they send some random data to the ports the RTSP server negotiated for media, to let it know the packet sender is where the media should be sent?

lminiero on 27 Apr 2018

If I remember correctly GStreamer sends some dummy UDP packet to create hole in NAT (https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-rtspsrc.html#GstRTSPSrc--nat-method). And if it not working it could embed streams data to rtsp control channel (so called interleaved mode).

RSATom on 27 Apr 2018

Looks like VLC is doing some sort of latching, which is similar to what I guessed above: VLC sends a packet to the RTSP server a couple of times (to be sure it's not lost, I assume), and then media starts flowing. Wireshark doesn't recognize the packet though, or at least it thinks it's RTP but it's not. I guess it will be one of the latching things discussed here: https://tools.ietf.org/html/rfc7604

lminiero on 27 Apr 2018

Anyway, tbh, don't think it worth try to fix it inside streaming plugin. Since, for example, add interleaved mode to current implementation could be complex task IMHO.

RSATom on 27 Apr 2018

Other missing feature in current streaming plugin is support of secure RTSP (rtsps://) - it was the main reason why I've started GStreamer pluigin... it will be much simple just use GStreamer for RTSP I think...

RSATom on 27 Apr 2018

Latching would be trivial to do, so I think I'll just do that. No intention of going towards RTSPS or interleaved mode instead :wink:

lminiero on 27 Apr 2018

🎉1 👍1

Of course it will be great if you will add it!

RSATom on 27 Apr 2018

Implemented latching in this commit here: https://github.com/meetecho/janus-gateway/commit/49416c236f14b0a1aa51d0e6b8d565cc1cb90a06
Verified that it works as expected with a public RTSP stream, where it failed before.

@RaymondLeow notice that I don't believe this is related to your issue, as media seems to be flowing in your scenario, where this fixes issues where media would never be set up. For your specific problem, make sure the camera is sending media to the right ports: you can check if the ports you see in Wireshark match those negotiated in RTSP (Transport header in the 200 OK to SETUP).

lminiero on 27 Apr 2018

🎉1

Yes, the ports are different but isn't that supposed to happen? The media is meant to be sent to different ports as opposed to the ports for reconnections? VLC has the same behaviour, having different ports.

Upon further investigation I find that the camera also sends these FIN-ACK and RST-ACK messages right before a streaming reconnection occurs. They look like the dump below.

"No.","Time","Source","Destination","Protocol","Length","Destination Port","Destination Port","Info"
"18694","41.095672078","192.168.1.65","192.168.1.223","TCP","66","","58804","554  >  58804 [FIN, ACK] Seq=1487 Ack=1132 Win=18768 Len=0 TSval=139052540 TSecr=3143333008"
"18695","41.095823914","192.168.1.65","192.168.1.223","TCP","66","","58804","554  >  58804 [RST, ACK] Seq=1488 Ack=1132 Win=18768 Len=0 TSval=139052540 TSecr=3143333008"

I checked on VLC and the behaviour is (according to me) pretty much similar in Wireshark. It also has FIN-ACK and RST-ACK as well as sending media to different ports than what it renegotiates.

@RSATom Both the camera and Janus are behind a local network.

RaymondLeow on 27 Apr 2018

Yes, the ports are different but isn't that supposed to happen?

No: Janus binds to whatever it puts in client_ports, and the RTSP server will send media from whatever it puts in server_ports. If you're in a local network, with no NAT, you should see RTP from the first server_port to the first client_port.

lminiero on 27 Apr 2018

Notice that I didn't mean different ports in different sessions... I meant, is the RTSP server sending RTP from the port it advertised, to the port Janus advertised? If there are mismatches there, Janus won't receive media, hence the packet losses and reconnections.

lminiero on 27 Apr 2018

@RaymondLeow any update?

lminiero on 1 May 2018

Sorry, I'm on a shorter vacation at the moment. Will come with an update tomorrow.

RaymondLeow on 1 May 2018

No worries! I'm abroad for a couple of days myself, so there's no hurry :+1:
Enjoy your vacation!

lminiero on 1 May 2018

I'm a bit confused to what to look for so I might bring more (or less) information than necessary. Excuse me for the long log below, but it may be relevant. It's part of what the Janus logs at startup.

I filtered to only watch RTP and RTSP messages (not UDP). What I found in Wireshark is that the audio RTP packets is sent from port 32769 to 37710, while video is being sent from 32771 to 47396. This fits with what the logs below are showing. When starting up Janus and connecting to RTSP server via RTSP, the Janus resides on port 55964. The audio and video keep sending RTP packets to the these ports (37710 resp. 47396) until the reconnection occurs.

In the reconnection to the RTSP server, it occurs between port 554 on the RTSP server (which is the same as in the startup) and 55968 (_not_ 55964) on the Janus. The ports which video and audio are sent to are different as well after reconnection, but they seem to fit with what the log is writing.

* Rebuilt URL to: rtsp://192.168.1.65/
* Found bundle for host 192.168.1.65: 0x15e5980 [can pipeline]
* Re-using existing connection! (#0) with host 192.168.1.65
* Connected to 192.168.1.65 (192.168.1.65) port 554 (#0)
* Server auth using Digest with user 'admin'
> SETUP rtsp://192.168.1.65/trackID=0 RTSP/1.0
CSeq: 3
Transport: RTP/AVP/UDP;unicast;client_port=37710-37711
Authorization: Digest username="admin", realm="Surveillance Server", nonce="85056201", uri="rtsp://192.168.1.65/trackID=0", response="8ea4a4d04e16009529290a0259e25103"

< RTSP/1.0 200 OK
< CSeq: 3
< Server: Rtsp Server V20
< Session: 73339728
< Transport: RTP/AVP;unicast;client_port=37710-37711;source=192.168.1.65;server_port=32769-32770;ssrc=a046
< 
* Connection #0 to host 192.168.1.65 left intact
* Rebuilt URL to: rtsp://192.168.1.65/
* Found bundle for host 192.168.1.65: 0x15e5980 [can pipeline]
* Re-using existing connection! (#0) with host 192.168.1.65
* Connected to 192.168.1.65 (192.168.1.65) port 554 (#0)
* Server auth using Digest with user 'admin'
SETUP answer:RTSP/1.0 200 OK
CSeq: 3
Server: Rtsp Server V20
Session: 73339728
Transport: RTP/AVP;unicast;client_port=37710-37711;source=192.168.1.65;server_port=32769-32770;ssrc=a046


  -- RTSP server_port (video): 32769
> SETUP rtsp://192.168.1.65/trackID=1 RTSP/1.0
CSeq: 4
Session: 73339728
Transport: RTP/AVP/UDP;unicast;client_port=47396-47397
Authorization: Digest username="admin", realm="Surveillance Server", nonce="85056201", uri="rtsp://192.168.1.65/trackID=1", response="41c518f3c57edd461c650d0606e13a0d"

< RTSP/1.0 200 OK
< CSeq: 4
< Server:  Rtsp Server V20
< Session: 73339728
< Transport: RTP/AVP;unicast;client_port=47396-47397;source=192.168.1.65;server_port=32771-32772;ssrc=a0c6
< 
* Connection #0 to host 192.168.1.65 left intact
SETUP answer:RTSP/1.0 200 OK
CSeq: 4
Server: Rtsp Server V20
Session: 73339728
Transport: RTP/AVP;unicast;client_port=47396-47397;source=192.168.1.65;server_port=32771-32772;ssrc=a0c6


  -- RTSP server_port (audio): 32771
RTSP audio latching: 0.0.0.0:32771
RTSP video latching: 0.0.0.0:32769
Sending PLAY request...
* Rebuilt URL to: rtsp://192.168.1.65/
* Found bundle for host 192.168.1.65: 0x15e5980 [can pipeline]
* Re-using existing connection! (#0) with host 192.168.1.65
* Connected to 192.168.1.65 (192.168.1.65) port 554 (#0)
* Server auth using Digest with user 'admin'
> PLAY rtsp://192.168.1.65 RTSP/1.0
CSeq: 5
Session: 73339728
Range: 0.000-
Authorization: Digest username="admin", realm="Surveillance Server", nonce="85056201", uri="rtsp://192.168.1.65", response="fc8f56c3b05d6caeea8108c88c621f56"

< RTSP/1.0 200 OK
< CSeq: 5
< Server: Rtsp Server V20
< Session: 73339728;timeout=120
< Range: 0.000-
< RTP-Info: url=rtsp://192.168.1.65:32769/0/trackID=0;seq=40136848;rtptime=841431124,url=rtsp://192.168.1.65:32771/0/trackID=1;seq=9203679;rtptime=4274317480
< 
* Connection #0 to host 192.168.1.65 left intact
PLAY answer:RTSP/1.0 200 OK
CSeq: 5
Server:  Rtsp Server V20
Session: 73339728;timeout=120
Range: 0.000-
RTP-Info: url=rtsp://192.168.1.65:32769/0/trackID=0;seq=40136848;rtptime=841431124,url=rtsp://192.168.1.65:32771/0/trackID=1;seq=9203679;rtptime=4274317480

On another note, I have written in the streaming configuration file to not have audio. It also got the log Audio NOT enabled, Video enabled, but there seems to still be an audio port open in the log above. I also don't know why I'm getting audio packets even if the audio is not enabled.

RaymondLeow on 2 May 2018

In the reconnection to the RTSP server, it occurs between port 554 on the RTSP server (which is the same as in the startup) and 55968 (not 55964) on the Janus.

Absolutely normal, the new RTSP client is bound to a different port. Your Wireshark checks seem to confirm the used ports are correct. What you should check, now, is if the packet loss you experience on the WebRTC side is reflected on the RTSP side as well: if the RTSP server misses some packets in its delivery, they'll be missing in WebRTC as well clearly, as Janus simply relays what it gets.

On another note, I have written in the streaming configuration file to not have audio.

I haven't tested what happens if you choose not to have audio but the RTSP server offers it anyway, It's likely we're not handling this properly right now, and offer it via WebRTC anyway.

lminiero on 3 May 2018

So I just read the ports wrong _(sorry!)_. The RTSP server does not send RTP packets all the time. The RTSP server stops sending RTP packets after the FIN-ACK and RST-ACK request, according in the log below. There is a five second delay until the Janus replies.

So the problem is probably something with this request. Do I have to "ping" the RTSP to stop this FIN-ACK and RST-ACK request? There is no such request coming from the RTSP server when I use VLC, according to Wireshark.

"No.","Time","Source","Destination","Protocol","Length","Source Port","Source Port","Destination Port","Destination Port","Info"
"18476","58.040745474","192.168.1.65","192.168.1.223","RTP","1497","32769","","54050","","PT=DynamicRTP-Type-96, SSRC=0xA046, Seq=53251, Time=2458928432"
"18477","58.040766349","192.168.1.65","192.168.1.223","RTP","134","32769","","54050","","PT=DynamicRTP-Type-96, SSRC=0xA046, Seq=53252, Time=2458928432, Mark"
"18486","58.121292773","192.168.1.65","192.168.1.223","RTP","1360","32769","","54050","","PT=DynamicRTP-Type-96, SSRC=0xA046, Seq=53253, Time=2458937432, Mark"
"18510","58.238897239","192.168.1.65","192.168.1.223","TCP","66","","554","","58506","554  >  58506 [FIN, ACK] Seq=1487 Ack=1132 Win=18768 Len=0 TSval=191455093 TSecr=3616331420"
"18511","58.239063208","192.168.1.65","192.168.1.223","TCP","66","","554","","58506","554  >  58506 [RST, ACK] Seq=1488 Ack=1132 Win=18768 Len=0 TSval=191455093 TSecr=3616331420"
"19182","63.206413485","192.168.1.223","192.168.1.65","TCP","74","","58824","","554","58824  >  554 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=3616368391 TSecr=0 WS=128"
"19183","63.207070886","192.168.1.65","192.168.1.223","TCP","74","","554","","58824","554  >  58824 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=191455590 TSecr=3616368391 WS=16"
"19184","63.207106598","192.168.1.223","192.168.1.65","TCP","66","","58824","","554","58824  >  554 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=3616368391 TSecr=191455590"
"19187","63.207175605","192.168.1.223","192.168.1.65","RTSP","141","","58824","","554","DESCRIBE rtsp://192.168.1.65 RTSP/1.0"
"19193","63.207623186","192.168.1.65","192.168.1.223","TCP","66","","554","","58824","554  >  58824 [ACK] Seq=1 Ack=76 Win=14480 Len=0 TSval=191455590 TSecr=3616368391"
"19201","63.210051361","192.168.1.65","192.168.1.223","RTSP","237","","554","","58824","Reply: RTSP/1.0 401 Unauthorized"
"19202","63.210070721","192.168.1.223","192.168.1.65","TCP","66","","58824","","554","58824  >  554 [ACK] Seq=76 Ack=172 Win=30336 Len=0 TSval=3616368394 TSecr=191455590"
"19203","63.210266054","192.168.1.223","192.168.1.65","RTSP","300","","58824","","554","DESCRIBE rtsp://192.168.1.65 RTSP/1.0"
"19204","63.211467739","192.168.1.65","192.168.1.223","RTSP/SDP","718","","554","","58824","Reply: RTSP/1.0 200 OK[Malformed Packet]"
"19205","63.211721338","192.168.1.223","192.168.1.65","RTSP","348","","58824","","554","SETUP rtsp://192.168.1.65/trackID=0 RTSP/1.0"
"19206","63.212912001","192.168.1.65","192.168.1.223","RTSP","261","","554","","58824","Reply: RTSP/1.0 200 OK"
"19207","63.213193152","192.168.1.223","192.168.1.65","RTSP","367","","58824","","554","SETUP rtsp://192.168.1.65/trackID=1 RTSP/1.0"
"19208","63.214285826","192.168.1.65","192.168.1.223","RTSP","261","","554","","58824","Reply: RTSP/1.0 200 OK"
"19209","63.214537919","192.168.1.223","192.168.1.65","RTSP","305","","58824","","554","PLAY rtsp://192.168.1.65 RTSP/1.0"
"19221","63.246175907","192.168.1.65","192.168.1.223","TCP","66","","554","","58824","554  >  58824 [ACK] Seq=1214 Ack=1132 Win=18768 Len=0 TSval=191455594 TSecr=3616368399"
"19227","63.630130839","192.168.1.65","192.168.1.223","RTP","99","32769","","53111","","PT=DynamicRTP-Type-96, SSRC=0xA046, Seq=53255, Time=2459432882, Mark"
"19228","63.630779889","192.168.1.65","192.168.1.223","RTP","60","32769","","53111","","PT=DynamicRTP-Type-96, SSRC=0xA046, Seq=53256, Time=2459432882, Mark"
"19229","63.632223560","192.168.1.65","192.168.1.223","RTP","1497","32769","","53111","","PT=DynamicRTP-Type-96, SSRC=0xA046, Seq=53257, Time=2459432882"

RaymondLeow on 3 May 2018

FIN-ACK/RST-ACK simply means one of the two is closing the TCP connection used for the RTSP signalling. Who's closing it?

lminiero on 8 May 2018

Very strange. I can't find any (simple) FIN messages being sent when reading from Wireshark and tcpdump.

In addition to VLC, I have now tried to stream the RTSP server with Kurento. It doesn't send this FIN-ACK/RST-ACK message either nor does it freeze on the client.

RaymondLeow on 8 May 2018

I close this issue for now as I have no clue what to do. Thanks anyway.

RaymondLeow on 21 May 2018

Was this page helpful?

0 / 5 - 0 ratings