Scrcpy: Huge latency and lags over TCP

[R3nd0gg]

Hello, I was trying to create apps for internal use. Basically app would ssh remote port forward port, device would listen for ADB on that port and it would allow me to connect to given device behind NAT. Problem is that device is totally uncontrolable using any arguments (-b1M -m800) when controlling it using mentioned SSH connections. I have read many issues where you talk about TCP blocking problem. Is there way I at least minimize latency given my use case? I have also tried to run over VPN, it is slightly better but still pretty much unusable. Device is connected in network with 5Mbit upload which I think is sufficient.
Thanks

[rom1v]

Do you connect via SSH using these commands: https://github.com/Genymobile/scrcpy/blob/master/README.md#ssh-tunnel ?

[R3nd0gg]

Yes, I have tried that method but without luck - it wont work at all. I am sucesfully connecte to remote ADB server (when I type adb devices on my machine, device comes up as connected) but when I run scrcpy it does this:

INFO: scrcpy 1.12.1 <https://github.com/Genymobile/scrcpy>
D:\Users\Adam\Desktop\SCRCPY\scrcpy-server: 1 file pushed. 0.1 MB/s (26202 bytes in 0.332s)
adb.exe: error: more than one device and emulator
ERROR: "adb reverse" returned with value 1
WARN: 'adb reverse' failed, fallback to 'adb forward'
adb.exe: error: cannot bind listener: Address already in use
ERROR: "adb forward" returned with value 1
Press any key to continue...

ADB devices output:

List of devices attached
localhost:55555 device

There is only one device and scrcpy returns with error.

[rom1v]

localhost:55555 device

If your remote device is connected over USB, do not use adb connect at all. Just kill the local adb server, establish the tunnel, and execute adb devices.

[R3nd0gg]

The way it is connected is like this:
Android device is behind NAT, my app after receiving command does remote port forwarding to my server on port 55555. After that, I just connect through SSH to that server and execute commands to local/remote port forward.

[rom1v]

Android device is behind NAT

It is connected wirelessly? Not over USB on a remote computer?

[R3nd0gg]

Yes, wirelessly. Basically:
Device <-- (Wireless) --> Server <-- (Wireless) --> MyPC

[rom1v]

adb.exe: error: cannot bind listener: Address already in use

Try another port? scrcpy --port 28000

[R3nd0gg]

Ok so I changed:

ssh -CN -L5037:localhost:5037 -R28000:localhost:28000 host@server

and used

scrcpy --port 28000

and output is:

ERROR: "adb reverse" returned with value 1
WARN: 'adb reverse' failed, fallback to 'adb forward'
connect: No error
connect: No error
connect: No error
connect: No error
connect: No error
connect: No error
connect: No error
connect: No error
connect: No error

[R3nd0gg]

Am I doing something wrong or there may be bug? If it helps I can provide you access to that server and device to test it out.

[rom1v]

Oh but over wifi you will use adb-forward instead of adb-reverse, so you need local port forwarding:

ssh -CN -L5037:localhost:5037 -L27183:localhost:27183 your_remote_computer

(-L instead of -R)

[R3nd0gg]

Yeah it works like that, thanks. It is better but still laggy, but thats limitation of TCP I guess. Do you think it would help to reimplement communication to UDP?

[LBensman]

@R3nd0gg, It's generally not just a matter of switching socket type to be UDP and figuring out how to chop up the H.264 NAL video packet to fit into UDP packet that have limited size (though RFC 6184 would already be a pretty good template for it). It would require some cooperation between consumer of the video stream (i.e. the viewer) and the producer of the stream (i.e. scrcpy server portion that runs on the device).
Namely, such cooperation would require viewer to detect sufficient packet loss (or delay of packet arrival beyond their useful consumption time window), ability to request that server running on the device produce a NAL unit that is a full intra frame refresh (aka key frame) or otherwise indicate loss of picture / bandwidth issues.
And that's just the simple case (request full frame, then keep going as if everything is back to rosy world). Reality is, however, that loss of packet(s) might've been a rare unfortunate event, but more likely due to congestion that is happening on a network segment that is causing packet loss, and thus keeping the stream as-is with just a key frame refresh is not likely to alleviate the issue. This in turn means (if you want to improve overall experience of the stream) you need a mechanism to detect available bandwidth and throttle encoder up or down to match available bandwidth. And then also possibly implement lost packet retransmits (so that you can try to minimize missing frames). Etc., etc., etc..
You can implement it "partially" - i.e. do key frame refresh but not other parts, but the biggest hurdle here is that at minimum you now you need to architect a two-way communication between viewer and scrcpy server to be able to control encoding process that's happening on the device. Not that it cannot be done, but just that it entails (requires?) more than just switching from TCP to UDP. Just switching from TCP to UDP by itself won't really do much.

[BigDogBoy]

Also, you know, it could be related to the WiFi itself; you might be trying this; with your phone and/or the path [the signal is traveling] between your phone, and the server; in a relative WiFi dead zone; try analyzing the wireless signal, where you set your phone down; it could also (remotely) possibly be a speed bottleneck related to your ISP, try creating a Wi-Fi hotspot, that doesn't use internet, and see if either of those solutions help.