Scrcpy: Get the video stream only from the android

[likezjuisee]

After I have executed the command adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server 0 8000000 true, how can I get the video stream from the phone? Maybe the reversed socket "scrcpy"?
My purpose is get the video stream and transform it to rtmp format, display it on the website.

[rom1v]

Yes, you can connect to the socket, and mimic the protocol used between the client and the server:

• read the device name (64 bytes, containing a NUL-terminated string)
• read the initial device width (2 bytes unsigned big endian)
• read the initial device height (2 bytes unsigned big endian)

(see device.c)

After these 68 bytes, you receive the raw H.264 stream (decoder.c).

[likezjuisee]

I have tried the method you mentioned, but come with some problems:
first: adb forward tcp:8080 localabstract:scrcpy
and then, run the python script:

#coding=utf-8
import socket
import struct

port = 8080

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('127.0.0.1', 8080))
deviceName = sock.recv(64)
print len(deviceName), deviceName[-1]
print str(deviceName), 1111111
width = sock.recv(2)
print struct.unpack("H", width)
height = sock.recv(2)
print struct.unpack("H", height)
sock.close()

1. sometimes, it can receive the right device name, but sometimes can only receive 1 byte, and width and height are also wrong.
2. when the socket been closed, the jar also exited

[rom1v]

sometimes, it can receive the right device name

It receives it exactly once. If you retry after the server is shutdown, your script will just print 0 (len(deviceName)) and fail.

but sometimes can only receive 1 byte, and width and height are also wrong.

I guess sock.recv(64) fails (with EOF or exception?) because it can connect to the tunnel but not to the server (which is not started), so the values you use afterwards are probably garbage.

when the socket been closed, the jar also exited

Yes, that's expected (when you close the scrcpy client, you want the server to disappear). Just start the server at the beginning of your python script.

[likezjuisee]

Thanks for you answer, it helps a lot!

[likezjuisee]

This is a very nice project!

[rom1v]

I forgot to mention a very important detail: in adb forward mode, scrcpy initially writes a dummy byte (0). See this commit and its message.

[likezjuisee]

Thank you for your mention, I got what the first byte means.

[likezjuisee]

One more question:
I am not familiar with the media codec, so how can I transform the h264 stream to rtmp?

[Allong12]

I got a working example based off what you started, @likezjuisee .

It connects to the SCRCPY server as you've done, then creates a SubProcess Pipe to FFplay from the FFmpeg suite and pipes all video data received directly into there. See the Gist linked below including Bash and Windows helper scripts to launch the server.

https://gist.github.com/Allong12/a752decf49e6c789c2425e35028137a5

Next step is piping into FFmpeg and recieving the decoded image into a Python NumPy array or the like

[Allong12]

In fact I kinda just went ahead and started my own project on all of this. I've got FFmpeg processing all of the network frames, and then Python converting them into a friendly Numpy Array

See what you think:
https://github.com/Allong12/py-scrcpy

[likezjuisee]

Hi, with the newest scrcpy-server.jar, I can't received the h264 data as before, is there something changed?
Why accept twice?

            try {
                Ln.d("waiting client connect...");
                videoSocket = localServerSocket.accept();
                Ln.d("client connected and write 0");
                // send one byte so the client may read() to detect a connection error
                videoSocket.getOutputStream().write(0);
                try {
                    controlSocket = localServerSocket.accept();
                } catch (IOException | RuntimeException e) {
                    videoSocket.close();
                    throw e;
                }

@rom1v

[likezjuisee]

I got a working example based off what you started, @likezjuisee .

It connects to the SCRCPY server as you've done, then creates a SubProcess Pipe to FFplay from the FFmpeg suite and pipes all video data received directly into there. See the Gist linked below including Bash and Windows helper scripts to launch the server.

https://gist.github.com/Allong12/a752decf49e6c789c2425e35028137a5

Next step is piping into FFmpeg and recieving the decoded image into a Python NumPy array or the like

Nice, I guess you may use the screen data as th ML input data?

[rom1v]

with the newest scrcpy-server.jar, I can't received the h264 data as before, is there something changed?

Yes, that changes all the time. https://github.com/Genymobile/scrcpy/issues/673#issuecomment-516360374

Why accept twice?

https://github.com/Genymobile/scrcpy/commit/ec71a3f66ab48c2fdd1728753acc09edbd4db570

[likezjuisee]

with the newest scrcpy-server.jar, I can't received the h264 data as before, is there something changed?

Yes, that changes all the time. #673 (comment)

Why accept twice?

ec71a3f

I have read the code, videoSocket and controlSocket must are connected both, I will use the videoSocket only.
Thanks.

[razumeiko]

For those who is looking for the answer:
You can decode video using FFMPEG as separate process, write to stdin and read from stdout, but this method introduce delay 1-2 seconds and was not ideal for me. I spend some time trying to optimize it and found pyAV bindings to ffmpeg directly.
I created class that you can use to get frames from scrcpy server almost without delay.
https://github.com/razumeiko/py-android-viewer

Also there is control mixin class that can send swipe command to scrcpy server. It can be extended.

[qaisbayabani]

a single file solution in c#
https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client