Scrcpy: Is it possible to redirect the stream to another application ?

[Darkroll76]

Hi,

Currently, _scrpcy_ opens independent windows to render video stream but I'm trying to integrate it inside an application. Is there any way to do it? By redirecting the stream?

It could be a Unity App or Java App or C# UWP App but I would like to be able to share android-based VR headset screen (Oculus Go, Quest, PICO, etc) inside a desktop application.

Thanks for your time,
Killian

[iiasceri]

https://github.com/barry-ran/QtScrcpy

[Darkroll76]

https://github.com/barry-ran/QtScrcpy

Thanks, I will try if I could find what I need with this project. For now, it just seems to offer the same features.

[rom1v]

Currently, scrpcy opens independent windows to render video stream but I'm trying to integrate it inside an application. Is there any way to do it?

The proper way would be to split scrcpy in two parts: libscrcpy and the UI (https://github.com/Genymobile/scrcpy/issues/707#issuecomment-520941669). But this is not done.

Another way could be to use only the .jar on your side, and write your client which mimic the protocol used between the client and the server (but this protocol should be considered private, i.e. it may change at any time).

Alternatively, you could fork current master and replace some parts…

[LBensman]

That's pretty much what I did - just take the Java portion that already sends back H.264 stream (with initial custom header), and just handle H.264 stream inside your desktop application (or where ever you want to consume it). Basically reuse the server jar, but dump viewer in favor of your own stream handling and visualization.

On a separate note, @rom1v , did you notice from screenshots above that QtScrcpy seems to be supporting audio? I know you tried USBAudio (I'm looking into it now myself), but quick look (really, probably not even 2 min spent on it) at QtScrcpy source, I couldn't see much for audio handling. Have you looked into it and see if it something more solid than USBAudio?

[rom1v]

did you notice from screenshots above that QtScrcpy seems to be supporting audio?

I don't think so. They're just shortcut for volume up/down on the device. Also see #612.

Have you looked into it and see if it something more solid than USBAudio?

The real solution will be the Playback Capture API introduced in Android Q. See #933.

[LBensman]

Yeah, I know about Playback Capture API (I've been all over the threads, esp #14 one) - unfortunately it is insufficient for what I need...

[rom1v]

unfortunately it is insufficient for what I need...

Why?

[LBensman]

I mean it would "help somewhat" in that it will allow for a small sliver of audio to be captured, but it is a co-operative solution between apps and the capturing app (the app has to explicitly allow for its audio to be captured), and, as far as I understand, would only work with a small subset of apps and none of the system sounds. It seems PCAPI was designed for a very very narrow use case like streaming of game playing, rather than general-purpose API. In other words, in scrcpy analogy, only being able to capture screen when a co-operating game is being played, but otherwise getting a blank screen in all other cases - much more limited and thus much less useful.

[rom1v]

I did not investigated yet, but by calling private API with shell permissions, it is maybe possible to capture the audio from any app (I'm not sure).

For example, the MediaProjection API requires to get an authorization from the user, but scrcpy does not.

[Darkroll76]

Hi @LBensman ,

That's pretty much what I did - just take the Java portion that already sends back H.264 stream (with initial custom header), and just handle H.264 stream inside your desktop application (or where ever you want to consume it). Basically reuse the server jar, but dump viewer in favor of your own stream handling and visualization.

Could you help me with that part? Should it work if I'm just launching server .jar? Because, I guess, it requires to modify the _scrcpy_ code, but It could be very helpful if you help me where.

I want to display the video stream in a Unity app, I can handle H.264 stream from the "receiver" side, but I'm just stuck trying to find a way to modify _scrcpy_ to send back H264 to it.

Thanks,

[rom1v]

A proper solution could be to implement an option --serve, which would wait for a client to connect before starting the mirroring, and it would forward the video stream.

For example, it would be used like this:

scrcpy --serve=tcp:localhost:1234

That way, once implemented, anyone could receive the raw video stream.

It's not a priority for me, but if you work on something that requires it, you could implement it that way and submit a pull request. 😉

[Darkroll76]

Yes, this solution could be great and easy to use. Thanks for the suggestion!

I can for sure understand that isn't a priority for you ;).

Since I'm not an expert in java development, I'm not sure I will be able to do it properly but I'm still going to start investigating the code and try (If anyone who is reading this post wants to join ;) ).

Have you any tips? I have read the developer readme but it would be helpful to know which IDE (for example) should be used to edit your project. I'm using Android Studio for the server-side but which IDE are you using for the client development and to build scrcpy.exe?

Thanks for your time,

[rom1v]

Since I'm not an expert in java development

This feature has to be implemented on the client-side, which is in C :wink:

Basically, when you receive a packet, it is decoded + recorded (if necessary):
https://github.com/Genymobile/scrcpy/blob/39356602ed472cc3f533e36ae04a110b247c29e0/app/src/stream.c#L81-L97

We can, in addition, write it to a socket.

But in fact, it could even be written earlier (without even parsing it):

https://github.com/Genymobile/scrcpy/blob/39356602ed472cc3f533e36ae04a110b247c29e0/app/src/stream.c#L229

I have read the developer readme but it would be helpful to know which IDE (for example) should be used to edit your project.

Whatever you want. I use vi.

[Darkroll76]

Hi,

I don't know if I should create another topic about creating an --serve option to forward video stream. I just want to be sure I'm doing the right thing since I only started to manage and understand the scrcpy client source code.

First in scrcpy/app/scr/cli.c, I added the option {"serve", required_argument, NULL, OPT_SERVE}, then into the switch I added

case OPT_SERVE:
                opts->serve = optarg;
                break;

Then in scrcpy/app/scr/scrcpy.c : scrcpy(const struct scrcpy_options *options)

if (!stream_start(&stream)) {
        goto end;
    }
    stream_started = true;

if (!serve_init(options->serve)){
        goto end;
    }

I would like to split the args 'tcp:localhost:1234' and use net_connect function from net.c to create a socket. Then the idea is to send the packet through this socket each time process_frame is called. But in code, I don't know the best way I should do it. Create another class? Or just create a function serve_packet(AVPacket *packet) in stream.c ?

Thanks again for your time,
Killian

[qaisbayabani]

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client is a demo to show a self made application for scrcpy output

[Darkroll76]

Hi @qaisbayabani !

Thanks for the demo ! I tried but I got a java.io.IOException: Connection refused
Any idea ?

[qaisbayabani]

i have made 100% working clients for both c# and java is available on request if the party needs it so deperetly a one page 100 line code.

[qaisbayabani]

mr darkroll u r using an emulator?
are you connecting your phoen directly via USB?
and confirm u are using scrcpy 1.12.1 version?

adb commands must be reconfigured according to your own environment other things will work fine. i have deleted this demo repository because i have developed a flawless full alpha version now which is working 100% fine soon i will load it to repository

if you could get me screenshots of output window than may i could understand that will be a thanks for your response.

[overvox]

i have made 100% working clients for both c# and java is available on request if the party needs it so deperetly a one page 100 line code.

Hey, any chance you could share the C# Code or are you still working on a functioning version?

[Darkroll76]

@qaisbayabani I'm interested to see it too.

[qaisbayabani]

i have made 100% working clients for both c# and java is available on request if the party needs it so deperetly a one page 100 line code.

Hey, any chance you could share the C# Code or are you still working on a functioning version?

the version mr.darkroll was downloaded is also a working one.

[qaisbayabani]

mr darkroll u r using an emulator?
are you connecting your phoen directly via USB?
and confirm u are using scrcpy 1.12.1 version?

adb commands must be reconfigured according to your own environment other things will work fine. i have deleted this demo repository because i have developed a flawless full alpha version now which is working 100% fine soon i will load it to repository

if you could get me screenshots of output window than may i could understand that will be a thanks for your response.

is it working or still error

[qaisbayabani]

mr darkroll u r using an emulator?
are you connecting your phoen directly via USB?
and confirm u are using scrcpy 1.12.1 version?
adb commands must be reconfigured according to your own environment other things will work fine. i have deleted this demo repository because i have developed a flawless full alpha version now which is working 100% fine soon i will load it to repository
if you could get me screenshots of output window than may i could understand that will be a thanks for your response.

is it working or still error

as i guess u were using an emulator u should use a phone and tell the room is it working at ur side

[Darkroll76]

@qaisbayabani I'm using it with a VR Headset based on Android. I still have a Connection refused with the first demo you posted.
It's not directly connected by USB but I added the adb tcpip 5555 and adb connect command in your script.
I was using 1.12.1 then I changed it by the new one (1.13).

My version allows forwarding the packet received on another socket. In your demo, you directly push the scrcpy-server and connect to it. It is very helpful to understand how you integrated it into a C# solution.

[qaisbayabani]

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client
it covers.
multithreading
a single file program.cs converting smooth capture from scrcpy-server to c# client
adb commands using c#
server port opening
listening to stream and forwarding the stream to other port.
opencv capture from network stream and disply H264 stream from android screen capture.
cross plateform stuff.
enjoy

[qaisbayabani]

Mr. Overvox

Hey, any chance you could share the C# Code or are you still working on a functioning version?

its here

[Darkroll76]

@qaisbayabani I don't have the connection refused any more. But It's still not working for me. I always have a gray screen with "Empty" outputs.
I'm using it on Windows with an Android VR Headset.

[qaisbayabani]

@qaisbayabani I don't have the connection refused any more. But It's still not working for me. I always have a gray screen with "Empty" outputs.
I'm using it on Windows with an Android VR Headset.
i have prepared it and tested with android mobile phone and working perfect. may be its needs fine tunning for VR Headset. try to attach VR with USB Cable may be a Network ports speed issue. try it tell me if its working on android phone as its working fine here on 50 sets i have tried form andorid 6.0 to 9.0.

[Darkroll76]

@qaisbayabani
i have prepared it and tested with android mobile phone and working perfect. may be its needs fine tunning for VR Headset. try to attach VR with USB Cable may be a Network ports speed issue. try it tell me if its working on android phone as its working fine here on 50 sets i have tried form andorid 6.0 to 9.0.

I tried using USB Cable on VR Headset (Android 8) and my phone too (Android 10). I got the "Third is Gneeal" error and receiving "Empty" outputs each time.

[hgedik]

@qaisbayabani hi,

I have Android 9 (samsung galaxy s8+). I tried over wifi and usb cable. And same result. just "empty" outputs.

[Darkroll76]

Hi @hgedik,

I manage to make it works for me. I change from http://127.0.0.1:27184 to tcp://localhost:27184. Then I changed Cv2.WaitKey to wait only 1 sec instead of sleepTime because the framerate was slow for me with it.

[qaisbayabani]

@qaisbayabani hi,

I have Android 9 (samsung galaxy s8+). I tried over wifi and usb cable. And same result. just "empty" outputs.

I guess that u just have to just start the screen of mobile (or play some thing youtube etc) before running the debug. and as drakol may u have to adjust Cv2.Waitkey(1). if u change the waitkey than u may have not even to start the screen before debug. waiting from ur side. :)

[qaisbayabani]

Hi @hgedik,

I manage to make it works for me. I change from http://127.0.0.1:27184 to tcp://localhost:27184. Then I changed Cv2.WaitKey to wait only 1 sec instead of sleepTime because the framerate was slow for me with it.

its good to listen that its working for u. u can see its just taking few lines of codes and needs just few more for universal plug and play. Cv2.WaitKey(1) is the best and ideal for all.

[qaisbayabani]

I think the basic question is solved that yes it is possible to redirect the stream to another application.

• you can made that another application which can read that stream.
  the challenge was for me that that application would be a one page simple code easy to understand.
  unlike jumbled classes as used by scrcpy org.
  now its a challenge to write a simple server that can serve the stream from android phone to this application.

[rom1v]

Hi @qaisbayabani,

I don't really know C#, but just a few remarks.

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client/blob/b356aae19dd1e0a348ffd75866f951f43c5f3504/Program.cs#L203

I don't know how BufferedStream behaves, but in theory you want to consume the video stream as soon as it is available. I guess it could delay frames, especially "small" frames (when the changes compared to the previous frame are minimal).

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client/blob/b356aae19dd1e0a348ffd75866f951f43c5f3504/Program.cs#L325

The framerate of the video is very variable: a new frame is produces whenever the screen content changes. If it does not change for 10 seconds, there are no frames. If it changes 90 times per second, then the framerate is 90fps. The WaitKey will also delay the frame.

[hgedik]

@qaisbayabani i cant fix this. :) maybe i missed something. I set to Cv2.WaitKey(1) but same.

and video: https://www.screencast.com/t/bPdjdgkD9

[qaisbayabani]

@qaisbayabani i cant fix this. :) maybe i missed something. I set to Cv2.WaitKey(1) but same.

and video: https://www.screencast.com/t/bPdjdgkD
this error indicates that your opencv is not working well

you have to just add nuget package opencvsharp4 from nuget manager in visual studio.

and try with http://localhost:27184 why i see tcp://localhost:27184 in your outputimage
its error from your opencv if your opencv repond well then this "third is geneal " will not appear.
it means that receiver application is not receiving the stream from server and denied the connection if you want me the detail how to instal opencv in visual studio then i can help out more.

[qaisbayabani]

Hi @qaisbayabani,

I don't really know C#, but just a few remarks.

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client/blob/b356aae19dd1e0a348ffd75866f951f43c5f3504/Program.cs#L203

I don't know how BufferedStream behaves, but in theory you want to consume the video stream as soon as it is available. I guess it could delay frames, especially "small" frames (when the changes compared to the previous frame are minimal).

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client/blob/b356aae19dd1e0a348ffd75866f951f43c5f3504/Program.cs#L325

The framerate of the video is very variable: a new frame is produces whenever the screen content changes. If it does not change for 10 seconds, there are no frames. If it changes 90 times per second, then the framerate is 90fps. The WaitKey will also delay the frame.

thanks for remarks but i strongly rely on "WYSIWYG" WHAT YOU SEE IS WHAT YOU GET.
if the frame rate is higher than its good for game stream. and the Waitkey is resolved as per previous comments that Cv2.WaitKey(1) is the best perfroming at most devices.

i have checked it with stop watch app running in mobile that there is no delay at all.

thaks for your input and i want your point to be elebortated more that i can learn more from your input.

[Darkroll76]

@qaisbayabani I don't get why http://localhost:27184 doesn't work for me (Empty outputs) but tcp://localhost:27184 did. I'm on Windows.

Also, if you want to upgrade using scrcpy-server.jar 1.13 instead 1.12.1, here is my command with the new arguments :

ps.Arguments = "shell " + "CLASSPATH=/data/local/tmp/scrcpy-server.jar " +
         "app_process " + "/ " + "com.genymobile.scrcpy.Server " +
        "1.13 " + "1024 " + "2000000 " + "0 " + "1 " + "false " + "563:1000:800:470 " + "true " + "true" + " 0"; 

[qaisbayabani]

@qaisbayabani I don't get why http://localhost:27184 doesn't work for me (Empty outputs) but tcp://localhost:27184 did. I'm on Windows.

Also, if you want to upgrade using scrcpy-server.jar 1.13 instead 1.12.1, here is my command with the new arguments :

ps.Arguments = "shell " + "CLASSPATH=/data/local/tmp/scrcpy-server.jar " +
         "app_process " + "/ " + "com.genymobile.scrcpy.Server " +
        "1.13 " + "1024 " + "2000000 " + "0 " + "1 " + "false " + "563:1000:800:470 " + "true " + "true" + " 0"; 

you may have to instal IIS for windows and start it
for http://localhost:27184
but if u are getting it via tcp than its fine. cuz u may have a linux based server installed on pc.

u can also use ip address assigned by your local wifi router to your pc
as mine http://192.168.0.3:27184

[qaisbayabani]

there are thousands of pages written on web about ffmpeg conversion, listening H264 stream and conversion, encoders decoders bla bla.... i have just here to cover as written in my repository readme

it covers.

multithreading

a single file program.cs converting smooth capture from scrcpy-server to c# client

adb commands using c#

server port opening

reading server received data and forwarding the stream to other port.

opencv capture from network stream and disply H264 stream from android screen capture. cross plateform stuff. enjoy

only for teaching purpose.

and a real world working example of serving that purposes.

[qaisbayabani]

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client will be closed in few days

[ahmed-shariff]

@qaisbayabani Thanks for sharing the script man. I am trying to run this and I am getting a similar issue as the other. When throw the exception that results in "Third is Gneeal" I get the following:

.
.
.
Connected!========>2
Third is Gneeal 

Unhandled Exception: System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at Program.ForFun.Fun() in F:\temp\scrcpytest\Program.cs:line 251
   at Program.<>c__DisplayClass0_0.<Main>b__0() in F:\temp\scrcpytest\Program.cs:line 41
   at System.Threading.Thread.ThreadMain_ThreadStart()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

I tried what others had suggested and am still getting the same issue.

@Darkroll76 would you be able to share your version of the working script?

[ahmed-shariff]

I recreated the same script as @qaisbayabani in python, which seems to work. The python script:

import cv2
import socket
from multiprocessing import Process
from tqdm import tqdm

def main():
    s_input = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s_input.bind(('127.0.0.1', 27183))
    s_input.listen(1)
    conn, addr = s_input.accept()

    print("Connection details: ", conn, addr)
    print("Device: ", conn.recv(64).decode('utf-8'))
    print("Width? ", int.from_bytes(conn.recv(2), "big"))
    print("Height? ", int.from_bytes(conn.recv(2), "big"))

    s_output = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s_output.bind(('127.0.0.1', 27184))
    s_output.listen(1)

    p = Process(target=client)
    p.start()

    out_conn, addr = s_output.accept()

    print("\n", "Connection details: ", out_conn, addr)
    while True:
        out_conn.send(conn.recv(1))
    p.join()


def client():
    cap = cv2.VideoCapture("tcp://127.0.0.1:27184")
    print(cap)

    for _ in tqdm(range(1000)):
        r, img = cap.read()
        if not r:
            continue
        cv2.imshow("", img)
        cv2.waitKey(20)


if __name__ == "__main__":
    main()
    # client()

I tried removing the ForFun3 line https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client/blob/91717b1faefec31854111919329431004396e0e3/Program.cs#L194 from the csharp script and try connect to the tcp server using only the client function in the python script, and I still get the same issue as I mentioned above.

[qaisbayabani]

@Darkroll76 would you be able to share your version of the working script?

correct question