Scrcpy: Can it support another screenshot stream on android - secondary screen

[chaooe]

1. Normaly we can capture screen 0 by 'adb shell screencap -p -d 0 /sdcard/sc.png', so if we have two or more display, we can use 'adb shell screencap -p -d ${displayid} /sdcard/sc.png', so can we have this supported?

[rom1v]

IIUC, you would like to mirror a secondary screen?

It would be great.

Something should be configured to request another screen in this code:

https://github.com/Genymobile/scrcpy/blob/v1.5/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L63-L70

But in SurfaceControl.java (in AOSP), I didn't find how to request another screen.

If someone have an idea…

EDIT: also see option -d in screencap.cpp.

[chaooe]

Yes, i want to mirror a secondary screen. I am a QA to test Android Device, and we have some future running on another screen.. we can just use screenshot to capture another screen. It is slowly and unstable.

[webian]

I'm another one that will find this feature precious to command Samsung Dex from my linux box without the need to switch mouse/keyboard/monitor.

I'm not an android dev but I found some little things that I hope can help:

1. if I run this apk https://github.com/mportuesisf/ScreenInfo starting from the smartphone display it returns me the smartphone display resolution, if I run it from the Dex monitor it gives me the resolution of the monitor. So, maybe in the code of ScreenInfo there are some hints on how to achieve this.
2. AnyDesk can reproduce Dex screen but only if started from the Dex monitor, if started from the smartphone monitor it reproduce the smartphone's one. Also, mouse click are always made on the smartphone even if started from the Dex monitor.

[webian]

Is there maybe a way to make Android to think that the scrcpy server is started on the external (Dex) monitor?

[webian]

Other info that maybe can help (see "Detect Samsung DeX"):
https://developer.samsung.com/samsung-dex/modify-optimizing

[tchangeme]

请问，这个怎么实现呢，手机上有另外一个屏幕的时候

[nakele]

@rom1v I'm not a developer but if I "adb" wirelessly to my phone I get information about other displays, might be useful to this. See screenshots below:

[nakele]

@rom1v btw, if the server was wrapped in an apk, and the apk was launched in Samsung Dex mode, it might mirror the second display.

[chaooe]

@rom1v I found minitouch have soultion to touch on second screen. before it use, we need to setup minitouch server on device.
minitouch -d /dev/input/event0 /dev/input/event0 is the first screen id. and minitouch -d /dev/input/event1 is the second display id
Does this scrcpy use minitouch to touch the screen?

[rom1v]

No, it does not use minitouch. I guess it requires root access on the device to write to /dev/input/event*?

[chaooe]

No, the /dev/input/event is the display id. I don't know where to configure the /dev/input/event*

1. When i copy minitouch and add execution permission to android folder. /tmp/data/minitouch.
2. /tmp/data/minitouch , it will show the display id.
3. /tmp/data/minitouch -d /dev/input/event0 to start the minitouch server

[rom1v]

Could you test #1177 please?

[chaooe]

Ok. I will have a try for this. and reply to ASAP.

[chaooe]

For https://github.com/Genymobile/scrcpy/pull/1177 , it should be ok to mirror the display 1 screen on ubuntu 18.04. But i can't do some touch or text commands.
Add adb shell dumpsys display for reference.

[peanutwolf]

@rom1v @chaooe Thank you for testing. I think i've missed the touch functionality for the second screen cause my second screen is not touchable. I'll investigate how it could be supported and will make commit to my PR.

[chaooe]

Thanks @peanutwolf , And i have 3 display for this. The default id 0 and display id 1 can be touched. but display id 2 is only for view. just some information, i am not sure if this is helpful for you. And thanks again

[webian]

Thanks @peanutwolf !
I tested #1177 too and it's working except for clicks.
I can see the Dex screen (secondary monitor) but the clicks that I make on scrcpy window are sent to the smartphone instead of Dex, so I guess that a small step is missing.

[peanutwolf]

@chaooe @webian what android API does your devices have? ($ adb shell getprop ro.build.version.sdk | tr -d '\r')
It seems that input events for multiple displays can't be supported for android 9 and lower.
Or maybe you have custom InputFlinger patched for your devices?

[webian]

Mine is a Samsung Galaxy S10e with Android 10:
$ adb shell getprop ro.build.version.sdk | tr -d '\r'
29

No custom InputFlinger as far as I know.

[webian]

Please, also note n.2 of my https://github.com/Genymobile/scrcpy/issues/397#issuecomment-482964379 I had the same problem about clicking with AnyDesk.

[chaooe]

@peanutwolf My version is 27
$ adb shell getprop ro.build.version.sdk | tr -d '\r' 27

[chaooe]

Another things for display id 1 which can be input. for my automation test. we found we can use minitouch for touch the second screen.
setup minitouch server on device.
minitouch -d /dev/input/event0
/dev/input/event0 is the first screen id. and minitouch -d /dev/input/event1is the second display id

[chaooe]

Add more information from my devices
$ adb shell msm8996_gvmq:/ # cat /sys/class/input/ event0/ event1/ input0/ input1/ mice/

getevent -l
could not get driver version for /dev/input/mice, Not a typewriter
add device 1: /dev/input/event1
name: "QVM virtio-input (touchscreen)"
add device 2: /dev/input/event0
name: "QVM virtio-input (touchscreen)"

And i found this define on this page
http://newandroidbook.com/Book/Input.html?r

[peanutwolf]

Hi @webian. I've made a small extension for my pull request. I've added support for input events for secondary display. But it could work only for android 10 and higher. Can you check me theory and test https://github.com/Genymobile/scrcpy/pull/1177 againg
Hi @chaooe. I've examined mini-touch util. Unfortunately that solution is not suitable for every android device. Some of them require root privileges: https://www.pocketmagic.net/programmatically-injecting-events-on-android-part-2/

Without root access, this fails, with “operation not permitted”

[webian]

Hi @peanutwolf I tested #1177 and mouse interactions work perfectly now, good job! Thank you!

Just side note FYI...
I had to use option "-d 2" to display second monitor (Dex) not "-d 1" that instead returns this error:

[server] ERROR: Exception on thread Thread[main,5,main]
java.lang.AssertionError: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Class java.lang.Object.getClass()' on a null object reference
        at com.genymobile.scrcpy.wrappers.DisplayManager.getDisplayInfo(DisplayManager.java:26)
        at com.genymobile.scrcpy.Device.computeScreenInfo(Device.java:52)
        at com.genymobile.scrcpy.Device.<init>(Device.java:31)
        at com.genymobile.scrcpy.Server.scrcpy(Server.java:19)
        at com.genymobile.scrcpy.Server.main(Server.java:166)
        at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
        at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:339)
Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Class java.lang.Object.getClass()' on a null object reference
        at com.genymobile.scrcpy.wrappers.DisplayManager.getDisplayInfo(DisplayManager.java:18)
        ... 6 more

[webian]

It could also be useful to trigger Samsung Dex or similar desktop modes without have to physically attach the monitor. Do you know if it is possible?

[chaooe]

@peanutwolf Our device have root permission , Can you help to add the minitouch solution for me to have a try? I don't know how to add in this. Thanks very much.
Next monday i will ask developer from our SW team how they deal with the event even though we are using Android 8.

[rom1v]

Could you test the new --display option on dev (the development branch), please?

[rom1v]

Here is a win64 release for the current dev branch:

• scrcpy-server
• scrcpy.exe

SHA256

09489ac673ed3b0bb7aba7e0858c43137a7480d1a2d4f8a51f698acbc49be735  scrcpy-server
c9be0cd73651a376932ca0e4ca1c07275aa776a5e5520b5b184fcff943affdd5  scrcpy.exe

(replace these two files in the v1.12.1 release)

[chaooe]

@rom1v Ok. I will have a try and response ASAP

[chaooe]

@rom1v It works nice on windows 10. And have an message when using display 1 with android API 27

[rom1v]

OK so you can mirror but not control if all conditions are not met. That's expected :+1:

What happens if you click?

[chaooe]

Nothing happened on both display 0 and display 1

[chaooe]

Is that possible to add minitouch solution to support this?

[webian]

Hi @rom1v, I tested dev replacing the 2 files and launching with "--display 2" and it works both display and input.
Windows 10 LTSC 1809 + Samsung Galaxy S10e SM-G970F Android 10

[rom1v]

@webian Cool :tada: Thank you for your test.

(How do you get multiple displays in practice with your phone?)

[webian]

@webian Cool 🎉 Thank you for your test.

You don't need to thank me... I thank you a lot for the amazing scrcpy!

(How do you get multiple displays in practice with your phone?)

see https://github.com/Genymobile/scrcpy/issues/898#issuecomment-610996513
I add that in practice I use an USB to HDMI adapter and connect it to a monitor with an HDMI cable. This way I can mirror the phone screen. To start Dex on the external screen I also have to power the adapter with an external USB charger.

[rom1v]

I guess this issue can be closed with the new --display option implemented on dev branch.

[rom1v]

Released in v1.13.

[tchangeme]

The second screen can be displayed normally, but the mouse and keyboard cannot be used。

[tchangeme]

In addition, the screen cannot be fully displayed

[rom1v]

but the mouse and keyboard cannot be used

That's expected before Android 10.
https://github.com/Genymobile/scrcpy/releases/tag/v1.13

In addition, the screen cannot be fully displayed

What do you mean?

[jvisser99]

Greetings,

Thanks for the great program, I use it daily.

I can't get DeX to work. I tried the --display option, and for anything other than 0 i get an error of this form:

$ scrcpy --display 1
INFO: scrcpy 1.13 <https://github.com/Genymobile/scrcpy>
/usr/local/share/scrcpy/scrcpy-server: 1...pushed. 3.1 MB/s (27694 bytes in 0.009s)
[server] INFO: Device: samsung SM-G960U1 (Android 10)
[server] ERROR: Exception on thread Thread[main,5,main]
com.genymobile.scrcpy.InvalidDisplayIdException: There is no display having id 1
    at com.genymobile.scrcpy.Device.<init>(Device.java:46)
    at com.genymobile.scrcpy.Server.scrcpy(Server.java:20)
    at com.genymobile.scrcpy.Server.main(Server.java:177)
    at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
    at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:339)
[server] ERROR: Try to use one of the available display ids:
[server] ERROR:     scrcpy --display 0

I'm running Red Hat Enterprise Linux Workstation 7.6, Samsung Galaxy 9, Android 10. Connected via a Lenovo Thunderbolt doc to a Lenovo X1 Carbon Gen. 6, 2 displayport-connected monitors.

Any help appreciated. Happy to test.

jv

[rom1v]

Is DeX running when you execute this command?

[jvisser99]

Thanks for the quick reply.
I don't know. I get a notification that says "you can use Sasung DeX on your computer. I searched for DeX, and found four apps, DeX for PC, Samsung DeX, Samsung DeX home, and Samsung DeX System UI, all installed, but no indication that they are running.

[rom1v]

AFAIK, DeX can only be enabled when you plug your device to some HDMI monitor.

[jvisser99]

AFAIK, DeX can only be enabled when you plug your device to some HDMI monitor.

If you do this, how do you run scrcpy? And, scrcpy 1.13 supports DeX screen mirroring.

[nakele]

@jvisser99 if u want to run Dex on PC you need to install Dex appliaction on your pc as well, plug the cable to phone, wait to say it's connected and you should get a window dex on your pc monitor. No needed for for anythingelse. HoWEVER scrcpy could be used to mirror dex on devices that don't support Dex on PC like the S8/Note8 line + S9/Note9 on Android 9.

First you need to trigger Dex via HDMI cable or HDMI dummy plug (my case), then use scrcpy via WiFi (because your port will be used for the HDMI)
In your case you don't need scrcpy, the samsung app "Dex on Window", also available on Mac.

[rom1v]

scrcpy 1.13 supports DeX screen mirroring.

It supports mirroring a secondary display.

On devices supporting DeX, a secondary display is created when DeX is enabled. And DeX is enabled when you plug the device via HDMI.

[jvisser99]

@jvisser99 if u want to run Dex on PC you need to install Dex appliaction on your pc as well, plug the cable to phone, wait to say it's connected and you should get a window dex on your pc monitor. No needed for for anythingelse. HoWEVER scrcpy could be used to mirror dex on devices that don't support Dex on PC like the S8/Note8 line + S9/Note9 on Android 9.

First you need to trigger Dex via HDMI cable or HDMI dummy plug (my case), then use scrcpy via WiFi (because your port will be used for the HDMI)
In your case you don't need scrcpy, the samsung app "Dex on Window", also available on Mac.

I'm running linux on my laptop

[rom1v]

In your case you don't need scrcpy, the samsung app "Dex on Window", also available on Mac.

IIUC, this app allows to enable DeX without plugging the device via HDMI?

If this is the case, then it shows that it is possible to enable DeX programmatically. If someone knows how, it could be implemented in scrcpy :wink:

[jvisser99]

In your case you don't need scrcpy, the samsung app "Dex on Window", also available on Mac.

IIUC, this app allows to enable DeX without plugging the device via HDMI?

If this is the case, then it shows that it is possible to enable DeX programmatically. If someone knows how, it could be implemented in scrcpy

I thought it was implemented in 1.13

[rom1v]

No, in scrcpy 1.13, a new option has been added to mirror a display other than the one having id 0.

It occurs that DeX creates such a secondary display when enabled, so scrcpy can mirror it.

[nakele]

@jvisser99 if u want to run Dex on PC you need to install Dex appliaction on your pc as well, plug the cable to phone, wait to say it's connected and you should get a window dex on your pc monitor. No needed for for anythingelse. HoWEVER scrcpy could be used to mirror dex on devices that don't support Dex on PC like the S8/Note8 line + S9/Note9 on Android 9.
First you need to trigger Dex via HDMI cable or HDMI dummy plug (my case), then use scrcpy via WiFi (because your port will be used for the HDMI)
In your case you don't need scrcpy, the samsung app "Dex on Window", also available on Mac.

I'm running linux on my laptop

Ah, I suspected that. I believe wine won't work with Samsung Dex on PC.

[nakele]

In your case you don't need scrcpy, the samsung app "Dex on Window", also available on Mac.

IIUC, this app allows to enable DeX without plugging the device via HDMI?

If this is the case, then it shows that it is possible to enable DeX programmatically. If someone knows how, it could be implemented in scrcpy 😉

It's possible, definitely.
2 things comes to mind, if not 3: Dex on PC will trigger dex programmatically, Dex on a Tab has a toggle to show DeX on the tablet itself, so no hdmi required. Lastly someone has seen Dex Wireless options (might be a thing in the future)

[webian]

There's also this thread in Samsung forum where a moderator says, in October 2018, that they are "working on a wireless connection for DEX but a few improvements need to be made first". If you want you can add your post to the thread so they know how much this feature is requested.

[rom1v]

I don't have any DeX device. Could you please check if enabling dex with "Dex on Windows" changes settings:

# with dex disabled
adb shell "settings list global | sort" > global.txt
adb shell "settings list secure | sort" > secure.txt
adb shell "settings list system | sort" > system.txt
# with dex enabled
adb shell "settings list global | sort" > global_dex.txt
adb shell "settings list secure | sort" > secure_dex.txt
adb shell "settings list system | sort" > system_dex.txt
# compare
diff -U3 global.txt global_dex.txt
diff -U3 secure.txt secure_dex.txt
diff -U3 system.txt system_dex.txt

[webian]

These are the diffs before/after connecting my Galaxy S10e with USB cable and starting Dex on PC:

• global: no diff
• secure
• system

[timerickson]

A couple more datapoints:

• https://www.sammobile.com/news/samsung-dex-wireless-mode-coming-with-galaxy-note-20/
• https://kmyers.me/blog/android/dexonlinux-dexonchromeos-how-to-no-root/