Scrcpy: Add On Screen buttons

Created on 9 Mar 2018 · 35Comments · Source: Genymobile/scrcpy

My phone has capacitive buttons, and because of that i don't have on-screen buttons. A command line argument for showing the buttons would be nice.

feature request

Source

S0und

👍12

Most helpful comment

Yes, we discussed it internally, but it's not trivial in SDL since there are no widgets.

Meanwhile, you can use:

right-click or Ctrl+b (or Ctrl+Backspace) for BACK,
middle-click or Ctrl+h (or the Home button) for HOME,
Ctrl+m for the thirst button, APP_SWITCH.

rom1v on 9 Mar 2018

👍19 ❤7 🎉4 😄3

All 35 comments

Yes, we discussed it internally, but it's not trivial in SDL since there are no widgets.

Meanwhile, you can use:

right-click or Ctrl+b (or Ctrl+Backspace) for BACK,
middle-click or Ctrl+h (or the Home button) for HOME,
Ctrl+m for the thirst button, APP_SWITCH.

rom1v on 9 Mar 2018

👍19 ❤7 🎉4 😄3

+1 for that! I hope you will find time to investigate that...

davcec on 9 Mar 2018

If this isn't easily done onscreen, then perhaps a sidebar would be doable, such as how the Android Emulator or Vysor handles this.

In addition to the navigation buttons, I'd love to have buttons for screenshots and video capture as well, but perhaps that's out of scope for this issue specifically and should have its own issue.

Edit: I see that issue #21 has been opened to request a keyboard shortcut to start screen record/capture.

screen shot 2018-03-09 at 2 06 18 pm screen shot 2018-03-09 at 2 08 23 pm

Kyle-Falconer on 9 Mar 2018

👍3

For information, on dev, I just implemented new shortcuts:

_Right-click_ presses BACK (when screen is on) (https://github.com/Genymobile/scrcpy/commit/675704c71c0d105af2242d9bbeee83127097fa0d)
_Middle-click_ presses HOME (https://github.com/Genymobile/scrcpy/commit/c87d94ee2779838cf4576f6505a918bc485620d5)

I think it may help.

rom1v on 10 Mar 2018

❤6

For Macbook， “Esc” is back, And "Fn + Left" is Home.

snowdream on 10 Mar 2018

Can you please use the same keys as used in the emulator, and even allow to customize?
Here are those that are used there, for MAC :
https://i.stack.imgur.com/eNB8x.png
There are others for Windows/Linux.

For me, the most commonly used are Home (which is the home button on the keyboard) and Back (which is Escape button on the keyboard).

AndroidDeveloperLB on 11 Mar 2018

For me, the most commonly used are Home (which is the home button on the keyboard)

It also works.

and Back (which is Escape button on the keyboard).

Escape sends ESCAPE to the device, which behaves similar to BACK (but not totally). I don't understand the logic though. Maybe it's the same on the emulator…

rom1v on 11 Mar 2018

@rom1v Home button seems to work.
But Escape button doesn't do anything.

AndroidDeveloperLB on 11 Mar 2018

@Kyle-Falconer Hey, how did you get this working on emulator?

Works fine for devices but I get this error when i run for emulator

➜  ~ scrcpy
/usr/local/share/scrcpy/scrcpy-server.jar: ...le pushed. 1.3 MB/s (19038 bytes in 0.014s)
2018-03-13 12:20:57.823 scrcpy[6894:419807] INFO: Initial texture: 540x960
ERROR: Exception on thread Thread[main,5,main]
java.lang.IllegalStateException
    at android.media.MediaCodec.native_stop(Native Method)
    at android.media.MediaCodec.stop(MediaCodec.java:1901)
    at com.genymobile.scrcpy.ScreenEncoder.streamScreen(ScreenEncoder.java:69)
    at com.genymobile.scrcpy.Server.scrcpy(Server.java:21)
    at com.genymobile.scrcpy.Server.main(Server.java:70)
    at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
    at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:251)
2018-03-13 12:20:58.428 scrcpy[6894:419909] ERROR: Could not open video stream

ArnoldLaishram on 13 Mar 2018

I tested on emulator with Oreo. Works for only Oreo. Doesn't seem to work for Nougat, Marshmellow and below.

ArnoldLaishram on 13 Mar 2018

What does the mouse-right-click do?
To me it seems to only be able to wake the device.

AndroidDeveloperLB on 13 Mar 2018

What does the mouse-right-click do?

If the device screen is off, it turns it on. Otherwise, it presses BACK. But currently, it's only on the dev branch (https://github.com/Genymobile/scrcpy/commit/675704c71c0d105af2242d9bbeee83127097fa0d).

rom1v on 13 Mar 2018

@rom1v It doesn't do BACK for me. Only wakes the device.

AndroidDeveloperLB on 13 Mar 2018

@AndroidDeveloperLB Did you compile from the dev branch (server included)?

rom1v on 13 Mar 2018

@rom1v I only launched the binaries for Windows OS. Did you mean the BACK will work on the dev branch?

AndroidDeveloperLB on 13 Mar 2018

Did you mean the BACK will work on the dev branch?

Yes. It has been implemented after the release.

rom1v on 13 Mar 2018

@rom1v OK sorry.

AndroidDeveloperLB on 13 Mar 2018

@rom1v :
Please, bring "APP_SWITCH" to the mouse! This is the only missing thing! (Maybe using mouse wheel up/down).

"Home" button is NOT working here. Other keys - Ctrl+H and Middle click - both works fine.

"Long press" is not supported.
"Long press Ctrl+H/Home" is NOT working like "long press HOME" on device.
(My setup is "show notifications').
"Long press Ctrl+B" or "Right click and hold" is NOT working like "long press BACK" on device.
(My setup is "lock device").

"Double tap" is not supported.
"Double press Ctrl+S" is NOT like "double press RECENTS" on device.
(Default action is "switch to last used application". Like press once Alt+Tab on PC).

Fast press Ctrl+P and Ctrl+Down (or Ctrl+Down and Ctrl+P) is NOT creating screenshot.

(And last - for screen resize, there could be "resize screen to actual device size").

Version 1.11, device OnePlus 3T. I am using Debian buster but installed the packages from Debian sid/testing repository (dpkg with "--force-depends" because libavcodec58, libavformat58, libsdl2-2.0-0): scrcpy_1.11+ds-1_i386.deb, scrcpy_1.11+ds-1_i386.deb

Very useful application.
Thanks!

mmm4m5m on 5 Jan 2020

Please, bring "APP_SWITCH" to the mouse! This is the only missing thing!

That's true, it's annoying to use the keyboard for that. But there are only 3 buttons on many mice (click, home, back).

(Maybe using mouse wheel up/down).

Mouse wheel up/down are used for scrolling. I would like to keep Ctrl+wheel for zoom or something if we need to. Maybe Cltr+click_wheel?

"Home" button is NOT working here. Other keys - Ctrl+H and Middle click - both works fine.

That's on purpose: 5d473efeb5c4f42a75aad6ab5892e2813358c0c9.

"Long press" is not supported.

Hmm, it is. If I long press with left-click, it acts as if I long pressed with the finger.

"Long press Ctrl+H/Home" is NOT working like "long press HOME" on device.
(My setup is "show notifications').

I don't know, it correctly sends AKEYCODE_HOME with ACTION_DOWN when you press, and AKEYCODE_HOME with ACTION_UP when you release.

"Long press Ctrl+B" or "Right click and hold" is NOT working like "long press BACK" on device.
(My setup is "lock device").

Same with AKEYCODE_BACK.

"Double tap" is not supported.

If I double click on a text, it selects it, while if I single-click, it does not. What do you mean by "double tap" is not supported?

"Double press Ctrl+S" is NOT like "double press RECENTS" on device.
(Default action is "switch to last used application". Like press once Alt+Tab on PC).

On my device (OnePlus 7 Pro), it behaves that way.

Fast press Ctrl+P and Ctrl+Down (or Ctrl+Down and Ctrl+P) is NOT creating screenshot.

On my device, it does. (I didn't know that, but I just tested)

(And last - for screen resize, there could be "resize screen to actual device size").

Ctrl+g

rom1v on 5 Jan 2020

Mouse wheel up/down are used for scrolling. I would like to keep Ctrl+wheel for zoom or something if we need to. Maybe Cltr+click_wheel?

No, "Ctrl" is again a keyboard key.

"Long press" is not supported.
"Double tap" is not supported.

Both are about HOME, BACK, APP_SWITCH (Recents). Android 9 Pie can assign actions (in Setting/Buttons/Buttons).

"Double press Ctrl+S" is NOT like "double press RECENTS" on device.
(Default action is "switch to last used application". Like press once Alt+Tab on PC).

On my device (OnePlus 7 Pro), it behaves that way.

I did mistake! Maybe my double press was very slow.
It works for HOME and APP_SWITCH.
It does NOT work for BACK - Ctrl+b, Ctrl+Backspace, Right-click.

Fast press Ctrl+P and Ctrl+Down (or Ctrl+Down and Ctrl+P) is NOT creating screenshot.

On my device, it does. (I didn't know that, but I just tested)

I did mistake! Somehow my screenshot action (application?) is completely disabled.

(And last - for screen resize, there could be "resize screen to actual device size").

Ctrl+g

Ctrl+G is 1:1 pixels, I mean 1:1 physical size/inches (approximately).

Sorry for my mistakes! :-/

mmm4m5m on 5 Jan 2020

I don't know, it correctly sends AKEYCODE_HOME with ACTION_DOWN when you press, and AKEYCODE_HOME with ACTION_UP when you release.

Oh, actually, it only does that for keyboard shortcuts.

With the mouse, it sends both down/up immediately. It will be a bit more complex for BACK, because the right click is used both for "power on" and "back", and the choice is done by the server (on the device).

Could you please open a separate issue for correctly handling long press of HOME and BACK using the mouse?

rom1v on 5 Jan 2020

Created issue 1062 . (Mouse and keys also, because I do not understand fully your answer)
Thanks! :)

mmm4m5m on 5 Jan 2020

👍1

What was done? Did you add long pressing on navigation buttons (both mouse and keyboard) ?

AndroidDeveloperLB on 5 Jan 2020

@AndroidDeveloperLB - No. Sorry for confusion. Just created new issue for long press. Edited my previous comment.
Also, based on above talk, I added one feature request - 1063 - "APP_SWITCH with middle-click".

mmm4m5m on 6 Jan 2020

@mmm4m5m
Wait, so how do you use HOME using the mouse?
I thought that maybe the middle click would be for home

AndroidDeveloperLB on 6 Jan 2020

@AndroidDeveloperLB - Yes, middle-click is currently for "HOME". I think, it is better if both are switched - keyboard for HOME, mouse for APP_SWITCH. Command line parameter to change this behavior could help.
Anyway, the described workaround can be used in both cases! And with "long press", (when it is fixed/implemented), we will have even more options.

mmm4m5m on 6 Jan 2020

So what would long pressing the middle mouse button do? Will it go to recent tasks?

I also wonder if there is a function to switch to the recent-tasks (double click on recent-tasks button), or even switch left/right (using the new navigation gestures). I ask this because my mouse actually have 2 more buttons. One for back-navigation and one for forward . I use them mostly on the web browser

AndroidDeveloperLB on 6 Jan 2020

@AndroidDeveloperLB - It will do the action you assign in your phone settings.
Android 9 Pie, in settings/buttons/buttons, you can assign additional actions for "double tap" and "long press". I have android 9, but this feature is maybe available for older android versions also.
The only missing part, the only needed thing is - "long press" does not work with scrcpy program.

Switching to last task works fine. If you press "double Ctrl+S" will send "double APP_SWITCH" (double RECENTS) to device.

If your mouse is smart enough, or maybe using some software, then you can assign "Ctrl+S" to your mouse extra buttons.

Gestures, I assume, are more complicated to assign to a key/mouse button. But you can do gestures using the mouse pointer.

mmm4m5m on 6 Jan 2020

Maybe you have some non-vanilla rom, because I have Pixel 4 and I have no setting of what to do upon long/double pressing the home button.
The only thing there is, is the assistant settings.

AndroidDeveloperLB on 6 Jan 2020

And with "long press", (when it is fixed/implemented), we will have even more options.

The fix for #1062 is to correctly send DOWN and UP actions separately for mouse shortcuts (like keyboard shortcuts). The "long press" is still (implicitly) managed by the device.

rom1v on 6 Jan 2020

Maybe it's best to just make a nice UI for the app?
One that allows us to set the various settings of it?
For example, I could set what the side buttons of the mouse would do... Maybe one of them should show the recent tasks, for example...

AndroidDeveloperLB on 6 Jan 2020

Maybe it's best to just make a nice UI for the app?

"just"

rom1v on 6 Jan 2020

@rom1v Sorry. Isn't Google in some good relationship with Genymobile? Maybe they could help?
This is such a great tool. Can't they buy it or something?

AndroidDeveloperLB on 6 Jan 2020

Isn't Google in some good relationship with Genymobile?

I don't know. I'm not a Genymobile employee anymore.

Since the initial release of the project has been published, I develop/maintain scrcpy on my free time (which is scarce, I have a full time job on other projects).