Scrcpy: Scrcpy interaction with OPPO Phone is not working

[EloZ27]

I have tried scrcpy and various guidelines for mirroring phones via scrcpy but it seems that the interaction with keyboard and mouse and audio does not work with my OPPO F11.

Help

[rom1v]

https://github.com/Genymobile/scrcpy/blob/master/FAQ.md#mouse-and-keyboard-do-not-work

?

[EloZ27]

Yes, I have already read that. But based in my phone I don't have such option for that. I even looked entirely in my phone, but I haven't found any option for it.

[rom1v]

Any hint in adb logcat?

What happens if you focus some text area on your device, and execute adb shell input text hello from your computer?

[EloZ27]

Cant seem to find any hint in the logcat.

adb shell input text hello worked

[rom1v]

adb shell input text hello worked

Oh, that's very weird, then. Pressing h e l l o in the scrcpy windows does not work?

What if you start with scrcpy --prefer-text?

[EloZ27]

Hello. Here is the result for the scrcpy --prefer-text.

[rom1v]

But what happens when you enter text in the scrcpy window?

[EloZ27]

Hmmm. Somehow, typing in the keyboard worked in scrcpy. But the mouse still does not interact

[rom1v]

Probably similar to #1347.

Does this work: https://github.com/Genymobile/scrcpy/issues/1347#issuecomment-624039383

[EloZ27]

Yea. It some how tapped my screen.

[rom1v]

Please test with older versions of scrcpy, if you find one which work, it will help a lot to find the cause.

[EloZ27]

I have tried versions down to v1.4, scroll wheel and left mouse button does not work. Middle click and right click worked as well as the keyboard.

[GetCurious]

scrcpy 1.12.1

dependencies:

• SDL 2.0.10
• libavcodec 58.54.100
• libavformat 58.29.100
• libavutil 56.31.100

adb shell input tap 179 179

log.txt

same phone (F11 Pro), same situation, most keyboard, mid click & right click works except left click.

[rom1v]

adb shell input tap 179 179

Does it work or not?

[GetCurious]

now running build master version

scrcpy 1.14

dependencies:

• SDL 2.0.10
• libavcodec 58.54.100
• libavformat 58.29.100
• libavutil 56.31.100

the command works. but left click doesnt.

[rom1v]

Oh, could you test with this change, please:

diff --git a/app/src/input_manager.c b/app/src/input_manager.c
index 9c22ee0a..7d500efe 100644
--- a/app/src/input_manager.c
+++ b/app/src/input_manager.c
@@ -511,7 +511,8 @@ convert_mouse_button(const SDL_MouseButtonEvent *from, struct screen *screen,
     to->inject_touch_event.position.screen_size = screen->frame_size;
     to->inject_touch_event.position.point =
         screen_convert_window_to_frame_coords(screen, from->x, from->y);
-    to->inject_touch_event.pressure = 1.f;
+    to->inject_touch_event.pressure =
+        from->type == SDL_MOUSEBUTTONDOWN ? 1.f : 0.f;
     to->inject_touch_event.buttons =
         convert_mouse_buttons(SDL_BUTTON(from->button));

[GetCurious]

```static bool
convert_mouse_button(const SDL_MouseButtonEvent *from, struct screen *screen,
struct control_msg *to) {
to->type = CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT;

if (!convert_mouse_action(from->type, &to->inject_touch_event.action)) {
    return false;
}

to->inject_touch_event.pointer_id = POINTER_ID_MOUSE;
to->inject_touch_event.position.screen_size = screen->frame_size;
to->inject_touch_event.position.point =
    screen_convert_to_frame_coords(screen, from->x, from->y);
to->inject_touch_event.pressure =
    from->type == SDL_MOUSEBUTTONDOWN ? 1.f : 0.f;
to->inject_touch_event.buttons =
    convert_mouse_buttons(SDL_BUTTON(from->button));

return true;

}
```
no difference.
can i have your full input_manager.c?

[rom1v]

no difference.

Could you please test this change instead (or in addition), on dev:

diff --git a/server/src/main/java/com/genymobile/scrcpy/Controller.java b/server/src/main/java/com/genymobile/scrcpy/Controller.java
index 71e7ec9c..1f249225 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Controller.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Controller.java
@@ -11,7 +11,7 @@ import java.io.IOException;

 public class Controller {

-    private static final int DEVICE_ID_VIRTUAL = -1;
+    private static final int DEVICE_ID_VIRTUAL = 0;

     private final Device device;
     private final DesktopConnection connection;

can i have your full input_manager.c?

(I use the one from current dev branch)

Note: adb shell input tap ... calls this: https://github.com/aosp-mirror/platform_frameworks_base/blob/22e3e74e4b3b989d20bb17ca7d54f95b67d6c02c/cmds/input/src/com/android/commands/input/Input.java#L221-L226

[GetCurious]

i got nothing (instead and in-addition).

[rom1v]

@GetCurious OK, just to be sure, you are building the server (you don't use the prebuilt-server)?

What we have to understand is why this works (adb shell input tap ...): https://github.com/aosp-mirror/platform_frameworks_base/blob/22e3e74e4b3b989d20bb17ca7d54f95b67d6c02c/cmds/input/src/com/android/commands/input/Input.java#L221-L226

but this does not:
https://github.com/Genymobile/scrcpy/blob/5fa46ad0c71169f9369f9f3bde9cce3d29ed4b88/server/src/main/java/com/genymobile/scrcpy/Controller.java#L198-L200

[GetCurious]

$ cd scrcpy
$ vim app/src/input_manager.c
$ vim server/src/main/java/com/genymobile/scrcpy/Controller.java
$ meson x --buildtype release --strip -Db_lto=true
$ ninja -Cx 
$ ./run x

anything i miss?

[rom1v]

Let's try to use a device id matching the input source, like they do:

diff --git a/server/src/main/java/com/genymobile/scrcpy/Controller.java b/server/src/main/java/com/genymobile/scrcpy/Controller.java
index 71e7ec9c..c1949f36 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Controller.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Controller.java
@@ -195,12 +195,26 @@ public class Controller {
             }
         }

+        int deviceId = getInputDeviceId(InputDevice.SOURCE_TOUCHSCREEN);
+        Ln.i("device id = " + deviceId);
         MotionEvent event = MotionEvent
-                .obtain(lastTouchDown, now, action, pointerCount, pointerProperties, pointerCoords, 0, buttons, 1f, 1f, DEVICE_ID_VIRTUAL, 0,
+                .obtain(lastTouchDown, now, action, pointerCount, pointerProperties, pointerCoords, 0, buttons, 1f, 1f, deviceId, 0,
                         InputDevice.SOURCE_TOUCHSCREEN, 0);
         return device.injectEvent(event);
     }

+    private static int getInputDeviceId(int inputSource) {
+        final int DEFAULT_DEVICE_ID = 0;
+        int[] devIds = InputDevice.getDeviceIds();
+        for (int devId : devIds) {
+            InputDevice inputDev = InputDevice.getDevice(devId);
+            if (inputDev.supportsSource(inputSource)) {
+                return devId;
+            }
+        }
+        return DEFAULT_DEVICE_ID;
+    }
+
     private boolean injectScroll(Position position, int hScroll, int vScroll) {
         long now = SystemClock.uptimeMillis();
         Point point = device.getPhysicalPoint(position);

[GetCurious]

no difference

[rom1v]

What's the id printed in the console when you click with this change?

[GetCurious]

how do i do that?

[rom1v]

When you execute ./run x, what is the output in the console?

[GetCurious]

➜  scrcpy git:(dev) ✗ ./run x                                                      
INFO: scrcpy 1.14 <https://github.com/Genymobile/scrcpy>
x/server/scrcpy-server: 1 file pushed, 0 skipped. 107.0 MB/s (33374 bytes in 0.000s)
[server] INFO: Device: OPPO CPH1969 (Android 10)
INFO: Renderer: opengl
INFO: OpenGL version: 4.6.0 NVIDIA 440.100
INFO: Trilinear filtering enabled
INFO: Initial texture: 1080x2336
INFO: New texture: 888x1920

[rom1v]

And nothing more, even when you click in the scrcpy window?

EDIT: in itself:

INFO: Initial texture: 1080x2336
INFO: New texture: 888x1920

it's a bit surprising (the device sends frame at a different resolution from what has been requested). But probably not related.

[GetCurious]

that's right... nothing more.
everything except left-click works

[rom1v]

With that change: https://github.com/Genymobile/scrcpy/issues/1518#issuecomment-655440431 it should, since a log is added on every click.

Unless the client does not send the events to the device. Let's add more logs:

diff --git a/app/src/input_manager.c b/app/src/input_manager.c
index 7d500efe..fc0edb6d 100644
--- a/app/src/input_manager.c
+++ b/app/src/input_manager.c
@@ -523,6 +523,14 @@ void
 input_manager_process_mouse_button(struct input_manager *im,
                                    const SDL_MouseButtonEvent *event,
                                    bool control) {
+    LOGI("mouse button: type=%d, which=%d, button=%d, state=%d, clicks=%d, x=%d, y=%d",
+            (int) event->type,
+            (int) event->which,
+            (int) event->button,
+            (int) event->state,
+            (int) event->clicks,
+            (int) event->x,
+            (int) event->y);
     if (event->which == SDL_TOUCH_MOUSEID) {
         // simulated from touch events, so it's a duplicate
         return;
@@ -559,9 +567,12 @@ input_manager_process_mouse_button(struct input_manager *im,

     struct control_msg msg;
     if (convert_mouse_button(event, im->screen, &msg)) {
+        LOGI("==== convert_mouse_button OK");
         if (!controller_push_msg(im->controller, &msg)) {
             LOGW("Could not request 'inject mouse button event'");
         }
+    } else {
+        LOGW("==== convert_mouse_button KO");
     }
 }

[GetCurious]

INFO: mouse button: type=1025, which=0, button=1, state=1, clicks=1, x=102, y=631
INFO: ==== convert_mouse_button OK
INFO: mouse button: type=1026, which=0, button=1, state=0, clicks=1, x=102, y=631
INFO: ==== convert_mouse_button OK

[rom1v]

OK, so the events are correctly forwarded by the client.

One more diff:

diff --git a/server/src/main/java/com/genymobile/scrcpy/Controller.java b/server/src/main/java/com/genymobile/scrcpy/Controller.java
index 71e7ec9c..1a5fad7f 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Controller.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Controller.java
@@ -71,6 +71,7 @@ public class Controller {

     private void handleEvent() throws IOException {
         ControlMessage msg = connection.receiveControlMessage();
+        Ln.i("Received msg type = " + msg.getType());
         switch (msg.getType()) {
             case ControlMessage.TYPE_INJECT_KEYCODE:
                 if (device.supportsInputEvents()) {
@@ -163,6 +164,7 @@ public class Controller {

     private boolean injectTouch(int action, long pointerId, Position position, float pressure, int buttons) {
         long now = SystemClock.uptimeMillis();
+        Ln.i("=== injectTouch");

         Point point = device.getPhysicalPoint(position);
         if (point == null) {
@@ -170,6 +172,8 @@ public class Controller {
             return false;
         }

+        Ln.i("Point = " + point);
+
         int pointerIndex = pointersState.getPointerIndex(pointerId);
         if (pointerIndex == -1) {
             Ln.w("Too many pointers for touch event");

[rom1v]

https://github.com/Genymobile/scrcpy/issues/1518#issuecomment-655499111

But probably not related.

Oh in fact, that might be related: the client sends clicks for a window having dimensions different from the ones provided by the device.

If you remove these lines, it should "work" (it may click at a wrong location though):

https://github.com/Genymobile/scrcpy/blob/e99b896ae2b6a59d5f7f2b76a3dd9c29f4838013/server/src/main/java/com/genymobile/scrcpy/Device.java#L134-L138

INFO: Initial texture: 1080x2336
INFO: New texture: 888x1920

What if you run scrcpy -m1920? Do clicks work in that case?

[GetCurious]

what the...
scrcpy -m1920 worked

note: using original master build

[rom1v]

One more test, please (with just ./run x, without -m parameter):

diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
index d722388c..41afd35b 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
@@ -82,6 +82,9 @@ public class ScreenEncoder implements Device.RotationListener {

                 setSize(format, videoRect.width(), videoRect.height());
                 configure(codec, format);
+                MediaFormat actualFormat = codec.getInputFormat();
+                Ln.i("width = " + actualFormat.getInteger(MediaFormat.KEY_WIDTH));
+                Ln.i("height = " + actualFormat.getInteger(MediaFormat.KEY_HEIGHT));
                 Surface surface = codec.createInputSurface();
                 setDisplaySurface(display, surface, videoRotation, contentRect, unlockedVideoRect, layerStack);
                 codec.start();

[GetCurious]

INFO: Initial texture: 1080x2336
INFO: New texture: 888x1920
[server] INFO: Device rotation requested: landscape
[server] INFO: width = 2336
[server] INFO: height = 1080
INFO: New texture: 1920x888
[server] INFO: Device rotation requested: portrait
[server] INFO: width = 1080
[server] INFO: height = 2336
INFO: New texture: 888x1920

[rom1v]

OK, and with

MediaFormat actualFormat = codec.getOutputFormat();

instead of

MediaFormat actualFormat = codec.getInputFormat();

?

diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
index d722388c..7a35aee0 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
@@ -82,6 +82,9 @@ public class ScreenEncoder implements Device.RotationListener {

                 setSize(format, videoRect.width(), videoRect.height());
                 configure(codec, format);
+                MediaFormat actualFormat = codec.getOutputFormat();
+                Ln.i("width = " + actualFormat.getInteger(MediaFormat.KEY_WIDTH));
+                Ln.i("height = " + actualFormat.getInteger(MediaFormat.KEY_HEIGHT));
                 Surface surface = codec.createInputSurface();
                 setDisplaySurface(display, surface, videoRotation, contentRect, unlockedVideoRect, layerStack);
                 codec.start();

[GetCurious]

INFO: scrcpy 1.14 <https://github.com/Genymobile/scrcpy>
...
INFO: Initial texture: 1080x2336
[server] INFO: width = 1080
[server] INFO: height = 2336
INFO: New texture: 888x1920

[rom1v]

OK, thank you for the tests.

So on the device, we request a specific size (1080x2336), the codec decides to encode in 888x1920 instead (probably because it does not support higher definition), but we can't know what size it decided without decoding the frames :confused:

[rom1v]

Just in case, try to move it after the codec is started:

diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
index d722388c..3ee09aad 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
@@ -104,6 +104,10 @@ public class ScreenEncoder implements Device.RotationListener {
         boolean eof = false;
         MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();

+        MediaFormat actualFormat = codec.getOutputFormat();
+        Ln.i("width = " + actualFormat.getInteger(MediaFormat.KEY_WIDTH));
+        Ln.i("height = " + actualFormat.getInteger(MediaFormat.KEY_HEIGHT));
+
         while (!consumeRotationChange() && !eof) {
             int outputBufferId = codec.dequeueOutputBuffer(bufferInfo, -1);
             eof = (bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0;

[GetCurious]

INFO: scrcpy 1.14 <https://github.com/Genymobile/scrcpy>
x/server/scrcpy-server: 1 file pushed, 0 skipped. 159.5 MB/s (33294 bytes in 0.000s)
[server] INFO: Device: OPPO CPH1969 (Android 10)
[server] INFO: width = 1080
[server] INFO: height = 2336


[server] INFO: encode width = 1080
[server] INFO: encode height = 2336


INFO: OpenGL shaders: ENABLED
INFO: Created renderer: opengl
INFO: Renderer: opengl
INFO: OpenGL version: 4.6.0 NVIDIA 440.100
INFO: Trilinear filtering enabled
INFO: Initial texture: 1080x2336
INFO: New texture: 888x1920

[swamikamal]

i am also getting same problem. i have oppo f1. I HAVE tried all different version of scrcpy but nothing worked. help me if you are able to correct it

[rom1v]

@swamikamal I just gave you the solution: https://github.com/Genymobile/scrcpy/issues/1645#issuecomment-670433990

[eku]

May I join this party. I have a Xioami with Adnroid 7.1. and the solution from #1645 does not help.

[rom1v]

@eku whtat is the full output in your console when you run scrcpy?

[eku]

@rom1v

$ ./run x -b 2M -m 960
INFO: scrcpy 1.16 <https://github.com/Genymobile/scrcpy>
x/server/scrcpy-server: 1 file pushed. 1.9 MB/s (33634 bytes in 0.017s)
[server] INFO: Device: Xiaomi MI MAX 2 (Android 7.1.1)
INFO: Renderer: opengl
INFO: OpenGL version: 3.0 Mesa 20.1.5
INFO: Trilinear filtering enabled
INFO: Initial texture: 544x960

That's all. Scrcpy receives e.g. meta-key combinations and logs them but there is no reaction of the device.

[server] INFO: Device rotation requested: landscape

Nothing happens.

[rom1v]

https://github.com/Genymobile/scrcpy/blob/master/FAQ.md#mouse-and-keyboard-do-not-work ?

[eku]

Yes, I know the FAQ and this point. The setting also exists in my phone but cannot be changed for unknown reason.
Please excuse the disturbance by me.