Scrcpy: Touch input offset on secondary monitor

[esnaultdev]

Hello,
First of all, thank you for this tool.

I know I don't have an usual setup, but here's a bug with it.

This is my setup, I'm on macOS Sierra 10.12.4 and I have two screens:

• The landscape one is the main display for the OS and is 2880x1800
• The portrait one is a secondary display and is 1080x1920 (This is where my IDE lives)

Now, on the main display, everything works fine with scrcpy, but when the window is on the secondary display, there is an offset for the touch input:

[rom1v]

Thank you for reporting this.

I am aware of the problem, but for now, I didn't find a way to fix it without breaking the one-screen scenario.

For now, you can disable hidpi support:

meson x --buildtype release -Dhidpi_support=false

But the video quality will be bad (pixels will be 2×2).

[rom1v]

Does SDL 2.0.9 fix the issue on MacOS?

There is a commit from Oct 12th: metal: Fix high dpi and resizing on macOS, and clean up iOS code. Fixes bug #4250..

[eurekaqq]

@rom1v I have tested latest scrcpy with SDL 2.0.9 on macos 10.14.2, but this problem still exists.

[DougNg]

I can also confirm the issue still exists

scrcpy --version
scrcpy 1.7

dependencies:

• SDL 2.0.9
• libavcodec 58.35.100
• libavformat 58.20.100
• libavutil 56.22.100

[lpkruger]

This issue was bothering me as well, I still think it's a SDL upstream bug but this workaround is simple and safe. If SDL fixes it upstream this workaround won't break, but maybe could be removed eventually.

Normally in HiDPI mode, the Renderer resolution is exactly double the window resolution, which is easy to see by comparing the following:

SDL_GetWindowSize(screen.window, &w, &h);
LOGD("Window is %d %d\n", w, h);
SDL_GetRendererOutputSize(screen.renderer, &w, &h);
LOGD("Renderer is %d %d\n", w, h);

But in the mixed-monitor configuration it gets confused after a window resize and they become the same. This leads to an inconsistency where some part of the SDL code still wants to double the X,Y coordinates of the mouse events.

This fix brute-forces things by reinitializing the rendering state after a window resize. I've tested it in a multi-monitor config with every combination of dragging, resizing, fullscreening, and using control-keys to make the app do resizing I can think of, I think it's harmless. Maybe a little inefficient but it only happens in response to resizes.

[lpkruger]

PR https://github.com/Genymobile/scrcpy/pull/848

[lpkruger]

Hi @rom1v I tested the workaround using SDL_RenderSetLogicalSize() on resize events - unfortunately it does not help when I tested it, that call doesn't fix whatever part of the SDL state gets corrupted.

I was able to update my existing patch so that the re-creation of the rendering and texture state never occurs when only a single monitor is connected - it does this by testing whether that ratio of window "size" to rendering pixel size has unexpectedly changed and only then resets the state. I believe if the SDL bug only affects the multi-monitor case, then this reset call also would only happen when multiple monitors have been connected.

Please let me know your thoughts when you have a chance to look it over.

[rom1v]

Could you please test logical_size.3? All sizes and location are computed manually, so maybe it makes the hidpi issue obsolete.

[esnaultdev]

I just tested the fixhidpi.4 branch and the touch input is now ok on both the hidpi screen and the normal one :+1:
The phone screen is indeed displayed as hidpi on the hidpi screen.

@rom1v I haven't quite followed the issue and the related PRs, do you need me to test some specific behaviour?

[rom1v]

Great :+1: Thank you.

Could you confirm that it is broken without fixhidpi.4 (on current dev) on your configuration?

do you need me to test some specific behaviour?

Especially test clicks after moving the scrcpy window from one screen to one another.

[esnaultdev]

Hmmm. That's odd, it already seems to be working on the dev branch.

I tried deleting the x folder and rebuilding to come from a clean state, but I have the same behaviour.

[rom1v]

OK, thank you for your tests.

Do someone still have the issue on dev? If yes, could you test fixhidpi.4 please :)

[rom1v]

@AoDevBlue Oh, could you test if the problem exists on tag v1.10 (or few commits later to fix build issues on macOS: 9a50b65), please?

It may have been fixed on master/dev by https://github.com/Genymobile/scrcpy/issues/829.

(EDIT: ah, in fact, probably not, #829 is a PR from @lpkruger, which is also the author of #848 to fix this problem)

[esnaultdev]

v1.10 does not build successfully on my mac, indeed.

Building from 9a50b65b338699ab6f0916712cda806353127434, I don't have the problem, but the display isn't hidpi on the mac screen.

[rom1v]

OK, thank you for your test :+1:

[maggu2810]

Hi, I am running KDE on Wayland using two monitors.
My laptop monitor uses a resolution of 3840x2160 and a scale of 2.0
My external monitor uses a resolution of 1920x1080 and a scale of 1.0.

After the initial start the scrcpy shows me only a part of the screen.
After pressing ctrl + x the mirrored screen looks correctly but the touch events I generate with my mouse are at the wrong position.

Is there something I can test, debug, analyze, ...?

[rom1v]

Hi, I am running KDE on Wayland using two monitors.

How, great, it happens also on Linux :tada:

My laptop monitor uses a resolution of 3840x2160 and a scale of 2.0
My external monitor uses a resolution of 1920x1080 and a scale of 1.0.

Is it possible to customize the scale per-display? (for example if you have two monitors at 1920x1080, does KDE allow to assign a scale of 1.0 to the first and 2.0 to the second?)

I just tested a Kubuntu livecd, but it seems the "scale display" applies to all? (or not, I'm not sure) It says the setting will be applied after a reboot (I'm on a liveusb :confused:) so I can't test.

Is there something I can test, debug, analyze, ...?

Maybe compare master with fixhidpi.4?

[maggu2810]

AFAIK to use multiple monitors with different scale is not possible with X11, but possible using Wayland.
For me this is a really great feature as you can still use old (non high DPI) monitors with more recent laptops.

I will test a few things and comment here later.

[maggu2810]

Laptop:

• native resolution (scale 1.0): 3840x2160
• resolution using scale 2.0: 1920x1080

External monitor:

• native resolution (scale 1.0): 1920x1080

Environment:

• Wayland
• KDE Plasma
• scrcpy 1.21.1

| Laptop | External Monitor | Positioning | Comment |
| ------ | ---------------- | ------- | ------- |
| 1920x1080, 2.0 | 1920x1080, 1.0 | Laptop and external one both starts at 0,0 (both use the same visible area) | All works as expected |
| 1920x1080, 2.0 | 1920x1080, 1.0 | Laptop is left of the external one (but at the same y-pos) | All works as expected | If scrcpy starts on the laptop monitor and the window is kept there, all is fine. As soon as the window is moved to the external monitor and the first touch / mouse event is received the content of the window is resized and only a small part is visible. So, not usable. If the window is kept on the external monitor and ctrl + x is pressed, the visible part looks normally again. But the input events are not at the correct position. If it is moved back to the laptop, the input events are correctly again after pressing ctrl + x once. |
| 3840x2160, 1.0 | 1920x1080, 1.0 | Laptop and external one both starts at 0,0 (the laptops area exceed the external one on the right and bottom side) | All fine (for sure, if the window is moved to the right or bottom side of the laptop monitor it is not visible because both monitors starts at the same position and the laptop one has a much more area that can be shown -- expected behavior. Just note it to make it clear.) |
| 3840x2160, 1.0 | 1920x1080, 1.0 | Laptop left of external | If scrcpy starts on the laptop monitor and the window is kept there, all is fine. As soon as the window is moved to the external monitor and the first touch / mouse event is received the content of the window is resized and only a small part is visible. So, not usable. If the window is moved back to the laptop monitor it can be used normalle (I need to press ctrl + x once because a automatically resize is to small). |

[carstenhag]

Hello, I have tried out the mentioned branch and commented everything I knew on https://github.com/Genymobile/scrcpy/pull/848. It would be awesome if you'd have a look on my comment. Thanks in advance!

[rom1v]

@carstenhag Could you please test the vranch logical_size.14?

[carstenhag]

Sure. Here's the output:

carstenh  ~  git  random  scrcpy  %  ./run x                                             logical_size.14
2020-05-18 14:08:10.866 scrcpy[9026:275612] INFO: scrcpy 1.13 <https://github.com/Genymobile/scrcpy>
x/server/scrcpy-server: 1 file pushed. 3.3 MB/s (30558 bytes in 0.009s)
[server] INFO: Device: samsung SM-A705FN (Android 10)
2020-05-18 14:08:11.335 scrcpy[9026:275612] INFO: Renderer: metal
2020-05-18 14:08:11.335 scrcpy[9026:275612] WARN: Trilinear filtering disabled (not an OpenGL renderer)
2020-05-18 14:08:11.338 scrcpy[9026:275612] INFO: Initial texture: 1080x2400

Screenshot just after starting scrcpy. After resizing the window, it's fine again:

When I press command + g the window is being resized, but there are black borders around it. This exists on the latest version and on this branch, so it's probably a different thing.

The main issue is fixed though, the mouse clicks are accurate again 🎉 . Now only the initial size has to be fixed.

[rom1v]

Screenshot just after starting scrcpy. After resizing the window, it's fine again:

OK, (almost) perfect :+1:

If you start scrcpy on your primary screen, then moves the window to the secondary screen, does it happen too? (or does it happen only if you start scrcpy directly on the secondary screen?)

Are you available on IRC or somewhere to do some tests to fix the problem? (preferably some day in the evening)

When I press command + g the window is being resized, but there are black borders around it. This exists on the latest version and on this branch, so it's probably a different thing.

Yes, this is different: Ctrl+g resizes the window to the device size, but on some platforms (some window managers) if the window is bigger than the computer screen, the size is clamped (so it does not work as expected).

The main issue is fixed though, the mouse clicks are accurate again :tada:

:tada:

[carstenhag]

Yes, let's talk on irc, I have time today or the next days, I don't mind. Just tell me which irc server/channel I should join :).

[carstenhag]

But yeah, doesn't matter where my terminal window is or where I last closed scrcpy, it always opens scrcpy on my main screen.

[rom1v]

Cool, thx :)

Just tell me which irc server/channel I should join :)

My nickname is rom1v on freenode.

[rom1v]

@carstenhag Thank you for your help. So it is fixed now :tada:

Before merging, I would like more tests from users. If you had this problem (or even if you had not), please test #1408.

Thank you for your feedback :)

[rom1v]

Fixed by #1408.