Scrcpy: Umlauts don't work

[metaa]

Hey there, it seems that keyboard input doesn't quite work with umlauts. Every time I try to enter one, it inserts the base character and then adds the character for the dots after it and then weirdly selects and sometimes deletes it, but sometimes ... it doesn't ... I don't know.

Just watch this gif:

Is that something wrong on my side or is that scrcpy's fault?
My PC and phone both use the same language.

[rom1v]

Thank you for your report.

Could you test with the old v1.3, please?

I know about this problem, it is a consequence of the "compromise" to fix input in games (https://github.com/Genymobile/scrcpy/commit/411aa4fcfd6ed245252be29660777f9c286448ec).

When I merged it, I didn't think about this side-effect. I plan to add an option to configure how raw inputs are handled (several modes have both upsides and downsides).

[metaa]

Could you test with the old v1.3, please?

Same thing 🤔

[rom1v]

OK, so this is the other possible cause: your keyboard does not handle key composition correctly (IIRC, this is the case for the Google keyboard). I have no solution for this problem unfortunately (this key composition is already a poor solution to workaround Android limitations for injecting non-ASCII characters).

[tomekdev]

How about adding support for handling input using Linux uinput feature ? It handles raw keycodes like phone has connected an external keyboard. However I don't know about umlauts or other non-ASCII characters, maybe I'll check it later. The disadvantage is that it requires ioctl() handling what invokes need of root access. Sample code I've tested (from my abandoned project): https://github.com/tomekdev/android-live-control/blob/master/server/uinput.cpp
Moreover this approach will also need cross-compiling or not unless there is java interface for this

[rom1v]

need of root access

Yes, that's a problem.

Also see: https://github.com/Genymobile/scrcpy/issues/279

[tomekdev]

On StackOverflow I've found something like this: https://stackoverflow.com/questions/14224549/adb-shell-input-unicode-character
Hope this could help a little bit

[rom1v]

In the comments: https://github.com/senzhk/ADBKeyBoard

[npes87184]

In the comments: https://github.com/senzhk/ADBKeyBoard

After testing, I can confirm that ADBKeyBoard works.

The other question is how to integrate ADBKeyBoard into scrcpy.

Maybe we can simply send broadcast to ADBKeyBoard and it normally works. However, this method will suffer some delay, not reliable, etc. I recommend to implement a similar input method which relies on scrcpy socket.

[rom1v]

Thank you for your test :+1:

This method requires to install an app on the device (and enable the keyboard in settings). So if we integrate such a thing, I think it needs to be optional (explicitly requested on the command line).

[npes87184]

Thank you for your test

This method requires to _install_ an app on the device (and enable the keyboard in settings). So if we integrate such a thing, I think it needs to be optional (explicitly requested on the command line).

Yes!

There is a simple proposal, create a scrcpy-input-method.apk which listen to scrcpy-input socket and implement similar method from ADBKeyBoard. If user explicitly setup, we create scrcpy-input socket and install scrcpy-input-method.apk. Functions like copy_text will send to scrcpy-input socket. Also, when leaving, we should uninstall scrcpy-input-method.apk and change input method back to the original method.

[metaa]

Also, when leaving, we should uninstall scrcpy-input-method.apk and change input method back to the original method.

Always consider either the app, the phone or scrcpy crashing. How would that be reset then?

[npes87184]

Hi, all

I am working on the proposal I presented.
There is some result, see YouTube.

The SocketIME repository is here.

In the next step, I will keep improving it and try to integrate to scrcpy.

Any recommendation is welcome.

Cheers!

[rom1v]

Cool, thank you for the PoC! :+1:

[npes87184]

Hi, all

There are 3 options for integrating SocketIME into scrcpy.

1. SocketIME as a git submodule for scrcpy. We provide a build option for building and supporting SocketIME. We may need to add an additional thread for handling input window, events, etc.
2. Instead of building SocketIME, we provide a hint for user to download it from the release page of SocketIME. Like option 1, we need to implement text windows in scrcpy.
3. SocketIME as a different project. We don't integrate it into scrcpy. Instead, we start to implement pc client for SocketIME. Like now, user can use scrcpy as normal. If user want to type some complex text, user can turn on SocketIME client.

My concern is about implementing such things in scrcpy is not a elegant way...

Before doing something, I'd like to hear your opinion on it.
Cheers!

[metaa]

In my opinion, being able to type whatever character is needed in the user's language of choice should be considered part of scrcpy's basic functionality just as much as its default keyboard text input is.

Otherwise this issue renders scrcpy completely unusable (as far as text input goes) for people whose languages don't even use latin characters like most Asian languages, Russian or Greek do for example.

I really hope that everyone can agree on this with me.

[rom1v]

@npes87184 I'm not sure the keyboard should use a socket. IMO it could be controlled via intents/binder from the _scrcpy server_ (the client would just send text events, and the server would send intents to the keyboard). What do you think?

[npes87184]

@npes87184 I'm not sure the keyboard should use a socket. IMO it could be controlled via intents/binder from the _scrcpy server_ (the client would just send text events, and the server would send intents to the keyboard). What do you think?

The code may more clear in this way. However, we may suffer some delay when user types, I think.
Also, if we want to provide an input field, we still need to add an additional thread to handle, IMO.

[metaa]

This seems to work for (German) umlauts now (I'm on v1.11), but not alt keys (like there is Alt Gr + E for the Euro € symbol).
Apart from that, that's a nice improvement already. 😀

[PureIncompetence]

@metaa With what parameters do umlauts work for you ? I (currently on v.1.14) do not get them to work.

[metaa]

@metaa With what parameters do umlauts work for you ? I (currently on v.1.14) do not get them to work.

No parameters at all.
Maybe it depends on the language. Some characters are their own ASCII symbol, some consist of multiple symbols and include modifiers.

[PureIncompetence]

Thats interesting. What characters can you use, for example ? The ones I'm missing mostly are ä ö ü. They only exist on german-layouted keyboards, but writing without them is horrible.

[metaa]

ä ö ü ß work for me and I actually use 1.11.

[metaa]

1.14 works just the same, but I noticed that (also in 1.11) if you hold the key (or look at it while you press it), you can see for äöü that it first inserts the dots and then the rest (building it from two characters?)

[PureIncompetence]

I wonder if that has anything to do with what keyboard is used. Because in my case when I press any of them, nothing is sendt to the device. I use Hackers Keyboard (https://f-droid.org/en/packages/org.pocketworkstation.pckeyboard/) on Android 9.
Edit: Does not seem to be related. I just tested anyoft keyboard and simple keyboard which made no difference.

[rom1v]

Yes, it can depend on the device keyboard (but it should work with the keyboards you tested).

Did you test with scrcpy --prefer-text?

[PureIncompetence]

Yes, I did. was the first thing I tried after reading through this thread. Sadly it didn't work with that option either. Is there anything else I can provide to find out why it does not work for me ?

[mrbgfromv]

I'm using ubuntu and can't get the umlauts to work ( v. 1.16 ).

[PureIncompetence]

I can narrow it (or at least my specific umlaut-problem) down to using the snap-version.
I had to use the snap package at the beginning, because there was no package in the ubuntu repositories, so I kept using it though there is one, now.
I tried using umlauts in the scrcpy-snap-shell ("snap run --shell scrcpy") which didn't work either, so it seems to be a problem with the particular snap package.

Using the regularly (via apt) installed package as well as the self-compiled (https://github.com/Genymobile/scrcpy/blob/master/BUILD.md) binary umlauts are working as expected. I don't even need to enable --prefer-text. @mrbgfromv Maybe your problem has the same cause ?

Thanks to @rom1v & @metaa for trying to figure this out with me.

[mrbgfromv]

@PureIncompetence THANK YOU very much, I can confirm umlauts working under ubuntu using the package provided via apt.