Monogame: Support text compositions

[Jjagg]

MonoGame does not support getting IMM compositions. We should expose an API so games can render these compositions instead of having the OS show an ugly window for it.

We should discuss the API before talking about the implementation. The API should support starting and stopping composited input and getting the current composition (possibly using an event similar to TextInput). TextInput should be invoked when a composition is finished.

API Suggestion

Below methods and event will be exposed in GameWindow.

/// <summary>
/// Enable the system IMM service to support composited character input.
/// This should be called when you expect text input from a user and you support languages
/// that require an IME (Input Method Editor).
/// </summary>
/// <seealso cref="EndTextComposition" />
/// <seealso cref="TextComposition" />
void StartTextComposition();

/// <summary>
/// Stop the system IMM service.
/// </summary>
/// <seealso cref="StartTextComposition" />
/// <seealso cref="TextComposition" />
void StopTextComposition();

/// <summary>
/// Invoked when the IMM service is enabled and a character composition is changed.
/// </summary>
/// <seealso cref="StartTextComposition" />
/// <seealso cref="EndTextComposition" />
event EventHandler<TextCompositionEventArgs> TextComposition;

/// <summary>
/// Arguments for the <see cref="GameWindow.TextComposition" /> event.
/// </summary>
class TextCompositionEventArgs : EventArgs
{
    /// <summary>
    /// The full string as it's composited by the IMM.
    /// </summary>    
    public string CompositedText { get; }

    /// <summary>
    /// The position of the cursor inside the composited string.
    /// </summary>    
    public int CursorPosition { get; }

    /// <summary>
    /// The suggested alternative texts for the composition.
    /// </summary>    
    public string[] CandidateList { get; }
}

EDITS:

• Added TextCompositionEventArgs.CandidateList
  To render an IME you at least need a candidate list.

Links

Forum post: http://community.monogame.net/t/i-cant-use-ime-for-chinese-input/10737
Microsoft docs: https://docs.microsoft.com/en-us/windows/desktop/DxTechArts/using-an-input-method-editor-in-a-game
SDL docs: https://wiki.libsdl.org/Tutorials/TextInput

Implementation notes

• The API should be exposed in GameWindow
• The Windows implementation needs to handle certain window messages in the WndProc
• The SDL implementation (DesktopGL) handles events in SDLGamePlatform.

Questions

• What about mobile and consoles? Do they support IME? How?

[Jjagg]

cc @tomspilman @cra0zy @dellis1972 because this adds API.

[mrhelmut]

All consoles/mobiles have their own native IME. Most of the (console) ports have some wrapper support for it, but nothing that would be candidate for a high level cross-platform API.

Your API suggestion looks to be solid and it would be cool to convert all targets to it.

There's a private UpdateIME() that should be called anytime the IME is active, that would help some IME API that require active polling of the dialog state.

Not sure if they all support cursor positioning, and most of them have specific parameters (such as the supported character set, modes, screen positioning, etc.), but I believe that a generic IME would do the trick for most inputing tasks.

If that may help, I can probably list and/or figure out what would be the broadest param set to support.

[ClEngine]

6348

[Jjagg]

Looks like SDL wants to render its own IME, so not sure about the low-levelness of the suggested API now. From what I gather, you can't get a candidate list on SDL and must either let the OS handle the IME or let SDL render it. Docs are limited, so hard to figure this stuff out.

[tomspilman]

I am totally ignorant of what IMM or IME or whatever it is called is. I assume it is "show a on screen keyboard or input method of some sort".

How is this different from KeyboardInput.Show which we already have?

https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Input/KeyboardInput.cs#L24

[EmptyKeys]

@tomspilman Very, it's not about showing keyboard, but about showing what words you can write (candidate list) based on your input.

IMM
https://docs.microsoft.com/en-us/windows/desktop/intl/about-input-method-manager

@Jjagg Be careful with implementation. Windows have different ways for the same thing, so for example Windows builtin chinese will not work with IMM and it's crap. I would suggest to make support for https://pinyin.sogou.com and be ready for some crazy hacks like hidden input text field out of the screen.

[ryancheung]

https://github.com/xenko3d/xenko I tested the demo project, it handles the IME. Xenko may be used as reference.

[ryancheung]

Here is the SDL IME implementation: https://github.com/xenko3d/xenko/blob/master/sources/engine/Xenko.Input/SDL/KeyboardSDL.cs

[ryancheung]

After few months experience with MG and game developing, I found MG definitely need to provide a solid IME/IMM API (like other commercial Game Engines did) for the games that have a textbox and target users with CJK languages.

And after some experiences on https://github.com/ryancheung/MonoGame.IMEHelper, I'm gonna make a PR to fix all the IME hassle in MG.

The new IME APIs suggested by me:

A IImeService is added MonoGame.Framework/Input folder. This service is going to provide text input or composition APIs for SDL2, WindowsDX, Android and iOS platforms. More platforms like (UAP, PlayStation 4, Nintendo Switch etc.) could be added in the future by someone, if the platform has the support.

public interface IImeService
{
    /// <summary>
    /// Enable the system IMM service to support composited character input.
    /// This should be called when you expect text input from a user and you support languages
    /// that require an IME (Input Method Editor).
    /// </summary>
    void StartTextInput();

    /// <summary>
    /// Stop the system IMM service.
    /// </summary>
    void StopTextInput();

    /// <summary>
    /// Show the Ime Candidate window renderer by the OS.
    /// In SDL2 platform it's always true since there's no API to get candidate texts in SDL2.
    /// In WindowsDX platform, it's possible to choose show candiate window renderer by the OS or render it yourself.
    /// </summary>
    bool ShowOSImeWindow { get; set; }

    /// <summary>
    /// Position Y of virtual keyboard, for mobile platforms has virtual keyboard.
    /// </summary>
    int VirtualKeyboardHeight { get; }

    /// <summary>
    /// Update the Ime service. For instance, stop the text input when clicking on the other
    /// screen area than the virtual keyboard poped up.
    /// </summary>
    void Update(GameTime gameTime);

    /// <summary>
    /// Return true if text input enabled else false;
    /// </summary>
    bool IsTextInputActive();

    /// <summary>
    /// Set the position of Candiate Window rendered by the OS, supported platforms includes SDL and WindowsDX.
    /// In SDL2 there's no API to get the candiate list. We have to use the candidate windows provided by the OS.
    /// But, in windows SDL2 has bug that not showing the candidate window of the OS, so SDL2
    /// platform was suggested not to use in Windows, use WindowsDX instead.
    /// </summary>
    void SetTextInputRect(Rectangle rect);

    /// <summary>
    /// Invoked when the IMM service is enabled and a character composition is changed.
    /// </summary>
    event EventHandler<TextCompositionEventArgs> TextComposition;

    /// <summary>
    /// Invoked when the IMM service generates a composition result.
    /// </summary>
    event EventHandler<TextInputEventArgs> TextInput;
}

The TextCompositionEventArgs class:

    /// <summary>
    /// Arguments for the <see cref="IImeService.TextComposition" /> event.
    /// </summary>
    public class TextCompositionEventArgs : EventArgs
    {
        /// <summary>
        /// The full string as it's composited by the IMM.
        /// </summary>    
        public string CompositionString { get; }

        /// <summary>
        /// The position of the cursor inside the composited string.
        /// </summary>    
        public int CursorPosition { get; }

        /// <summary>
        /// The candidate text list for the composition.
        /// Note candidate properties are only valid in  WindowsDX(or UAP) platform.
        /// It could be empty if compsition string does not generates candidates.
        /// </summary>    
        public string[] CandidateList { get; }

        /// <summary>
        /// First candidate index of current page.
        /// </summary>
        public uint CandidatePageStart { get; }

        /// <summary>
        /// How many candidates should display per page.
        /// </summary>
        public uint CandidatePageSize { get; }

        /// <summary>
        /// The selected candidate index.
        /// </summary>
        public uint CandidateSelection { get; }
    }

Platform specific IImeService implementations I'm going to add:

• WinFormsImeHandler.cs
• SDLImeHandler.cs
• AndroidImeHandler.cs
• iOSImeHandler.cs

Some notes on the Android/iOS implementation:

The implementation of ImeHandler in Android or iOS is basiclly by creating a native invisible textbox (Android is EditText, iOS is UIBackwardsTextField) and then listening the text change events on them.

@Jjagg @mrhelmut @harry-cpp Could you give any review on this?

[Jjagg]

@ryancheung From what I know of IME services I think that's a great API!

@tomspilman This uses an interface to abstract across platforms, is that okay for you? IMO for new API we should go that route.

One thing I think we should watch out for is allocations. The event arg type can be a struct and we could either use an enumerable for the candidate list or pool the array.

cc @harry-cpp @mrhelmut