Vrtk: VR text input / Keyboard

[dantman]

Text input in VR is currently problematic.

Steam VR has a builtin keyboard that can also be used in-game and comes built in with intuitive Steam Controller style dual-touchpad input, however there are caveats:

• Because it's a Steam VR feature it won't work in OculusVR games or new SDKs
• While it can be positioned within the world, it cannot be themed to fit into the design of the game
• It's implemented as an OpenVR overlay, as a result without a significant hack the rendering of the game and the keyboard are separate and the keyboard can sometimes overlap incorrectly

You can also manually make your own in-game keyboard with a UI canvas, but that comes with caveats as well:

• Laying out a whole keyboard is tedious, a button for every individual key must be manually created, styled, positioned, and given a ClickKey event
• Additional scripting work is necessary if you require a shift key or symbols keyboard
• After you've finally created a keyboard, you cannot simply drop in an alternative keyboard layout when localizing your game for a language that needs a different keyboard layout; at least not without tediously duplicating and editing your previous keyboard manually
• Your keyboard is stuck with pointer style text input, no option for the Steam VR/Controller style touchpad input

The Proposal

A set of components for virtual keyboard implementation in VRTK:

• A KeyboardLayout scriptable object + editor, allowing you to easily define rows of keyboard keys and what they do; and create multiple layouts
• A KeyboardRenderer (or set of them) responsible for taking a KeyboardLayout and various style parameters and then rendering a keyboard layout to a UI canvas with working modifier keys (shift, symbols keyboards)
  
  
  
  • Enough flexibility to subclass and implement multiple renderers will probably be desirable to implement a few different types of keyboard layouts. Such as split keyboards rendered to two canvases (say, if you wanted to attach them to the controllers) or implement different layout styles (straight grid, staggered keys like on a physical keyboard, or Steam VR/Controller style varying button widths)
  
  
• A keyboard actor responsible for displaying and positioning a keyboard dynamically when a text input is focused or the game asks for a keyboard
• An optional script that'll intercept/override the control mapping when a keyboard is displayed, replacing them with Steam VR/Controller style touchpad + trigger input.
  
  
  
  • This should work not by direct interaction with the keyboard renderer, but instead by interacting as some form of twin-cursor on the UI canvas (in theory you might be able to apply this script to control some non-keyboard input canvases)
  
  

[thestonefox]

This forms part of what the Kickstarter campaign was offering

[burnedikt]

Might be worth looking at the integration of https://github.com/NormalVR/CutieKeys

[dantman]

@burnedikt My WIP PR actually mentions the possibility of an alternative mesh based key layout renderer implementation with either collider based or physics enabled keys.

CutieKeys would mostly just be the latter, a mesh key layout renderer, where each key has physics. After that it doesn't matter how the buttons are pressed as long as it goes through the physics system. So it's up to you what kind of object you want to rig up to hit the keys with.

...speaking of that, you could probably make a giant keyboard where you physically punch keys with your whole hand. Absolutely terrible for practical use, but it could be a fun experiment, or an interesting way of implementing input for a 3 letter initial based local high score in a boxing game.

[Vytek]

Very interesting project: https://github.com/rjth/Punchkeyboard
(MIT license and prediction word!)

[Vytek]

What news?

[dantman]

@Vytek No news, all my free time at the moment is being taken up by Breath of the Wild ;)