Mixedrealitytoolkit-unity: 3D Keyboard

[jbienzms]

Something that’s come up in several conversations with developers is the challenges with virtual keyboard support in 3D. Looking in the forums, this is a common problem there too.

The suggested workaround is covered in the article Keyboard input in Unity, but that guidance requires that the user leave the 3D application and requires that the app be built for Xaml instead of D3D.

This is driving many developers to create their own Unity keyboard or purchase one from the Asset Store. I propose that a solid, basic, skinnable keyboard should be included in the Holo Toolkit.

[StephenHodgson]

Great idea, but a skinnable keyboard might be a little difficult. Depends on the implementation, with either uGUI or an alternative using geometry and text meshes.

IMO the application design should avoid the need for a virtual keyboard, if possible.

I think we should also make sure speech to text dictation is also a feature of this virtual keyboard as well.

[thebanjomatic]

Not sure if that documentation is just out of date, but this is from the Unity 5.5 release notes:

Windows Store: New implementation for TouchScreenKeyboard on UWP. XAML and D3D apps are now supported, as well as IME input. To turn on older implementation, pass the command line argument -forceTextBoxBasedKeyboard.

https://unity3d.com/unity/whats-new/unity-5.5.0

Or does that change not apply to HoloLens? (Forgive my ignorance, I've yet to have a need for text input in the things I've worked on).

[Edit] Either way a standard keyboard that works in the context of your application instead of having to pop out of the volumetric view would be useful.

[StephenHodgson]

Not sure. A quick search didn't produce any results though.

The unity documentation hints that it _could_ be used.

Interface into the native iPhone, Android, Windows Phone and Windows Store Apps on-screen keyboards - it is not available on other platforms.

[edit]
Here's the Microsoft documentation on the TouchScreenKeyboard usage in Unity but only specifies XAML

[cre8ivepark]

@jbienzms Thanks for the suggestion! Currently, we are looking into some of the options for this.

[MatrixInception]

Hello Everyone, on behalf of Matrix Inception team, I would like offer to donate our HoloLens D3D Keyboard to this great community. Currently, this is an asset available for sale in the Unity Asset Store (store link: http://u3d.as/KS8 and video link: https://youtu.be/83_oR3Oy4CA).

This D3D Keyboard came out of necessity in our own app development. We hope it will be useful to many other developers to come, and we welcome further enhancements.

Thank you Jared @jbienzms for making this request. Please let me know what you think and how to proceed.

-- Sky @ Matrix Inception

[cre8ivepark]

Thanks @MatrixInception for your contribution! We(Microsoft's HoloLens design team) are working on the keyboard prefab that has same design & interaction as system keyboard to provide consistent UX. It supports various modes including symbol mode and voice dictation. Currently we are doing final fit & finish. We will share the update on this thread.
Thanks!
Yoon

[MatrixInception]

Thanks for the update. I guess I was too late! I would suggest one thing though. As a user, I struggled with the layout of the existing OS keyboard on HoloLens, when entering a URL or password. I think you might want to optimize the layout a bit, by mimickng what most PC users are already accustomed to, but prioritize high frequency keys, such as , . /

[paseb]

Hi everyone,

I have a forked repository here: "https://github.com/paseb/HoloToolkit-Unity" that has the keyboard and associated test scene. I'm currently seeing a unity side bug with clicking on any InputField and getting a null reference spew in the log.

"NullReferenceException: Object reference not set to an instance of an object
UnityEngine.UI.InputField.ActivateInputFieldInternal () (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/InputField.cs:2146)
UnityEngine.UI.InputField.LateUpdate () (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/InputField.cs:698)"

Here's the class diagram:

I've created a new InputField that you can specify the type of keyboard you'd like to spawn in for the input field. The keyboard side should be pretty extensible with keyboards based on entry type.

public enum LayoutType{ Alpha, Symbol, URL, email }

I still have some clean up and bugs to resolve with the axis slider and initial placement but wanted to give an update.

thanks,
-pat

[zhao-zilong]

we really need this, really happy to see someone work on it!!!

[paseb]

There appears to be a bug in InputField targeting Windows Holographic. As I mentioned above here's the simple repro steps:

1. Add an InputField to the InputManagerTest scene
2. Click on it.
3. Notice the null references.

I thought it could be a carat issue but the carat is being generated and not null. Now looking into a possible system keyboard issue. If anyone has the time to help track this down, or have any insight into the issue that would be awesome!

[ignavus]

Is there any progress update or an ETA on when the keyboard would be finished and added to the HTK?

[firativerson]

@MatrixInception what happened to donation? =)

[MatrixInception]

@firativerson It appears that an official solution is just around the corner from Microsoft team, so we didn't proceed, to reduce confusion.

[Flanowski]

Hello, any updates on the process ? We also halted our virtual keyboard development since official is on it's way. :)

[konradsikorski]

@cre8ivepark how is it going? can you share some info about the progress?

[cre8ivepark]

@Flanowski @konradsikorski It looks like we are blocked by the bug that @paseb described above. @paseb Could we check with the latest HoloToolkit?

[paseb]

Hi everyone,

Sorry for the late reply I was out for the past 2 weeks but was able to solve the bug I mentioned above. In order to use any InputField with click events in Unity you need to have a Standalone Input Module active in the scene. Otherwise it blows up with the cryptic error above.

Everything is up and running and we have a couple minor bugs to fix before integrating:

1. Resolving a bug with the Carat placement and having it show up intermittently
2. Fix bug with entry slider for keyboard
3. Validate spatial placement in device
4. Provide option for enter to carrage returns rather than closing the keyboard like a submit

I've added @StephenHodgson as a contributor to get more eyes on it before committing. I'd encourage anyone to pull the keyboard repository and run the KeyboardTest scene under HoloToolkitUITest and provide any feedback you have on the implementation.

thanks,
-pat

[StephenHodgson]

I'm on vacation until next week, but I'll take a look as soon as I get a chance. Really excited to see this get added. Thanks for all the hard work Pat.

[konradsikorski]

@paseb Great news, I will check it on Monday :D

[jasonhbartlett]

Curious if there was any progress on this.

[paseb]

@jasonhbartlett, at the moment there are a couple lingering intermittent bugs I'm tracking down now that I have flashed a device to test against. After addressing those bugs I should be good to rebase and push what we have though I was hoping others would try it out and give me some feedback. We're in the middle of a production sprint and I haven't had the spare cycles to look into the issues. Sorry for the delay in getting this back into the HoloToolkit.

thanks,
-pat

[Flanowski]

Hello
It's been three weeks since last message which gave me sweet hope I can move on with my project ( after scraping own take on virtual keyboard when i stumbled across this thread ) are there any updates in this matter?

[paseb]

Hi @Flanowski,

The reason for the delay has been a bug with regards to native UI input with HoloLens in unity 5.6. There has been in intermittent bug that I've been tracking down for the past couple weeks regarding consistent input using the keyboard. There's an issue using HoloLens Input Module and input fields without StandAloneInput module you get exceptions in Unity code when selecting an input field.

I have that fixed and now have to just resolve carat and input placement within the slider before rebasing and requesting a pull. I should have it all updated by EOD today.

thanks,
-pat

[paseb]

Alright I have almost all the random issues tracked down accept for trying to mitigate the correct input module. So here's what I've run into with Unity Input modules:

1. Selecting an InputField without a StandAloneInputModule in your scene will give you exceptions (noted above) in the editor.

2. In order to get decent HoloLens input you have to us the HoloLens Input Module.

What I have in the keyboard test scene is both input modules on the Events gameobject. When testing in editor be aware that you can receive double clicks based on where the mouse is in the screen.

The keyboard test scene is under AssetsHoloToolkit-TestsInputScenes
The scene has a panel with two input fields by clicking on either one the keyboard will pop up. I'm running through the final tests then rebasing.

thanks,
-pat

[paseb]

Here are a couple other notes and things I ran into over the weekend:
There is this annoying bug - https://forums.hololens.com/discussion/6999/ui-button-on-click-event-firing-twice

If you're testing it in editor you need to enable or have an enabled StandAloneInputModule however when compiling for HoloLens you'll want to make sure to have HoloLensInputModule in the scene and disable the StandAloneInputModule.

The keyboard is setup to be modular and extended from just using the KeyboardKeyFunc and KeyboardValueKey.

Pull Request here: https://github.com/Microsoft/HoloToolkit-Unity/pull/765

[pnolen]

@paseb I have a fix for making the backspace key delete selected text. Should I wait until this is part of the normal holotoolkit-unity and then do a pull request there?

[paseb]

Hi @pnolen,

It's already been pulled and integrated so please grab the latest HoloToolkit-Unity and do a pull request there. Thanks for the contribution!

thanks,
-pat

[rachelrobynslater]

@StephenHodgson - looks like you're the right person to ask. Any idea how to get dictation working on the Keyboard prefab?

[StephenHodgson]

I'm not sure if it was implemented. I'll look into it.

[rachelrobynslater]

Haha quick! Thank you so much. :)) The little microphone key doesn't seem to do anything.

[StephenHodgson]

Looks like it is implemented, just make sure you've switched the build platform to WSA.

Looks like this should be refactored a little bit to make sure it's using the DictationInputManager.

[rachelrobynslater]

Sweet, thank you!

[reddo123]

Still getting a pile of null references as indicated above with latest relaease v1.2017.1.0 and Unity Unity 2017.1.0f3.
Also tried StandaloneInputModule while testing in Editor and HololensInputModule when compiling for Hololens with same results.
Thanks in advance for any updates on whether this is a reversion or whatnot.

[StephenHodgson]

HololensInputModule

You don't need the HoloLensInputModule for the toolkit's input. The StandardInputModule is all you need.

Still getting a pile of null references as indicated above

Where?

[reddo123]

Thanks for the info. Even using just StandaloneInputModule it was giving the same errors in the Unity Console.
The errors pointed to the Input Field and Input Field Slide objects of the keyboard.cs script somehow not being referenced.

NullReferencException: Object reference not set to an instance of an object
HoloToolkit.UI.Keyboard.Keyboard.Clear () (at Assets/HoloToolkit/UI/Scripts/Keyboard.cs:805)
NullReferenceException: Object reference not set to an instance of an object
HoloToolkit.UI.Keyboard.Keyboard.PresentKeyboard () (at Assets/HoloToolkit/UI/Scripts/Keyboard.cs:319)
NullReferenceException: Object reference not set to an instance of an object
...

The references to the InputField and InputFieldSlide were assigned as None in the Unity Inspector.
However, once manually assigning objects to InputField and InputFieldSlide, the errors ceased.

[StephenHodgson]

I'll take a look. Should probably be a separate issue opened for this.

[edwinckc]

Didn't find a separate issue for this, but wanted to elaborate a bit.

The documentation for Input Field says "if you are using the Keyboard prefab you can ignore this field as it will be already assigned." Before getting the latest release, the Keyboard prefab assigned the _search_ and _InputField_ objects to Input Field and Input Field Slide. After updating, both of these references went missing (_None_).

Edit: Sorry my bad, should have just created the new issue to being with for tracking as suggested.

[StephenHodgson]

No I meant to create a new issue please, so this can be tracked.

[tletie]

Does this work for Unity 5.6 instead of 2017?

[konradsikorski]

@tletie Yes this keyboard work for Unity 5.6 (we are using it in our project, not sure which version)

[The-MAZZTer]

The latest version of MRTK is designed for Unity 2017.2. If you want to run MRTK on earlier Unity, you have a few options:

1. Use the version released for Unity 5. This version may be too old to include the keyboard, but check it out first.
2. Use the version released for Unity 5 but manually pull in the keyboard files and manually fix any code issues relating to use of Unity 2017-exclusive APIs. I recommend doing this if your direction is taking too much effort.
3. Use the latest release and manually fix any code issues relating to use of Unity 2017-exclusive APIs
   (which it sounds like you are trying to do).

[Taghazog]

Hello Microsoft,

Probably a stupid question here but I couldn't find the answer as there is no clear documentation on that:
While the keyboard works pretty well in Unity with the mouse... how is it supposed to work in Hololens? How do we gaze at the keys, and how do we trigger the unity Button.onclick with the airtap...?

It seems nobody had the problem here, so I may have missed something pretty simple, but i don't know how to trigger the unity button.onclick on Hololens, where the keyboard should be useful (while i don't care in Unity Editor with the mouse)

Thank you!

[tletie]

@Taghazog did you have Hololens Input Module or Standard Input Module in the Event System of your scene?
Since you are able to trigger it in Unity, I assume you already have Input field that is using Keyboard Input Field to trigger the keyboard when focus is on this field.

[Taghazog]

Standard Input Module.

The thing is we do not use the Holotoolkit gesture/inputs system and i think their gaze/tap system triggers the input click event that works with Unity. (Our gaze doesn't even collide with the keyboard as we use our raycast system using colliders and there is no colliders with the Unity "Button" script)

[Augmentaio]

Hi All,
when I build for Hololens using XAML(!) the app crashes as soon as I focus on any (Keyboard)InputField.
Can you reproduce it? Any idea how to resolve this issue?
Thanks

[The-MAZZTer]

@Augmentaio You do not need to build as XAML to use the new keyboard, that was only needed for the old hack to use the built-in Hololens keyboard which you should no longer use (it's a very bad user experience the way it has to be implemented!). If you need to build as XAML for other reasons you should look at the output_log.txt file in your build's AppName_Data directory immediately after a crash. This should provide you with the error message related to the crash. If it's not enough to help you resolve the issue and it appears to be a problem with the latest versions of Unity and MRTK you can open a new bug report Issue with the full error details.

[HololensICAM]

Hi All,
Juste for those who want to know. I had a null reference object when i was trying to use the speech dictation (from the keyboard).
If you want to fix it create an empty object and then add the "dictation input manager" script to it. (You can find it in the holotoolkit)