Mixedrealitytoolkit-unity: vNext - Gaze cursor/Air tap issues

[cre8ivepark]

Overview

Gaze cursor/Air tap is not working properly on HoloLens.

• Head tracking stops working when a hand is raised and lost. Cursor does not follow head movement anymore. It follows head movement only when a hand is raised.
• Does not show interaction state changes (dot <-> dount)
• Does not trigger input events.
• No in-editor gaze cursor simulation yet?

Expected Behavior

Gaze cursor works on both HoloLens and editor

Actual Behavior

Gaze cursor does not work

Unity Editor Version

2018.1.9f

Mixed Reality Toolkit Release Version

Alpha

[StephenHodgson]

We don't have gesture support yet so this makes sense.

[StephenHodgson]

No in-editor gaze cursor simulation yet?

None yet. I think we should have a good discussion about hand simulation vs normal mouse input and what sort of options could go into that.

[keveleigh]

We don't have gesture support yet so this makes sense.

I don't think that explains the gaze cursor issue though. It stops moving when the hand is detected and then lost.

[keveleigh]

@StephenHodgson Is there a reason controller pointers are global listeners and then have to check against their input source to see if they should handle the event? It seems more straight forward (and probably saves perf, by reducing the number of global listeners) for the controller, which knows about all its pointers, to tell them that they're pressed. The current design causes an issue with the gaze pointer, as it's not a MonoBehaviour and thus doesn't have a GameObject to be a global listener.

[StephenHodgson]

probably saves perf, by reducing the number of global listeners)

Not sure how it would impact perf. That's the reason why we use the system the way we do.

The current design causes an issue with the gaze pointer, as it's not a MonoBehaviour and thus doesn't have a GameObject to be a global listener.

You need the GazeProvider, which is where you should be listening to events. The most of the pointers are intentionally not MonoBehaviours.

[keveleigh]

The most of the pointers are intentionally not MonoBehaviours.

Right, and I'm not saying they should be. I also don't think they need to be listening to all global events, when they only care about the events actually meant for them. Why do they need to listen to all global events?

Not sure how it would impact perf.

The system scales to any number of global listeners without a perf impact?

[keveleigh]

You need the GazeProvider, which is where you should be listening to events.

Which still will need a way to tell the pointer about the event.

I'm proposing that no pointers should listen for global input events, but should be explicitly told by their controllers that they are pressed/released. This removes the need to filter out events.

[StephenHodgson]

The system scales to any number of global listeners without a perf impact?

As far as I can tell. That's one of the main args for using it vs other input systems I've seen. Our events aren't _really_ events. We're calling the method directly via the interface, so it's about as fast as it could possibly be.

[StephenHodgson]

I'm proposing that no pointers should listen for global input events, but should be explicitly told by their controllers that they are pressed/released. This removes the need to filter out events.

I agree, as long as we have a controller reference. But that's the thing. Not all pointers have a controller reference.

[keveleigh]

Not all pointers have a controller reference.

Right. I suppose I'm more referring to InputSources then, as they're where the lists of pointers actually exist. Regardless, whatever the source of the input event (which is sometimes Controllers, sometimes InputSources) should have a way of telling its pointers about the action.

[StephenHodgson]

InputSources aren't always MonoBehaviours either.

[keveleigh]

InputSources aren't always MonoBehaviours either.

Okay, why do they have to be? Not sure why that was mentioned.

[StephenHodgson]

Okay, why do they have to be? Not sure why that was mentioned.

Yes, they intentionally don't inherit from Mono.

[krishnan-unni]

Hi @StephenHodgson , is there a plan for Hololens Gesture features (tap, hold etc) to be implemented in the near time? I couldn't find any hololens related input system support in the code currently.

[StephenHodgson]

This work has already been completed

[krishnan-unni]

Thank you. Could you please point to the story\merge id? Also is it in the mrtk-dev branch? Or release?

[StephenHodgson]

https://github.com/Microsoft/MixedRealityToolkit-Unity/pull/2637, https://github.com/Microsoft/MixedRealityToolkit-Unity/pull/2633, https://github.com/Microsoft/MixedRealityToolkit-Unity/pull/2632, and https://github.com/Microsoft/MixedRealityToolkit-Unity/pull/2594

[krishnan-unni]

@StephenHodgson I have a doubt. I checked out the latest dev_branch and found the hololens specific code in WindowsMixedRealityDeviceManager. But no matter how much I looked for I couldn't find any gesture handling for a Tap gesture - only Hold, Manipilation and Navigation. Also I see that all the Select gestures (Pointer clicks in general) are trying to be mapped as one input through the input system but I dont see where\how a Tap gesture is mapped to it.

Am I missing something here. Couldn't find much documents and Im combing through the whole code to understand the whole system overall :)

[StephenHodgson]

Tap is handed by the interaction manager via Select.

[StephenHodgson]

Assigning the tap (select) action in the controller mapping profile should result in that action being handled in your app when performing a tap

[keveleigh]

@krishnan-unni OnPointerClicked from IMixedRealityPointerHandler should expose the Tap event for you. We don't yet have it coming from the platform GestureManager, but the click event (plus checking to make sure the input action matches the expected select) there will mostly match the expected tap.

[StephenHodgson]

The input up/down and pressed events should be raised when tapping as well via IMixedRealityInputHandler

[krishnan-unni]

Found it thank you. 👍
@StephenHodgson Im still trying to figure out all the wirings between the profiles and the handlers. As for the Controller Mapping. I selected Hololens and It showed me the mapping for Tap, Hold etc. But all given as None. Tried creating few actions in the Input profile. That didnt show up there. I'll have to dig more there.
@keveleigh Thank you Kurtis, yes, I saw the PointerClickHandler implementing it. That brings me to the next question. Are the Handlers that are created are for a default implementation for guideline purpose? The Legacy toolkit lets the user create the implementations (I know we can do it here too)

[StephenHodgson]

Hmm 🤔 you should definitely see the input actions get updated appropriately. I'll look into this tomorrow to make sure the actions are getting updated. Also make sure the profile you're adding actions to is assigned to the mixed reality manager's input system.

[krishnan-unni]

@StephenHodgson actually I was checking that, and yes I have all the profiles created using the Create Context menu since the standard one doesnt allow editing anymore. So I created all the profiles and assigned them. The actions are showing up fine in the Speech profile just fine but not for the Controller mapping

[StephenHodgson]

And yes the handlers are essentially contracts stating you'll be receiving specific kinds of events and the type of data those events will have. From those interfaces you can define your implementation of your UX or consume the input as you see fit.

[krishnan-unni]

Once I add Controller mapping and select Hololens from it another window pops up and WMR Input Action Assignment with three options - Spatial Pointer, Spatial Grip and Air Tap. None is the only drop down option there.

[StephenHodgson]

The actions are showing up fine in the Speech profile just fine but not for the Controller mapping.

Please open a bug report and I'll check this out tomorrow. Be sure to include screen shots of the configuration, input, action, and controller mapping profiles.

The pop out windows are supposed to update the action list based on the currently active profile and it's children.

[StephenHodgson]

Did you assign an axis constraint on the actions?

[krishnan-unni]

No I did not assign axis constraints

[krishnan-unni]

Here is the link https://github.com/Microsoft/MixedRealityToolkit-Unity/issues/2901 I have screenshots there.

[StephenHodgson]

Yeah you'll want to assign digital axis constraints for inputs that have binary ( on/off ) types of input

[krishnan-unni]

Dummy me. Yeah I tried and its showing up now. Digital Action only shows up for Air Tap and not for the Spatial ones. There only those with DoF shows up. I'll add this comment there and close the bug (without the dummy part).

[StephenHodgson]

Naw, I'm the dummy for not making that more obvious :p

[krishnan-unni]

:) I'm pretty sure my next article is about Getting started with MRTK vNext. There is a lot to figure out for a new user or maybe because I'm looking into the details.

So another question - How does the actions newly defined in the profiles relate to the actions defined in the Handler interfaces? Is it like the toolkit is providing me two ways to hook into input\gesture\speech events -

Let me figure that out :)

[StephenHodgson]

How does the actions newly defined in the profiles relate to the actions defined in the Handler interfaces?

They're exactly the same. Action's are the abstracted concept we're approaching the system with. You listen for an actions in your handler that you set up in each of the other profiles.

That way you don't have to listen for platform specific code. The handler's only need to know that they've received data, and how to act on that data.

[krishnan-unni]

Found it :) Thanks for the info. Though Speech Handler looks like an exception to what you described. The Handler relies on the Keyword than the action to hook into. The Action details are getting lost (not used or propagated to the next level) as soon as it enters the Handler. The Speech Handler Inspector is using the Keyword than the Action to populate the KeyWordAndResponse dictionary. It works though I feel like it breaks from the design. Pointer Click handler relies on action itself as does Gesture Handler. So I don't know if it is a bug with Speech Handler or it was per design due to reasons unknown to me. Speech Handler also has a bug which I'll report it though a issue so that it can be tracked.

[StephenHodgson]

Thanks for the write up!

We don't use the actions for the speech handler as an action isn't the keywords we may want to listen for. It's a bit different than the others, that that reason, but if you think there's a better way I'm all ears :)

[krishnan-unni]

Well nothing wrong per say. As I said it works just fine. Though when I took a user\consumer perspective it looked odd to me. Since I already setup a Speech profile with key words and actions against it. So along with other input handlers when I configured Speech I was surprised to see keywords there than actions as I am handling app against user actions. My be this is an individual perspective, but for me Speech is just another input action from the user which represent an action that an app needs to react to. So when Speech alone diverted from the whole approach as a user it made me think a bit like "where are my actions which I defined against speech keywords?"

[krishnan-unni]

One more thing I would like to add as a proposal if you agree, we might have to document on how to configure the SpeechDeviceManager as an Additional Component in the Configuration Profile. And it is mandatory for the speech system to work. Thankfully Unity threw the error that the Speech Input Handler needs a IMixedRealitySpeechSystem.

[StephenHodgson]

I think that the whole component system needs documentation 😅

I need to get on that. So little time and so much do to. I'm sorry for dropping the ball there. Going forward we've put in place documentation first and then development second for new features.

[krishnan-unni]

Thats cool. It ll speed up the adoption rate among the general community and help find the defects and improve it as a whole

[StephenHodgson]

Definitely agree.

Still in beta, and I don't have plans to make any new features anytime soon. By the time we release I plan to have everything fully documented.

[krishnan-unni]

Cool. From my end , I am compiling your answers and I'll publish it so that people can get started with basic functionalities in their app (like setting up Profiles, Focus, Gaze, Select, Gesture, Speech ).

[StephenHodgson]

Much appreciated :) An outside perspective is good as well to make sure we're on the right path.

[StephenHodgson]

For the most part the default profiles should just support all this basic functionality.

Ideally I'm interested in having an easy setup and people how want to learn more about specific features can learn at their own pace.

Designing a system that is super easy to setup but deeply customizable (from a designer and Developer point of view) is hard 😅

[krishnan-unni]

:) Completely agree. The trick is to find a balance between Ease of setup vs Customization Support.
As for the Gesture support, is the feature complete? I see InteractionReceiver as the last end of the even chain, which is an abstract class. No Configurable Handlers like Speech or Pointer. Is this intended or the rest of the pieces are in the works? If it is intended I expect whoever wants to handle the gestures need to derive from Interaction receivers?

[StephenHodgson]

I'm honestly not sure as I didn't develop that part.

I'm not sure something like it is needed now that we've abstracted out the input into actions. That that's a conversation that belongs in another thread.

[krishnan-unni]

@StephenHodgson let me know if you want me to open new Issue for these kind of questions than adding these as comments.

[StephenHodgson]

The intended way to consume and use input is via the handler interfaces.

[krishnan-unni]

Well thats the problem. I dont see anyone handling the Gesture events from the WMRDeviceManager other than Interaction Receiver which is an abstract. The same with Basic Focus Handler - which is also an abstract class which means you need to derive from it to get the Focus Enter Exit events.

[StephenHodgson]

Remember these are also optional SDK components.

I need to write up an in depth example for each handler.

[krishnan-unni]

Yup, I see some other Utilities like DragandDrop which handle the Focus events. But, nothing that acts like a SpeechInput Handler or a PointerClick Handler. Its not difficult though, all works fine for handling events if you inherit these abstract classes and implement your handling logic (like the legacy holotoolkit interfaces). Since these are not like the Speech and Pointer handlers, I felt like the feature might still be in development. I dont see anything wrong yet, unless it is decided that these classes needs to be derived by the user every time they want to handle gesture\focus events. Then, it would be a departure from the design of Pointer click and Speech handlers.

[krishnan-unni]

One question related to Gestures, The Gesture profile supports adding as many actions as we want but must belong to 1 of the three Gesture types (Hold, Manup, Nav) unless we modify the GestureInputType enum to add more types. But the WMRDeviceManager takes only one action per gesture type. It overwrites the previous actions of the same Gesturetype defined in the profile while going through the loop. Is it intended, owing to expand GestureInputType in the future?

[StephenHodgson]

Please open an issue for the gesture issue you outlined above.

[krishnan-unni]

Done 2912. I have prefixed it as a question though. I'll add another one for InteractionReceiver (Gesture Handler) too