Mixedrealitytoolkit-unity: Cursor does not match shell when eye tracking is enabled

[julenka]

Describe the bug

The HandInteractionExamples scene aims to have behavior that matches the hololens 2 shell when used with the hololens 2 profile. However, when eye tracking is enabled (either in editor or on device), two issues occur:

1. The cursor in HandInteractionExamples follows the eyes instead of the head
2. When a hand is present, the eye cursor will apppear whenever a hand is near a grabbable or touchable object

To reproduce

1. Enable "simulate eye position" in MRTK input simulation profile
2. Run HandInteractionExample scene
3. Bring hand near grabbable
4. Notice that the cursor dot appears

Expected behavior

1. Cursor remains invisible and the cursor follows the head, not the eyes.
2. Eye tracking ray can be accessed via InputRayUtils.TryGetEyeRay
3. If the user wishes to have the cursor follow the gaze direction, he/she can enable this by having the gazeprovider use the eye gaze data, but by default it will use the head data.

Related to #6496

[keveleigh]

The cursor in HandInteractionExamples follows the eyes instead of the head

Is this incorrect? Since I don't think the shell ever has eyes enabled (or does it?), I'm not sure we have a well-defined "correct" behavior for handling eyes the way the shell does. Maybe @sostel has insight?

[julenka]

Is this incorrect? Since I don't think the shell ever has eyes enabled (or does it?), I'm not sure we have a well-defined "correct" behavior for handling eyes the way the shell does. Maybe @sostel has insight?

I think that having the default shell gaze cursor (the white donut / dot) follow the eyes is not recommended in our design guidelines, see "Eye Gaze is not the same as head gaze" here

I think the intention is to "Avoid attaching something directly to the user’s eye-gaze (e.g., a slider or cursor)" (source). Instead, use subtle feedback for eye tracking input.

[julenka]

@wiwei and @davidkline-ms I'd vote to make sure this makes it in for 2.3. Other than "consider for next iteration", is there some other way to track whether we want this in 2.3? For now I put it in the 2.3 milestone.

[davidkline-ms]

@julenka, Adding to the milestone is perfect. When we triage what is currently there, we will be sure to keep this one.

[sostel]

Is this incorrect? Since I don't think the shell ever has eyes enabled (or does it?), I'm not sure we have a well-defined "correct" behavior for handling eyes the way the shell does. Maybe @sostel has insight?

I think that having the default shell gaze cursor (the white donut / dot) follow the eyes is not recommended in our design guidelines, see "Eye Gaze is not the same as head gaze" here

I think the intention is to "Avoid attaching something directly to the user’s eye-gaze (e.g., a slider or cursor)" (source). Instead, use subtle feedback for eye tracking input.

+1 to what Julia said.

So, yes, not sure what you want me to do here. There is no equivalent to a Shell behavior for eye gaze. And using eye gaze input along with the head gaze cursor would be confusing. Is there a suggestion what you want me to do here?

[julenka]

Okay @sostel and @CDiaz-MS, I've updated the expected behavior as follows:

Expected behavior

1. By default, if eye tracking is enabled the cursor remains invisible and the cursor follows the head, not the eyes.
2. Eye tracking ray can be accessed via InputRayUtils.TryGetEyeRay
3. If the user wishes to have the cursor follow the gaze direction, he/she can enable this by having the gazeprovider use the eye gaze data, but by default it will use the head data.

@cre8ivepark @keveleigh, and @sostel please let me know if you have any comments. If there are not comments in a few days, I would assume that we can go ahead and try to implement this behavior for 2.3.

@sostel and @CDiaz-MS is this something you have bandwidth to work on for the 2.3 release? If not, we can ask @wiwei for help from others.

[julenka]

@wiwei It looks like in #4427 you changed the Gaze Provider to always enable eye gaze. I believe this had a few side effects which are likely not visible in many MRTK apps because they do not enable the eye tracking capability. However, with holographic remoting lit up, eye tracking is on by default and the issues are more obvious. The issues are:

1) When no hands are up, the white dot aka the "head cursor" that's supposed to follow the head actually follows the eyes. You can get rid of this by disabling "use eye tracking"

2) If you disable "use eye tracking", then it is not possible to get the eye gaze ray.

I believe the desired behavior as described in this bug is:

1) When no hands are up, the white dot still follows the head by default in all MRTK apps. If somebody wishes to instead have the cursor and thus input be driven from the eyes, they can choose to do that. One way to do this would be to have "use eye tracking" checkbox dictate that, and to be off by default. BUT still there should be a way to access the eye tracking via PointerUtils.GetEyeRay.

2) Even if "use eye tracking" on the gaze provider is disable, it should still be possible to get the eye gaze ray.

The impact of this is currently with unity remoting on, the head cursor follows the eyes and leads to various unexpected behaviors such as the cursor turning on if no hands are near grabbables. My current workaround is just to turn off "use eye gaze" in the gaze provider for whichever scene I'm working on. HOWEVER it would be great if the default behavior was to have "use eye tracking" disabled, but to still be able to access eye trackign data. Special demos that want to use the eyes to direct far interaction via gaze provider could enable this by checking "use eye gaze".

[wiwei]

Clearing the milestone, as we're getting close to 2.3 at this point and this feels like it could be riskier than I'd be comfortable doing.