Mixedrealitytoolkit-unity: Snapping cursors for easier focus selection

[provencher]

Describe the problem

Currently, when selecting objects with the far hand ray, or gaze pointer, small UI elements or focusables are difficult to target.

While it's true that making elements bigger makes them easier to target, there's a limit to the size of objects that make it possible to have presentable interfaces within the limited field of view.

Objects that are close to each other are also very difficult to select based purely on a raycast.

Describe the solution you'd like

It would be ideal to have some degree of snapping that accounts for hand jitter and lack of precision when not exactly within the bounds of a collider box.

To accomplish this, object bounds can be checked and when a cursor is near an object, the object whose bounds are closest to the cursor will get automatically selected, and stat selected even the cursor moves away from the object slightly.

This can also be done with some kind of dot product tolerance.

Describe alternatives you've considered

As I mentioned above, it is possible to make collider boxes bigger, but doing so requires a lot of manual work, and doesn't solve the issue when many objects are close to each other.

Additional context

Cursor snapping is used in the CAE VimedixAR app. I can provide a demo build to devs who require it.

[wiwei]

@provencher, the demo build would be really cool for us to explore

@julenka @cre8ivepark for UX planning and prototyping around this. This is likely something we'd want to prototype internally before we ship the feature out.

[provencher]

@wiwei I'll see about getting you a build to sideload.

[provencher]

Here is a bit of code from our snapping cursor
if (mOverCanvas) // cursor is on menu
{
// Raycast the canvas to see if the cursor is directly over a UI element.
mPointerEventData.position = PointerManager.CameraInput.WorldToScreenSpace(cursorPosition);

                mRaycastResults.Clear();
                mMenuRaycaster.Raycast(mPointerEventData, mRaycastResults);

                // pick a suitable result from the raycast hits
                foreach (RaycastResult result in mRaycastResults)
                {
                    FocusHandler focus = result.gameObject.GetComponentInParent<FocusHandler>();

                    if (focus == null || !focus.CanBeFocused()) continue;
                    newFocus = focus;
                }
            }

            if (newFocus == null) // the initial raycast didn't find a focusable element under the cursor so try snapping to a nearby element
            {
                // Get all focusable objects if not already found
                if (mFocusHandlers == null)
                {
                    FindFocusables();
                }

                float closestDistance = mFocusedObject == null ? mStartFocusDistance : mLoseFocusDistance;
                foreach (FocusHandler focusable in mFocusHandlers)
                {
                    if (focusable != null && focusable.CanBeFocused())
                    {
                        focusable.FocusRectTransform.GetWorldCorners(mTempWorldConers);

                        // Compute the distance from the cursor to the edge of the UI elements
                        for (int i = 0; i < 4; i++)
                        {
                            Vector3 vertex1 = mTempWorldConers[i];
                            Vector3 vertex2 = mTempWorldConers[(i + 1) % 4];
                            float distance = GeometryUtils.DistanceFromEdgetoPoint(vertex1, vertex2, cursorPosition);

                            // bias towards the currently focused element, stabilizing the gaze
                            if (mFocusedObject != null && focusable != mFocusedObject)
                            {
                                distance += mFocusBias;
                            }

                            if (distance < closestDistance)
                            {
                                closestDistance = distance;
                                newFocus = focusable;
                            }
                        }
                    }
                }
            }

[camnewnham]

This sounds pretty interesting - I was just pondering this earlier today when running into a similar issue trying to pull apart a small model/assembly with ManipulationHandlers! Some initial thoughts on a solution are:

• Perform a sphere cast while no object is focused, to give the pointer some tolerance
• Once an object is focused, it doesn't lose focus until the sphere cast no longer hits it, at which point other objects can be focused.

Might be an issue when you have more than two objects close together (i.e. how do you select the middle one when sweeping from left to right?). Would definitely help with small objects.

An extension to help selecting small objects in a cloud of other objects:

• If the center raycast (not spherecast) is continually hitting the same object and the spherecast-based focus is on another object, it switches focus after a dwell time.

This would also help when performing two handed operations (using hand-rays) to scale or rotate small objects - otherwise it's more or less impossible to get both cursors on at the right time!

Oh and lastly; in many use cases making the bounding boxes bigger isn't a viable solution, as in the example of a mechanical assembly mesh colliders need to be used (and oftentimes convex hulls aren't appropriate either).

[cre8ivepark]

This is great! Apps like Fragments had a good example of using the snapping cursor behavior on the menus. It makes the far interactions much easier.

[StephenHodgson]

Isn't this exactly what the ICursorModifier is for?