Mixedrealitytoolkit-unity: Support custom raycasting implementations

[chbecker-ms]

(Ported to GitHub, reference 20077217)

Some of the MRW apps use "custom" raycasting for certain interactions. It does not appear that there is a way to provide a custom raycaster with the MRTK vNext input system. This poses a problem when trying to adopt MRTK vNext (at least the input system) for MRW.

One of the requirements in was that we override the default RayCast logic that HTK has, and while performing Raycasts we consider whether we hit a hologram, SR mesh or a hole in the SR mesh. If we have a hole in the SR mesh, we still generate a hit point for our annotations to get placed at. Monterrey input system had the ability to allow consuming apps to override the raycast logic. And so this was identified as a potential feature parity gap between the app's existing input system and HoloToolKit/MRTK.

[ryantrem]

@wiwei - these folks would be good contacts to validate any proposed solutions to this issue: @Kjakubzak, @rferrese, @ritijain

[wiwei]

@Kjakubzak, @rferrese, @ritijain do you guys have some more specific asks here (i.e. are there some specific points in the code that you can help point to that are inflexible, or have snippets of code that show the overall pain points and illustrate things?)

I want to get sufficient detail within this issue that we can both understand what it is we're trying to improve and then also understand how to validate that as well.

[ryantrem]

Also @smcbeth may have thoughts here.

[kircher1]

This is a feature request that would benefit the maps sdk project as well.

It would be great if the focus/gaze cursors and controller rays would follow the surface of the map; however, the map doesn't generate a collider on the fly (a lot of perf overhead for that), so it's not ray-castable through Unity's physics system.

[ritijain]

@wiwei This ask was due to multiple scenarios that were missing in HTK, and due to some custom solutions that our apps had. Some of the things have definitely been fixed in MRTK and having the ability to write custom Gaze/pointer systems because of availability of more interfaces definitely makes this easier.
These were the specific scenarios which led us to this ask -

1. The version of HTK that we were using only did Layer prioritization based comparison of hit results, and not distance based comparison, which is why we had to override/swap what HTK did. An ability to override the prioritization logic would help.
2. Sometimes depending on scenario, we might need to prioritize a distance based comparison of raycast hit result rather than a layer based prioritization. So i would like the ability to update raycast hit prioritization logic based on application scenario on the fly.

I guess the ask boils down to a customizable raycast result prioritization rather than the raycast itself. (i am looking at GetPrioritizedHitResult in https://github.com/microsoft/MixedRealityToolkit-Unity/blob/mrtk_release/Assets/MixedRealityToolkit.Services/InputSystem/FocusProvider.cs)

[Yoyozilla]

@julenka would love your input here as this may affect interaction, near/far based on distance.

This is an ask for support both layered and distance based raycasting. for example, if an app wants to prioritize UI even when UI is behind SR mesh.

[Yoyozilla]

Not blocking currently, may cause regression in some edge cases. We will look to fix before GA

[julenka]

I think this falls under our principle of "easy to customize" and should be considered for GA. We want to provide reliable defaults that are easy to customize. If we don't make it easy to customize now it will get harder to do in future since once GA is out it will be harder to make breaking changes.

[wiwei]

Note that another issue was filed which has the same request:

https://github.com/microsoft/MixedRealityToolkit-Unity/issues/4379

I duped that issue to this.