Webxr: Handling tracking loss

[huningxin]

Handling tracking loss is an important aspect of VR/MR app development.

In current version explainer, the tracking loss handling is only mentioned in Main Render Loop:

Unless the "headModel" VRFrameOfReference is being used, this function is not guaranteed to return a value. For example, the most common frame of reference, "eyeLevel", will fail to return a viewMatrix or a poseModelMatrix under tracking loss conditions. In that case, the page will need to decide how to respond. It may wish to re-render the scene using an older pose, fade the scene out to prevent disorientation, fall back to a "headModel" VRFrameOfReference, or simply not update. For more information on this see the Advanced functionality section.

There might be several potential improvements for tracking loss handling:

1. getDevicePose may fail even with "headModel"

Even for "headModel" VRFrameOfReference, it is still possible to experience "tracking loss".

For example, on OpenVR/Vive, if headset moves out of the tracking area, OpenVR will stops updating mDeviceToAbsoluteTracking of TrackedDevicePose_t, sets bPoseIsValid to false and eTrackingResult to TrackingResult_Calibrating_OutOfRange. It means no any pose data update in this case. Similar for WinMR, app needs to be prepared for SpatialLocatability.Unavailable for no pose data. So the sentence "_Unless the "headModel" VRFrameOfReference is being used, this function is not guaranteed to return a value._ " might need to be changed.

2. Add handling tracking loss section with sample

It would be better to add code samples of tracking loss handling for different tracking experiences.

For example, in current API design, developer should know how to use a "headModel" VRFrameOfReference as fallback when "eyeLevel" or "stage" VRFrameOfReference fails to return pose data. e.g.

let pose = vrFrame.getDevicePose(eyeLevelFrameOfRef);
if (pose) {
  // Head position and orientation available, render the seated-experience.
} else {
  // Fail to get pose in eye level frame of reference, try head model frame of reference.
  pose = vrFrame.getDevicePose(headModelFrameOfRef);
  if (pose) {
    // Fallback to render orientation-only experience.
  } else {
    // No tracking data is available
    // Fade out content and render the head-locked warning message.
  }
}

3. Add tracking loss event

When tracking loss happening, it might be useful to expose the reason to app. So app is able to guide user to recover. For outside-in tracking, the reason might be "out-of-range". For inside-out tracking, the reason might be "vision-inhibited". Or "unavailable" for other hardware errors.

partial interface VRSession : EventTarget {
  attribute EventHandler ontrackingloss;
};

enum VRTrackingLossReason {
  "out-of-range",
  "vision-inhibited",
  "unavailable"
};

[Constructor(DOMString type, VRTrackingLossEventInit eventInitDict)]
interface VRTrackingLossEvent : Event {
  readonly attribute VRTrackingLossReason reason;
};

dictionary VRTrackingLossEventInit : EventInit {
   required VRTrackingLossReason reason;
};

Would love to hear community's thoughts on this. Thanks!

[huningxin]

@toji , this is the issue we talked in today's implementers call.

@NellWaliczek , your summary in the call is accurate.

Please review and share your thoughts. Thanks!

[toji]

No issues with points 1 and 2.

I'm not sure if your third point should be implemented the way that's described. On the call @NellWaliczek seemed to think that the events would be fired on individual frame of references rather than the session (which makes sense, I think). I also imagine we'd want the inverse (VRTrackingAcquiredEvent) if we're going to add a loss event.

I also wonder how necessary an event is in this case? Specifically, what does having an event to signify tracking loss/acquisition enable that's not as easily handled by an error state on getDevicePose? (Whether that's returning null or something else.) Specifically, most of the APIs that I've interacted with don't have equivalent events for tracking loss, but instead fail the pose query somehow, so a browser implementation would just be watching for a failed query and firing an event at that point. That's something that the web application can do easily, especially since we expect apps to be querying poses per-frame.

I'll put together a pull request for the first two items soon, and in the meantime could @huningxin or @NellWaliczek (or anyone else who would like to chime in) expand on the need for a tracking loss event?

[huningxin]

No issues with points 1 and 2.

Are we able to implement points 1 and 2 for TAG review? Point 3 could be an enhancement.

For point 3:

Specifically, what does having an event to signify tracking loss/acquisition enable that's not as easily handled by an error state on getDevicePose?

An event might be useful to notify the low quality of tracking for app to prepare for a tracking loss besides the normal data channel.

(Whether that's returning null or something else.)

I agree that this can be done by returning some error states by getDevicePose. For example, for OpenVR, TrackingResult_Running_OutOfRange can be notified in pose.eTrackingResult while still return valid pose data. My initial purpose is to expose more tracking errors/warnings to enhance the app experience, surfacing through either an event or error status in VRDevicePose looks good to me.

[NellWaliczek]

As mentioned during the course of landing #409, we need to get a better understanding of the behavior when tracking is lost on the various supported tracking systems. Based on that, we can figure out if there is consistent behavior that WebXR can present to developers for each of the frame of reference types. This behavior, if it exists, will likely need to be communicated to developers when it is occurring. The answer may be an event, but we'd need to have a clear idea of what guidance we are giving developers about what they should or need to do in response. The same is true if we decide that simply failing a getDevicePose() call is sufficient.

This topic will be discussed at TPAC with the goal of getting closure on understanding the ecosystem. After TPAC, I'll write up the conclusions from the conversation in a PR for inclusion.

[cwilso]

@thetuvix I think this is waiting on a writeup from you?

[thetuvix]

Indeed, apologies for the delay! In writing out all the details of our TPAC discussion, I uncovered some nuances that I've been sounding out with folks here now that they're back from the holidays. I'll post the full writeup by end of week.

[thetuvix]

_Update:_ Draft sent around internally for review. I'll post the final writeup here by EOD Monday.

[thetuvix]

At TPAC, I talked with @lincolnfrog about the tracking loss and tracking recovery behaviors that I've observed in use by VR and AR apps, engines and middleware. These behaviors have all been built (at varying developer effort) on top of the various native platform APIs that exist today.

The key balance to strike here in WebXR is enabling apps and engines to easily build out the tracking loss/recovery semantics their scenarios need, while minimizing the policy choices that need to be injected back down into the API. If WebXR consumers find it easy to build whatever behavior they need with a minimum of stateful "modes" in the API, that to me is the ideal.

Pose behavior during tracking loss

For heads, controllers and anchors across the three top-level reference space types we've defined (stationary, bounded or unbounded), I've seen apps, engines and middleware implement one of three tracking loss behaviors, either to end users (for apps) or to their own app developers further up the stack (for engines/middleware):

1. Lose tracking (null pose): When tracking is lost, remove poses entirely from the layer above, so that the app stops rendering its world-locked scene (if head tracking is lost) or the tracked object (if controller/anchor tracking is lost). At the end of the day, this is the observable behavior that any given island of world-anchored content will likely want on an AR device, as it's often better to just stop rendering that content if you know it will be dragged out of alignment with the real world.
   
   
   
   • While a top level AR app may choose to fully blank out their scene during tracking loss, this is often a painful API behavior for engine/middleware APIs to expose further up the stack for head tracking within a reference space. Specifically, if a platform's primary reference space exhibits this null pose behavior during tracking loss, it requires an app that wants to maintain continuity for dynamic tag-along content to do lots of math in advance to correlate their primary reference space with some backup orientation-only reference space, so that it's ready in advance to switch over whenever the primary reference space loses tracking.
   
   
   • The Windows MR APIs had originally taken this AR-focused approach for their equivalent of the "bounded" and "unbounded" spaces, and it's been fairly painful for engines to implement the VR tracking loss behavior they need below (option 2) on top of that API behavior. I recommend that the WebXR API avoid this approach :smile:
   
   
   • The one case where a null pose may be the least bad option is when tracking of a given object has never been established yet. In this situation, there is no "last known position" to report, and so all we could report is identity, which would render the scene or tracked object at a completely arbitrary location. In that case, just returning a fully-null pose expresses the truth about that tracked object, although it does risk apps hitting a less-tested corner case that may only manifest on some platforms.
   
   
   • While the null pose approach is limiting for head tracking and controller tracking, dragging anchors off their mark during tracking loss is almost certainly undesirable. The null pose approach feels best for any getPose call that involves an anchor, even if we pursue the "freeze last position" option below for heads and controllers.
   
   
2. Keep tracking orientation, freezing the last position: This gives naïve VR apps continuity of a tracked object's position and orientation by default during loss of 6DoF tracking, as the camera's position will simply stay put in the engine's world space, while its orientation keeps updating. No special correlation is required to align the "tracking lost" orientations with the "tracking nominal" orientations the app was using just prior to the loss. This gives VR apps a reasonable default behavior, while we can still provide an emulatedPosition bool to allow apps to then optionally detect the tracking loss state and add any extra handling on top.
   
   
   
   • This is the default scene camera behavior that engines like Unity often expose to VR app scripts, where dragging the world with the user for a bit is often the lesser evil as compared to an abrupt blackout of the world. That world-dragging behavior is often more acceptable in VR than AR, since there is no visible physical world that your virtual objects will tear away from. Of course, it's obviously best to minimize the time spent in tracking loss, to avoid user discomfort.
   
   
   • To me, this approach feels like the best "pit of success" for WebXR head tracking, giving naïve apps continuity for both position and orientation, while still allowing apps that want to do more to check a bool to know when the position is no longer tracked. This provides a common base upon which WebXR-based middleware can trivially and reliably build out behavior 1, 2 or 3 to app code, depending on their specific needs.
   
   
   • For controllers tracked by inside-out headsets, there is often a smooth transition from a visually-observed controller to one that is tracked with inertial dead reckoning and eventually a frozen position, as it leaves the view of the headset's cameras. Naïve apps that just want to point rays at distant objects will likely benefit from a controller target ray that the user can continue to rotate, even if the position is eventually frozen until the user looks down at their hand. This approach feels best for controllers that have lost positional tracking for that reason, with the emulatedPosition bool to tell apps when a controller is in this state.
   
   
3. Keep delivering orientation, with a null position: This still gives apps continuity of orientation data during loss of 6DoF tracking, with the app simply keeping the tracked object in its previous position during tracking loss.
   
   
   
   • This approach still avoids most of the heavy lifting required for apps to handle tracking loss, as they don't need to correlate a tracking and no-tracking reference space. Any bookkeeping for engines to do if they want to freeze the last position is trivial: just detect that position is not available and update only the orientation of the tracked camera or other object.
   
   
   • However, delivering an actual null position prevents 6DoF headsets from incorporating a neck model during tracking loss. If a given layer reports null for position, it's given up the ability to express a position that shifts slightly as the user tilts their head. This would make a 6DoF headset's tracking loss state less capable than a 3DoF headset's neck model steady state.
   
   
   • Even for 3DoF controllers, there is often a canonical implied source for the pointing ray, such as placing the controller at the user's hip. Therefore, even 3DoF controllers are likely to have at least an emulated position.
   
   
   • Unless we expect WebXR to represent tracked objects that expose a world-aligned orientation but never any position whatsoever, this approach may always be strictly less expressive than option 2.
   
   

Given the above, here's my strawman for how we should spec WebXR implementations to behave for various tracked entities under different tracking states, enabling middleware to itself expose any of the models above without the need to choose a modal state:

| Entity being located /
tracking state | Is there a meaningful position?
getPose(…) != null | Is the position emulated?
pose.emulatedPosition |
| ------------- | ------------- | ------------- |
| 3DoF head
orientation tracked (neck model) | true | true |
| 6DoF head
never tracked yet (no aligned orientation) | false | |
| 6DoF head
tracking lost (neck model) | true | true |
| 6DoF head
nominal tracking | true | false |
| 3DoF controller
orientation tracked (shoot from the hip) | true | true |
| 6DoF controller
never tracked yet (no aligned orientation) | false | |
| 6DoF controller
tracking lost (shoot from the hip) | true | true |
| 6DoF controller
nominal tracking | true | false |
| Anchor
down dark hallway or far away | false | |
| Anchor
nominal tracking | true | false |

With this approach, apps that don't want the "freeze last head position" behavior can easily implement either alternate behavior without having to set a mode in the WebXR API:

• If middleware wishes to expose a null position to script code during tracking loss, it can simply observe that the head pose's emulatedPosition is true for this frame, and return null to the next layer up.

• If an app wishes to blank out the screen or show a fallback 3DoF scene during tracking loss, it can simply observe that the head pose's emulatedPosition is true for this frame, and take whatever fallback action it wishes.

World origin behavior during head tracking recovery

While the choices above describe how components express the transition from tracking-nominal to tracking-lost, there is then a separate choice around how an app should adjust its world origin when head tracking is recovered again. While the obvious solution for most tracked objects is to just jump the object back to its "true" position relative to the current origin, for VR head tracking that's generally more jarring than just absorbing the drift, especially if there is an arbitrary teleportation offset being applied to the reference space anyway.

Here are the three world origin tracking recovery behaviors I've seen in apps/engines across stationary, bounded and unbounded spaces:

1. Snap the camera and other objects back into position (keep world origin in place): This is the most "honest" approach, keeping the world origin in place and positioning the newly tracked camera and other objects back where they belong relative to that origin.
   
   
   
   • This is often the desired behavior for stationary VR experiences (e.g. where a cockpit may be centered around the user's chair) and for bounded VR experiences with no teleporting, where the origin is fixed and meaningful relative to the virtual world.
   
   
   • The key gotcha here is that jumping the camera back into position means that any world content positioned directly in the reference space (as is common in bounded VR experiences) will suddenly jump for the user as tracking is recovered. Engines and apps will need to be careful to ignore any physics impulses that would be implied by the camera's jump.
   
   
   • It is obviously only possible to keep the world origin "in position" if tracking recovers in a tracking volume that is connected to the original tracking origin. If the user has instead walked down a dark hallway and recovered tracking in a new room, this approach is undefined, as the runtime simply does not know how the current device position relates to the previous origin. At minimum, one of the approaches below will be necessary for tracking recovery of an unbounded reference space when recovering into a disjoint tracking volume.
   
   
2. Keep camera and nearby objects at current world coordinates (adjust world origin to absorb discontinuity): This approach optimizes world coordinates for stability near the user/camera during tracking recovery.
   
   
   
   • This is often the desired behavior for bounded VR scenarios when there is a teleportation offset in place anyway. Often apps that desire this recovery behavior have continued to render during tracking loss, pulling the world along with the user (the "freeze last known position" tracking loss option above). Once tracking is recovered, it would further confuse or discomfort the user to then jump the camera back to respect the original arbitrary teleportation offset. Instead, the least jarring solution is for the app/engine to just absorb the discontinuity into the camera's teleportation offset. From the user's perspective, they remain where they are in the virtual world when tracking recovers, with the world origin having translated by the amount the device drifted during tracking loss.
   
   
   • This approach also provides continuity for an unbounded reference space across tracking loss and recovery. Regardless of whether tracking is recovered in the same tracking fragment or a disjoint tracking fragment, the origin can be adjusted to keep the scene camera at the same world coordinates that it was frozen to during tracking loss. This provides visual continuity to the user for any dynamic tag-along content that is "loose" in the engine's world coordinates across all three tracking loss phases: 1) tracking in fragment A, 2) tracking is lost, 3) tracking in fragment A or fragment B.
   
   
   • For floor-level reference spaces such as the bounded space, it may be necessary to adjust the origin only along the XZ axes.
   
   
3. Reset world origin to current camera position (jump world origin so camera is at origin): This approach resets the world origin when tracking is recovered so that the head's position is once again (0, 0, 0).
   
   
   
   • If all content in the world is attached to anchors, and there are no physics impulses to worry about, the app could just carry on with the world origin reset arbitrarily to the current camera position upon tracking recovery, starting the camera's world pose at zero again. However, there is often content placed directly in world coordinates (especially with stationary or bounded reference spaces), and this approach would be quite jarring for such apps.
   
   
   • This could be a possible approach for an unbounded space during tracking recovery into a new disjoint tracking volume, although it still introduces an unnecessary instantaneous change in position for tagalong content that had frozen in place during tracking loss, which the app would then need to reposition and ignore for physics.
   
   

Given the above, here's my strawman for how WebXR reference spaces should handle their origin (and thus head poses) when head tracking is recovered:

| Reference space | WebXR reference space origin behavior upon head tracking recovery |
| ------------- | ------------- |
| stationary | Snap back into position for current origin (option 1) |
| bounded | Open design question: (perhaps an option when creating the bounded reference space?)

Snap back into position for current origin (option 1) or
Adjust origin to keep head in position (option 2)

For option 1, an app that wishes to absorb the drift into a teleportation offset instead (option 2) can observe the jump in head pose since the previous frame and adjust the bounded reference space's originOffset by that amount. Open design question: Would we then provide some viewerReset event to directly provide the viewer's pose discontinuity?

For option 2, a reset event would then fire for the bounded reference space, with transform representing the origin's discontinuity |
| unbounded

(recovery into same tracking fragment) | Adjust origin to keep head in position (option 2)

A reset event will then fire for the unbounded reference space, with transform representing the origin's discontinuity |
| unbounded

(recovery into different tracking fragment) | Adjust origin to keep head in position (option 2)

A reset event will then fire for the unbounded reference space, although we'd need to provide a null transform, as the UA does not know the relationship from the previous origin to the new origin |

Next steps

The tracking loss and tracking recovery tables above represent my own strawman proposals for the following key questions:

• For each tracking loss state listed above: (across 3DoF head, 6DoF head, 3DoF controllers, 6DoF controllers and anchors)
  
  
  
  • When does .getPose return null?
  
  
  • When is .emulatedPosition true?
  
  
• For each reference space:
  
  
  
  • How should the origin (and thus head poses) behave upon head tracking recovery?
  
  

There will certainly be much bikeshedding across each of the rows above :smile: However, what's more critical than any particular choice is that we do come to a consensus decision for each of these rows. For WebXR content to successfully span across the world's VR and AR devices, it's critical that we spec a common expectation for how all WebXR implementations "should" behave in each of these tracking loss and tracking recovery situations, across all XR platforms.

Let's drive this to consensus at the F2F!

[NellWaliczek]

Thanks @thetuvix for putting all this information together! What a great summary of a very complicated problem space.

One question for you.... Do you see this behavior as needing to be different for stationary and bounded reference spaces created on devices that have inside out tracking?

[thetuvix]

Do you see this behavior as needing to be different for stationary and bounded reference spaces created on devices that have inside out tracking?

The general design intention with our reference spaces is that the app selects its desired space and then that space behaves equivalently across devices. A key thing we should nail down is whether we see anything in the definition of our spaces that would lead to divergence across devices. If so, we should aim to tweak those definitions so that our UAs and devices can align.

[NellWaliczek]

@thetuvix Thanks for driving this at the Jan 19 F2F! Do you have an idea when you'll have the PR put together that covers what we all agreed on?

[thetuvix]

I’m aiming to write up a PR here this week.

[thetuvix]

Apologies for the delay! Initial tracking loss coverage for the spatial tracking explainer is in #559 - I'll be adding to this throughout the week.