Webxr: Revisit transient input sources & XRInputSource.targetRaySpace

[bialpio]

It seems to me that the description of XRInputSources currently do not cover well the cases of XRInputSources that do not constantly produce positional data. The current approach to such input sources is to expose them only for the duration of their primary action (see transient input section in the spec).

I think that the existence of transient input sources is not really intuitive - to me there's a significant difference between "there is no input source" vs "input source exists but doesn't currently hand out data". For example, input sources corresponding to controllers might have to also be considered transient (I imagine that controller tracking can be occasionally lost), even though the input source could technically exists for as long as the controller hardware is connected.

In addition to the above problem, it feels like trying to fit the model of input sources that do not continuously hand out positional data into the API that assumes that input source always has positional data (non-nullable XRInputSource.targetRaySpace) is causing workarounds in the proposed hit test API - see automatic hit test creation that seems to exists solely to work around the transient input sources.

I think that we'd be able to solve the above issues by making XRInputSource.targetRaySpace nullable - this way, UAs could create a more persistent input source even for devices that do not constantly provide positional data (or in case there's tracking loss). I think it could be safe to guarantee that XRInputSource.targetRaySpace will be non-null during the input source's primary action, but otherwise, the application would have to check for its presence (for example in rAF callback). The UAs will still have to notify the application about hardware arrival/removal via XRSession.inputsourceschange, but it will be only used to signal input source availability, and not the the state of the input source itself.

What does the community think about that? Where there any other reasons for transient input sources (other than input sources sometimes don't have positional data) that I'm unaware of?

[bialpio]

/agenda I'd like to hear what others think fairly soon as this might cause a breaking change to the spec.

[Manishearth]

I think that we'd be able to solve the above issues by making XRInputSource.targetRaySpace nullable

We'd also need to specify that these spaces are not long-lived and instead tied to a frame. Currently you are allowed to hold on to XRSpaces.

OTOH we can just make getPose() return null for these spaces when necessary.

---

I am a bit unhappy with the status quo handling of transient input sources as well, I think it would be nice to have them handled in a more persistent fashion.

[bialpio]

OTOH we can just make getPose() return null for these spaces when necessary.

That's a good point - it might actually be the preferred way to signal that the input source's target ray space is not currently tracked / has unknown relationship to other XRSpaces. Just because an app is always able to obtain XRSpace from XRInputSource does not mean that it'll always be able to obtain poses out of it. This'll allow us to keep the target ray space non-nullable on input source & allow UAs to create input sources in a more persistent manner if they choose to do so.

[rcabanier]

I think that we'd be able to solve the above issues by making XRInputSource.targetRaySpace nullable

We'd also need to specify that these spaces are not long-lived and instead tied to a frame. Currently you are allowed to hold on to XRSpaces.

Can you give an example of a space that is tied to a frame?

OTOH we can just make getPose() return null for these spaces when necessary.

The spec already gives the UA an option for this:

if space’s pose relative to baseSpace has never been determined:
Set pose to null.

[Manishearth]

Can you give an example of a space that is tied to a frame?

There aren't any, I'm saying we would have to introduce such a concept if we make TRS nullable.

[rcabanier]

Can you give an example of a space that is tied to a frame?

There aren't any, I'm saying we would have to introduce such a concept if we make TRS nullable.

I meant: what would make a space tied to a frame? Is this an ARCore limitation?

We should avoid introducing APIs that only work or provide information during a RequestAnimationFrame.

[Manishearth]

I meant: what would make a space tied to a frame? Is this an ARCore limitation?

If targetRaySpace is nullable then we should prevent these spaces being used against frames when they are null, basically.

We should avoid introducing APIs that only work or provide information during a RequestAnimationFrame.

It doesn't have to be animation frames, it can be input source frames too.

But I agree, this is why I suggested making getPose for these spaces return null instead.

[bialpio]

But I agree, this is why I suggested making getPose for these spaces return null instead.

I also agree, we don't need to modify the spec about the nullability of XRInputSource.targetRaySpace to signal that the input source position is unknown.

I'd still love to get a bit more context about the existence of transient input sources - the fact that UAs can communicate that the input source's pose is unknown suggests to me that they can be implemented by the UAs in a persistent manner and introducing the concept of transient input source might be unnecessary. @NellWaliczek, @toji - I think you have more background on transient input sources - let me know if I'm missing anything here!

[Manishearth]

So the reason transient input sources exist is basically because for some kinds of input you can't know if an input source exists beforehand. In this case, screenspace touch input in a multitouch environment. You can't tell which finger is which, all you know is that there are n distinct fingers touching the screen at the moment. Once they're lifted, you can't tell if a further touch event is from the same or different finger.

So transient input sources are supposed to be used specifically when we don't have any ability to reason about the identity of that input source after the input event completes. OTOH, if a controller loses tracking you can still usually tell that it's the same controller when you get it back.

[Manishearth]

The case of "can no longer track targetRaySpace" should be handled with getPose() returning null on the space.

[bialpio]

So the reason transient input sources exist is basically because for some kinds of input you can't know if an input source exists beforehand. In this case, screenspace touch input in a multitouch environment.

I'd assume that it's possible to query the device beforehand on how many touch points it supports & pre-create the input sources for each of them. If it isn't, then it's still probably safe to assume that the device doesn't suddenly lose the capability, so UAs do not have to remove the XRInputSource as soon as the touch input is gone (for example, there might be some delay of inactivity after which the input source can be removed). Other cases where the input source's existence can't be determined beforehand could probably be handled in a similar way (raise input source added event, but don't remove it).

So transient input sources are supposed to be used specifically when we don't have any ability to reason about the identity of that input source after the input event completes.

In my opinion, we can still reason about the identity of input sources for touch screen input - it's the touch points that we can keep track of, not the fingers.

[Manishearth]

I'd assume that it's possible to query the device beforehand on how many touch points it supports & pre-create the input sources for each of them

We also want to avoid people associating identity with the input sources, presumably. We don't want people to attach hit tests to only the "first" touch point, for example.

(also do devices typically only support a limited number of touch points? i've seen apps like chwazi work with a ton of touch points)

In my opinion, we can still reason about the identity of input sources for touch screen input - it's the touch points that we can keep track of

Right, but touch points are transient.

[bialpio]

We also want to avoid people associating identity with the input sources, presumably. We don't want people to attach hit tests to only the "first" touch point, for example.

They can still subscribe to all of the touch points (even as they are added), and this is how they'd have to behave with the current hit test proposal. Why don't we want to allow them to subscribe to only the first touch point? Actually, exposing only the first touch point as a persistent input source might satisfy most of the use-cases that rely on obtaining hit test results from touch events.

Right, but touch points are transient.

They are activated transiently, and that difference in the approach is something that I think allows us to simplify the design of hit test by removing the concept of transient input sources (or at least allowing UAs to treat them persistently). I took a look at implementing automatic hit test source creation and I do not think that it is possible to implement that variant of hit test in Chrome (and possibly in other multi-process browsers) in a way that we're fine with (at least not if we want to be able to use arbitrary XRRays for those hit tests, and I'm assuming that this is an important use case which we want to keep - immersive-web/hit-test#48 is related to that).

Few straw-man proposals on solving the problem:

• Remove the concept of transient input sources from the spec and treat them in a persistent manner - automatic hit test source creation is not needed to provide pixel-perfect reticle in a synchronous way from touch-based events. Input sources that are not known to the UAs beforehand can still be created, but they will not be removed if the hardware availability does not change.
• Expose first touch-based input source in a persistent manner, remaining touch points will still be transient - automatic hit test source creation becomes an optional part of the hit test API, or is removed. The spec would have to relax its stance around which input sources must be treated transiently.
• Expose touch-based input sources in a semi-persistent manner - they can be removed after some grace period of inactivity. Automatic hit test source creation becomes an optional part of the hit test API, or is removed. The spec would have to relax its stance around which input sources must be treated transiently.
• Allow input source to carry information about multiple target ray spaces to account for the fact that single input source (ex. touch screen) can have multiple inputs and expose input sources in a persistent manner. The transient input sources concept is no longer needed in the spec.

[Manishearth]

I brought this up in the call: a potential different route to take is to keep transient input sources behaving the same, but also provide a way to attach hit tests to the abstract set of all transient input sources.

[AdaRoseCannon]

Removing the agenda since it was brought up in the call. Feel free to re-agenda it.

[NellWaliczek]

@bialpio It sounds like you and I should probably book some time to review and brainstorm about the challenges you're discovering. When I get back from Japan, I'll reach out and we can set up the time.

[bialpio]

@Manishearth - yes, that's also an option, although I think that we should be able to come up with a mechanism that works for both transient and persistent input sources. If handling transient input sources forces us to have a separate mechanism for hit test, then I think I'd rather remove the concept of transient input sources (make them persistent and try to solve the multiple touch points issue in some other manner).
@NellWaliczek - sounds good - let me know when you're available.

[NellWaliczek]

@bialpio and I talked through the concerns raised here and have put together a list of premises we'll write up and would like to validate with the group on the next call.

[Manishearth]

We solved the part of the problem here that was relevant to hit tests in https://github.com/immersive-web/hit-test/issues/51. @bialpio, do you still think there are things here that need resolving?

[Manishearth]

I'm closing this as resolved; if there are any remaining issues that have been missed please file a new bug!