Servo: No controller interaction in babylon.js demos

[jdm]

https://servo.org/hl-home/bbjs/#8Y780Y#57
https://servo.org/hl-home/bbjs/#9K3MRA#2
https://servo.org/hl-home/bbjs/#CHF3KW#48

All of these demos display a pointer from the user's hands, but the drumming demo doesn't register when it's pointing at one of the push buttons (on desktop it highlights in a unique color), and the tennis demo doesn't recognize a pinch action in the left hand to activate the ball.

[jdm]

cc @paulrouget

Manish and I spent a bit of time investigating this. One idea we're pursuing is trying to figure out whether the pointer is being rendered using a different reference space than the hit test (like how there's a difference between grip and pointer space in this code).

This code looks like the bit that's responsible for doing the actual hit test.

I verified that if I modify the drumming demo snippet to use immersive-vr (to allow me to use the WebXR emulator inside of Firefox to load it on desktop) then if I drag a controller so that its ray is touching a button then the button changes colour. This suggests that there is _something_ about the controller that our openxr backend exposes that is treated differently by babylon for some reason.

[jdm]

Also, is there some way to control where the camera starts for the drumming demo? All of the buttons are really far away by default and it's very difficult to test.

[paulrouget]

3 issues:

• hit test not working: talked to Raanan about this. Waiting for his answer.
• camera being too far: submitted a PR.
• can't push the button: I will need help here. I can't get the select event to fire on the emulator. I will put together some tests that someone with a device can run.

[jdm]

Please ask on Teams about the emulator select issue if you have not already.

[jdm]

https://github.com/BabylonJS/Babylon.js/blob/629b7e03e0ae6580be7849acea17aabbc1e21e27/src/XR/features/WebXRHitTestLegacy.ts#L126 looks like the code that should set up the select event. I wonder if we're not enabling the legacy hit test feature?

[jdm]

Corresponding three.js code that sets up the events is https://github.com/mrdoob/three.js/blob/672729dbd0693f3f3369f792fd621574c8c6ccde/src/renderers/webxr/WebXRManager.js#L206-L212.

[jdm]

For what it's worth, pressing the left controller "select button" button in the webxr emulator in Firefox for https://servo.org/hl-home/bbjs/#tennis.js doesn't cause anything to change in the scene.

[paulrouget]

Using const component = controller.motionController.getComponent("xr-standard-thumbstick"); should fix it iirc.

[jdm]

Where?

[paulrouget]

https://github.com/servo/servo.org/blob/gh-pages/hl-home/bbjs/snippets/tennis.js#L161

[jdm]

You're right; making that change allowed the "select button" button to work correctly in the webxr emulator. There's no difference in the hololens.

[paulrouget]

@jdm can you please try this snippet with the device:

// hittest.js
var createScene = async function () {
    var scene = new BABYLON.Scene(engine);
    var camera = new BABYLON.FreeCamera("camera1", new BABYLON.Vector3(0, 5, -5), scene);
    camera.setTarget(BABYLON.Vector3.Zero());
    camera.attachControl(canvas, true);
    var light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 1, 0), scene);
    light.intensity = 0.7;
    var sphere = BABYLON.MeshBuilder.CreateSphere("sphere", {diameter: 1.4, segments: 32}, scene);
    sphere.position.y = 1;
    sphere.material = new BABYLON.StandardMaterial("sphereMat", scene);

    const environment = scene.createDefaultEnvironment();
    const xrHelper = await scene.createDefaultXRExperienceAsync({
        floorMeshes: [environment.ground],
        inputOptions: { doNotLoadControllerMeshes: false}
    });
    scene.pointerMovePredicate = (mesh) => {
        return mesh.isPickable;
    }

    return scene;
};

Add it under hl-home/bbjs/snippets and load http://yourserver/hl-home/bbjs/#hittest.js

This could improve the hittest situation. You will see if the ray intersect with the sphere if a disk appears on the sphere.

It's very possible that it doesn't work - if it doesn't, does rolling your hand changes the situation?

[paulrouget]

Note about the select event not being registered: onButtonStateChangedObservable is not called on device according to the logs @jdm share with me.

[jdm]

@paulrouget I tried hittest.js, and no matter how I oriented my hand there did not appear to be any ray intersection.

[jdm]

Correction - the ray does successfully intersect with the sphere and displays the disc.

[paulrouget]

So we made some progress on the BBJS side. To fix part of the hit test issue, adding this to the demo for now will fix it:

   scene.pointerMovePredicate = (mesh) => {
        return mesh.isPickable;
    };
   // or scene.constantlyUpdateMeshUnderPointer = true;

The drum color is not changing yet. Investigating that part.

---

For the event not being received, there is a discussion ongoing on team about interaction profile. I don't know if this will help with the emulator issue (no events ever happen) or with the device issue (no event with BBJS demos).

[jdm]

Apparently this is supposed to trigger a select action in the emulator:

You can do it with mouse, when pressing the left ALT key and mouse right click it should register a "select" action.  that's effectively an "air-tap" gesture.  the same way you can use it in the start menu or shell on hololens emulator.

[paulrouget]

Yes, that's the gesture we usually use for non immersive mode interactions, but we are not getting any event in IM.

We are also getting coordinate issues only with the emulator. The drums demo in the emulator is not showing the ray properly, whereas on the device, it's working well.

[paulrouget]

For the select event not being caught by BBJS, citing @RaananW:

We don’t use the select event for pointer selection, we use the controller’s main component (for example in VR it would be the trigger) to detect changes in the user’s input. This is done when a motion controller is present and initialized:

https://github.com/BabylonJS/Babylon.js/blob/master/src/XR/features/WebXRControllerPointerSelection.ts#L393

The main component is actually defined in the controller’s specifications, which, I suspect, would be the problem here. I don’t know what type of controllers are loaded when using XR. My guess would be some form of generic-hands (thou you can probably tell better than me). The discussion about adding hand-profiles to the XR input lib is here - https://github.com/immersive-web/webxr-input-profiles/issues/105 . I can tell you that when using hands in the oculus quest, the default generic controller is loaded and click is working.

If the motion controller is not defined, there should be a fallback to a different generic controller. So first it would be great to know what profile is loaded, whether or not the motion controller is initialized, and if not – why.

To find out what profile is loaded, debug this: https://github.com/BabylonJS/Babylon.js/blob/master/src/XR/motionController/webXRMotionControllerManager.ts#L96 . This function will use the xr input and the data we have to find the best matching controller. At the end it should return a profile (string). Would be good to know which one it is. Ir returns the profile here - https://github.com/BabylonJS/Babylon.js/blob/master/src/XR/webXRInputSource.ts#L95 and notify the observers, which are very important for the pointer selection. My guess – this step fails or doesn’t execute.

To the xrInputSource is initialized correctly (pointer selection is attached to the target ray space), but the motion controller is not initialized correctly, causing the function that is responsible to the actual input to not execute.

TL;dr – I think you are right with suspecting that the observers are not notified, now we just need to find out why. We need to find out what motion controller profile is loaded, and whether or not the motion controller is initialized and have its components ready.

[paulrouget]

I think this is why we are not getting any event with the device: https://github.com/BabylonJS/Babylon.js/blame/072e28d1814c8849440caf020971e049d5e9df27/src/XR/webXRInputSource.ts#L93

[RaananW]

Gamepad input is needed to get the current state of the components.

The name might be a bit misleading, but the object is exposed on all WebXR-enabled browsers, when an input is available. The specs are here - https://www.w3.org/TR/webxr-gamepads-module-1/

The part we are using is this:

export interface IMinimalMotionControllerObject {
    /**
     * Available axes of this controller
     */
    axes: number[];
    /**
     * An array of available buttons
     */
    buttons: Array<{
        /**
        * Value of the button/trigger
        */
        value: number;
        /**
         * If the button/trigger is currently touched
         */
        touched: boolean;
        /**
         * If the button/trigger is currently pressed
         */
        pressed: boolean;
    }>;
}

[RaananW]

This explainer gives a better detailed view of the module: https://github.com/immersive-web/webxr-gamepads-module/blob/master/gamepads-module-explainer.md

[paulrouget]

Thanks - this explains why we're not getting events with BBJS.

https://github.com/servo/servo/blob/3df65c02fe73a4cce9e391fa1de0b9a4d613c51b/components/script/dom/webidls/XRInputSource.webidl#L25

@Manishearth is it just a matter of linking the proper XRSystem::gamepads to XRInputSource::gamepad?

[Manishearth]

We could, yes, but it's complicated, it's not clear in the spec if we should be exposing the attribute for non-gamepad controllers like hands: https://github.com/immersive-web/webxr-gamepads-module/issues/23 . It's also not clear what the input profile situation should be

Applications should be using the onselect and onsqueeze events if at all possible.

I might expose a basic gamepad anyway.

[RaananW]

We have discussed this internally - I will create a patch for the pointer selection using the select and squeeze events (of course including the squeeze start and end). Will work on that tomorrow and get back to you.
Are there any other events you are exposing?

[Manishearth]

We're only exposing select and squeeze, and on the hololens there's an issue with unconditionally exposing squeeze (https://github.com/immersive-web/webxr/issues/997). I'm hoping to eventually expose a gamepad, but we need to resolve the spec stuff first.

Thanks for making the fixes upstream! In general it's better to use those events wherever possible because it's not guaranteed that all user agents that implement the WebXR spec will also implement the gamepads module (they're separate specs).

[RaananW]

The pointer selection module is only in charge of pointer event simulation, so I will initially implement the select start and end. I'm leaving the squeeze out of this module's scope for now. Will be done tomorrow, I'll keep you updated here.

[Manishearth]

Thanks!

[RaananW]

Quick update - the implementation is now available in the playground, a hard refresh is recommended.

[jdm]

I have verified that https://joshmatthews.net/servo.org/hl-home/bbjs/#drums.js using the latest babylon.js preview now supports hit testing and the select action as expected!

[jdm]

However https://joshmatthews.net/servo.org/hl-home/bbjs/#tennis.js still does not recognize the pinch action in the left hand, so the ball just stays sitting on the ground.

The color picker demo works as expected, although there's a clear delay between when I stop a pinch action and when the pointer registers that it's no longer clicking.

The goalie demo now allows me to choose the easy or hard mode, but it was unclear whether I was able to successfully catch any of the balls that were launched at me. This was a combination of no visible feedback because our 2d canvas text is invisible (so there's no visible score change), but also it's not clear if I need to have my hand in front of the ball, or have the ray pointing and it and make a select gesture.

The physics demo did not have any interaction when I made the select gesture with either hand.

[RaananW]

The only interactions available are the ones involving the main button (select).

The physics demos will work only when a mesh will be attached to the position of the hands, which is usually done when there is a model for the motion controller.

The other interactions will only work when there are other components (like thumbstick etc'), but i can create a playground for you to check teleportation with select (the same way it works in gaze mode).

I will find time tomorrow to create a physics demo for the hololens so you can check interactions with physics impostors, and will try finding time to create a simpler demo to interact with objects in XR

[paulrouget]

@RaananW for the physics demo, isn't the bullet supposed to work with the select event? Also - we found that, with the WebXR API Simulator, in the tennis demo, replacing getMainComponent() with getComponent("xr-standard-thumbstick") get us the select event to work.

[RaananW]

Modified tennis demo - https://playground.babylonjs.com/pg/CHF3KW/revision/68
This will use the left hand's squeeze event to "hold the ball" in the air. when released the ball will be released as well, so you can practice throwing it.

[paulrouget]

@jdm To try @RaananW url in Servo, you can use: https://playground.babylonjs.com/full.html#CHF3KW#68

[RaananW]

@paulrouget -
The physics demo was checking the handness of the motionController (instead of checking the input source's handness). Here is a corrected version - https://playground.babylonjs.com/pg/B922X8/revision/24 . Reset with left select, shoot with right select. Note about squeeze - it can technically be used to lift objects, but this is currently unstable. I will get back to you with an updated version.

the getComponent() will only work if a motion controller is present, so I am not sure why it works...

About teleportation in the scene - I have a fix coming up that will allow you to teleport using select (point 3 seconds at the floor where you want to teleport to). This will come in the next nighly - https://github.com/BabylonJS/Babylon.js/pull/7999

[RaananW]

Also a note about the physics demos - they are usually CPU-hungry. The number of physics steps taken in each frame is kept to a minimum (to make it run at reasonable FPS), so if a ball goes through a wall, you will have to accept it :-) .

The goalie training - use your hands to hit the balls coming at you. The score goes up one point if it was not a goal, or down one point if it was. Use the easy mode, as it is much less hard on the CPU (and the collisions will work better).

[RaananW]

Another demo that you can use:

https://playground.babylonjs.com/pg/RQM31Y/revision/6

This will work in AR well, just need to re-enable the background remover feature.

Selecting a mesh will enable our gizmo manager, that will allow to manipulate the mesh. you can enable/disable the gizmo with the left squeeze

Right Squeeze will enable physics on all floating meshes. A second squeeze will randomize the meshes again (reset the scene).

Head and hands impostors are enabled, so you can push the meshes and bounce them on your head

[jdm]

Today's nightly successfully demonstrates controller interaction in the drumming and clouds demos, so I've promoted them to featured demos on the homepage. I'm going to go ahead and close this issue and file new ones for remaining Babylon.js controller interaction issues that I notice when using the demos.