Babylon.js: WebXR

[deltakosh]

• [x] Explore and understand how WerbXR works
• [x] Write a doc about required dev environment
• [x] Propose an API
• [x] Implement the API
  
  
  
  • [x] XRSessionManager
  
  
  • [x] XRCamera
  
  
  • [x] Engine/Camera modifications
  
  
  • [x] Controllers
  
  
  • [x] DefaultXRExperience
  
  
  • [x] supportsSession
  
  
  • [x] Keep depth in sync with camera https://forum.babylonjs.com/t/webxrcamera-minz-maxz-to-set-xrsession-depthnear-far/640
  
  
• [x] Documentation - https://doc.babylonjs.com/how_to/webxr
  
  
  
  • [x] clerify customRenderTargets vs outputRenderTarget on camera
  
  
• [x] Samples
  
  
  
  • [x] https://playground.babylonjs.com/#AM07G2#4
  
  
  • [x] https://playground.babylonjs.com/#PPM311
  
  

[veikkoeeva]

Further resources, discussion groups etc. from Mozilla blog: https://blog.mozilla.org/blog/2017/10/20/bringing-mixed-reality-web/. Further refined goals at a later post https://blog.mozvr.com/enabling-the-social-3d-web/ .

It appears to me this is about unifying the augmented reality experience. Concretely in near term, codewise, to me this looks like a proposal to add avatars and coordinate their actions via WebRTC (notably, by the way, MS also opened one peer server codebase).

[veikkoeeva]

To continue a bit, this also looks alluring for an outsider: https://blog.mozvr.com/experimenting-with-ar-and-the-web-on-ios/.

It might be useful to see how could WASM be utilized for more compute intensive tasks now that it's coming hard to the fore, so to speak. For instance, there's Blazor template already at https://dotnet.myget.org/feed/blazor-dev/package/nuget/Microsoft.AspNetCore.Blazor.Templates (dotnet new -i Microsoft.AspNetCore.Blazor.Templates::0.1.0-preview1-10071 --nuget-source https://dotnet.myget.org/F/blazor-dev/api/v3/index.json) for those in .NET Standard 2.0, adding SignalR, VSIX etc. in the next few weeks apparently. This is obviously .NET centric view, though likely other ecosystems have something similar.

[veikkoeeva]

Adding https://blog.mozvr.com/responsive-webxr-a-painter-xr/ too. Sorry (a little) for spamming (but goes with the first mentioned and seems to offer some more information). :) Also: https://blog.mozvr.com/building-mixed-reality-spaces-for-the-web/ (build virtual spaces and interact with other people in them).

[veikkoeeva]

Since the (Mozilla) WebXR seems to rely on WebRTC quite a lot, I'll add a few articles pertaining to the woes: https://bloggeek.me/webrtc-adapter-js/, and more tangentially https://webrtchacks.com/webrtc-sdp-inaki-baz-castillo/.

tl;dr; Using WebRTC without adapter.js seem to be _maddness_.

[deltakosh]

Thanks a lot for all these references. @trevordev is currently looking at the current state of the art to see how we can support it:)

[TrevorDev]

I now have a decently working prototype.
https://github.com/TrevorDev/Babylon.js/tree/webXR
Would love to get feedback. @phu-ms @brianzinn

Proposed approach

• XRSessionManager

  • Wrapper around XRSession
  • Initialize XRDevice
  • check if session type is supported (eg. environment integration, hmd)
  • creates renderTargetTexture for xrFramebuffer
  • talks to the engine to switch animation frame requester to be driven by the xrSession
  • rayCasting in ar scene

• XRCamera

  • New camera type which can be updated by the XRSessionManager
  • Position/rotation for this camera will always be what is returned by the XR session (If this is not desired a parent mesh should be used)
  • When updated by an XRSessions rig cameras cameras will be created/destroyed to match the number of views of the XRSession and their view/projection matrices will be updated
  • MultiView extension will be used to render this camera when > 2 views are present
  • For testing, this can be updated manually to perform visual tests.
• Engine/Camera modifications
  
  
  
  • Engine will add customAnimationFrameRequester which when set will be used to drive the render loop, this will be set to the XRSession to enter XR mode
  
  
  • frameBeginTime, frameBeginXRFrame properties, when XRSession is used as the frame requester the callback parameters will be time and XRframe, these will be stored in the engine to be optionally retrieved on each frame. (These are the best ways I found to modify the engine but would be interested in any cleaner ideas)
  
  
  • Camera will have added customDefaultRenderTarget property, when set it will render to that instead of the canvas render target
  
  
• Controllers
  
  
  
  • WebXR spec still doesn't seem finalized on this, sounds like they will be more restrictive and only provide one button.
  
  
  • Current thoughts are to either always use a custom vr controller mesh with one button or somehow reuse our existing vr controller classes if possible.
  
  
• DefaultXRExperience
  
  
  
  • This will replace the existing DefaultVRExperience
  
  
  • Supports VR/AR (Treated the same) and Magic window VR
  
  
  • Provides no customization, if this is desired this should be done manually
  
  
  • As webVR gets deprecated(Years away) I believe there will be a polyfill to get webVR from webXR api but we should recommend people use/move to webXR
  
  

Note: I'm recommending vr experience helper and webVR camera be marked as deprecated due to the inconsistent position behavior defined by the webVR camera where is position is always zero as well as the current complexity of the matricies in it making the moving car scenario difficult.

Core scenarios

• Magic Window
• Traditional VR 6dof/3dof
• AR
• Customizing defaultXRExpereince
• Moving Car
• Ray casting
• Custom controllers
• Scenarios shown here: https://immersive-web.github.io/webxr-samples/

Setup

Current setups I have tested to work:
Desktop setup:

• Vive
• Chrome canary
• In chrome canary go to chrome://flags search webXR and enable all xr related flags

Current mobile setup:

• Android phone that support AR core (Eg. Pixel 2 XL)
• Android 8 or higher
• Chrome canary
• In chrome canary go to chrome://flags search webXR and enable all xr related flags

Not supported

• Iphone, browsers dont support xr, https://github.com/mozilla-mobile/webxr-ios supports an older version of webXR, once it matches the current spec, it should work.

Debugging:

• https://developers.google.com/web/tools/chrome-devtools/remote-debugging/

Note: https://github.com/immersive-web/webxr-polyfill can be used in the future to polyfill browsers that dont support webXR

[deltakosh]

Some comments about naming:

• I would probably name XRSessionManager: WebXRSessionManager and so on
• "Provides no customization, if this is desired this should be done manually": I still think we can provide SOME if they make sense
• Deprecation should only be in comments

[brianzinn]

Great work!!

• DefaultXRExperience is not in your repo. My main comment is about checking capabilities and perhaps even dynamically broadcasting those capabilities (for listeners) and building button(s) accordingly? There is a good summary here:
  https://github.com/immersive-web/webxr/blob/master/explainer.md#detecting-and-advertising-xr-mode
• "Provides no customisation" - What if I just want XR for Magic Window only, for example? For some scenarios developer will want to choose ie: "stage" frame of reference. Enable/disable canvas + headset mirroring. Are those customisations?
• I feel it's a bit early to deprecate VR/webVR, but perhaps they could work together? How are you planning to fallback from XR to VR based on ie: older browser support? Not sure exactly either how you want to replace VRExperienceHelper and maintain backwards compatibility.

Thanks. Looking forward to seeing what's next! :)

[TrevorDev]

Thanks,

Yea, some pieces are still not prototyped, feature detection will definitely be part of the xrSessionManager, for no customization's what I mean is xrHelper should ship with very little options and be stricter on adding additional options, I would like to avoid the complexity which led to some issues/confusion in the vrHelper, eg the way multiple cameras were handles. VR/webVR will continue to exist and not be deleted for back compat but it would be recommended to switch to XR to gain additional features. Ideally the VR helper could be extend to become the XR helper but due to camera inconsistencies with the webVR cameras (eg. position of webVR camera isn't the actual position) a completely new class was recommended.

I'm sure this design will change as more issues will expose themselves after pieces are checked in and people start trying it out and especially since the w3c spec is still in flux

[soniapello]

@deltakosh It will be added to babylon 4 which is planned for April 30, 2019.
Is there any chance that a major browser like microsoft edge support webxr at that date ?

[TrevorDev]

@soniapello You can see the timeline in the current webXR charter here: https://www.w3.org/2018/08/immersive-web-wg-charter.html

The spec and implementation is still under churn and the recent canary update this week broke most of the magic window samples I've been building as well as other samples eg. https://codelabs.developers.google.com/codelabs/ar-with-webxr/#0

Major browsers such as Edge, Chrome and Firefox have all expressed interest in supporting WebXR. Based on this there is definitely a chance it will be supported (at least behind flags like chrome canary) by then but not guaranteed. I would probably give it 50% chance odds but that's just a guess.

[soniapello]

Thanks for your answer Trevor

[nikeinikei]

I couldn't get the example to work on chrome canary (version 74), it seems like they removed the concept of an XRDevice, at least according to https://immersive-web.github.io/webxr/#xr-interface. I also tried adjusting the code to make it work in the newest chrome canary version, but was unable to do so (This is my first attempt at a contribution to BabylonJS, and I'm still unsure how BabylonJS's internals work). Is someone else working on this?

[deltakosh]

@TrevorDev is the owner of that feature

[TrevorDev]

Hey @nikeinikei ,

You can see this thread: https://forum.babylonjs.com/t/web-xr-current-status/1354

yea the implementation is a bit out of date, it targets a version of chrome canary from early January (it should work if you have an apk). It’s hard to figure out what to target at the moment (firefox, polyfill, chrome chanary) and it is unclear to me how to find which spec which implements so I’ve been holding off until I have an implementation/spec that is more final.

The webXR spec was just updated Feb 13th https://immersive-web.github.io/webxr/ but once a browser announces they support the latest spec I will sync the code again. (I'm hoping this is soon)

WebVR works right now (https://www.babylonjs-playground.com/#TAFSN0#2) so you could take a look at that at the moment if you are not dependent on an XR feature while we wait.

[nikeinikei]

WebVR works right

So is the babylon api for webxr unlikely to change?

[TrevorDev]

@nikeinikei The webVR api won't change but webVR is not widely supported in all browers and is being phased out a bit in favor of webXR (w3c spec editors found flaws trying to extend webVR so they decided on making a new api). To better support webXR in babylon we will also have a new api, but it will be very similar to webVR so I would use webVR for now and port to XR if new features are required.

It looks like there is an issue to track what browsers/setup supports what version of the spec https://github.com/immersive-web/webxr/issues/529

[leon]

I've been doing some research into how for web-xr has come since january.
It looks like chrome 76 may contain the new web-xr specification.
https://github.com/immersive-web/webxr/blob/master/explainer.md

It's available in canary by enabling the xr stuff in chrome://flags

I'm going to try out babylons current implementation on my samsung s4 tablet and samsung s8 to see if it works.

@TrevorDev Maybe it's time to have another look :)

[leon]

@TrevorDev Seems that we need to do some updating.
I'm getting:
ERROR Error: Uncaught (in promise): TypeError: this._xrNavigator.xr.requestDevice is not a function

[TrevorDev]

@leon , Thanks, I have started trying to resync with the latest spec again and there are a couple interface changes.

I managed to get one spec samples (https://cs.chromium.org/chromium/src/chrome/test/data/xr/e2e_test_files/html/test_webxr_input.html) to work in the latest chromium (only works with windows mixed reality headset, not rift). Ill code against this for now, hopefully Ill have something working again soon.

[TrevorDev]

@leon I have synced Babylon again. The playgrounds linked in the recent PRs I've merged should work. Ill be working on documenting everything/getting feedback on the current API next week. Chrome with xr flags turned on finally seems to work with rift and wmr headsets 🥇 so hopefully the spec doesn't change a lot from now on. Let me know if you have any questions/issues.

[Yonet]

@leon examples do not work on Chrome Canary 79. Can we re-open the issue please. Thanks.

[deltakosh]

Do you mind creating a new one and pinging @RaananW ? Thanks a lot