Three.js: Three.js does not support the Windows Holographic platform

[lwansbrough]

Description of the problem

Three.js does not support Windows Holographics. I've been working on a platform for creating JS apps in Microsoft HoloLens using WebGL. This requires a bunch of technology (old and new) to get working. I'm using Microsoft's fork of ANGLE which has holographic support. I wrote my own WebGL implementation in C++ using ChakraCore as a bridge for the JS runtime. Currently, I have a working proof of concept app and I'd like to do something similar with Three.js. The issue I'm having is that Three.js of course doesn't support the required GLSL property uHolographicViewProjectionMatrix which is provided by Microsoft's ANGLE fork.

I suppose what I'm looking for is some guidance on how to achieve this with Three.js. We basically don't have a need for a camera (HoloLens is a mobile platform so speed is critical, avoiding JS is necessary when dealing with the projection matrix to maximize hologram stability.)

Three.js version

https://github.com/lwansbrough/three.js/tree/holographic

Browser

None, see: https://github.com/lwansbrough/HolographicJS

OS

• [x] Windows

Hardware Requirements (graphics card, VR Device, ...)

• [x] Microsoft HoloLens or HoloLens emulator

[Itee]

Lucky guy ! This is a nice project.
I always wanted to work with Hololens... But a bit expensive !

As you can see in WebGLRenderer doc the gl context come from the browser (take a look here), or could be given as parameter.

The WebGLRenderingContext interface provides the OpenGL ES 2.0 rendering context for the drawing surface of an HTML canvas element. So may be you should investigate how you could "override" the provide OpenGL ES 2.0 with that of windows.

Hope this, help !

[lwansbrough]

@TristanVALCKE Thanks, you are correct - I've got integration working that far. I had to shim the canvas element as well. There's something wrong with the shaders though - HoloLens requires a very specific way of rendering shaders which Three.js doesn't conform to. The result is my render surface flashes different colours.

[mrdoob]

uHolographicViewProjectionMatrix? If the glsl var starts with u sounds like it's a uniform? Which means that it's being passed from C? So you basically want to feed that matrix to three.js' camera.projectionMatrix.

[lwansbrough]

@mrdoob Would there be any way to accomplish this without accessing the matrix in JS? Through shaders exclusively, for example?

[mrdoob]

How would you accomplish this with shaders exclusively?

[lwansbrough]

I'm not sure actually, I'm not familiar with how Three.js works internally. Perhaps I would just conditionally use the hologram uniforms in the shaders based on whether or not I want to render holographically? So instead of using Three.js' defined projectionMatrix, I'd use uHolographicProjectionMatrix. Would that be enough?

[mrdoob]

How does uHolographicProjectionMatrix look like? And how come there is only one? Shouldn't there be 2? (One per eye)

[lwansbrough]

It appears to be a multi-dimensional array (left eye at 0, right at 1). You can see an example of the vertex and fragment shaders here: https://github.com/lwansbrough/HolographicJS/blob/master/HolographicJSTest/app.js

[lwansbrough]

Note that I have access to both uHolographicViewProjectionMatrix and uHolographicProjectionMatrix

[lwansbrough]

Also note that use of vRenderTargetArrayIndex is required by the holographic fork of ANGLE.

[lwansbrough]

@mrdoob More information can be found under the "Holographic Rendering" header. Should this sort of thing be accomplished with a new "HolographicCamera" or should I use something more like an "effect"? Really interested in getting this working, I think it would be the first major step in getting people really into using JS on HoloLens.

[carstenschwede]

I'm not sure how Microsoft ANGLE fork modifies the vertex shader, but at least for the "simple drawing mode" I understand that it should be sufficient to use a regular THREE.Camera with no special projection matrix and a modelView matrix based on the view matrix provided by the "Holographic App". MS ANGLEs "magic" will double the drawing calls, rewind the view matrix transformation and apply view and projection matrices for each eye. I'm not sure how this is going to work out for more complex shader setups (e.g. it will expect the view matrix operation to be the last operation that is applied to vertices and assumes that the view matrix is only being used during the transformation of vertices but not in the fragment shader).

The "advanced drawing mode" expects you to provide instanced drawing calls for stereo rendering. The current THREE.StereoEffect works by switching cameras and re-rendering the whole scenery. If you'd like to go that route it might make sense to create some form THREE.InstancedStereoEffect first - although you would have to adapt THREE.WebGLRenderer and probably lots of the shaders as well.

The result is my render surface flashes different colours.

You are using simple mode right? Your current example uses a THREE.PerspectiveCamera with a non-identity projection matrix. You also use a THREE.MeshNormalMaterial which means the color in the fragment shader is affected based on the viewMatrix which is not "magically" corrected by ANGLE because it does not know about it. Does THREE.MeshBasicMaterial work as expected? I'll see if I can get the HoloLens Emulator running and will give it a try.

[lwansbrough]

@carstenschwede Seems you've got a better concept of how this stuff fits together than I. I'll give your suggestion a shot tomorrow. Let me know if you have any troubles getting HolographicJS working (feel free to reach me on Twitter.) I'd like to get this working!

[lwansbrough]

@carstenschwede I tried the regular THREE.Camera as well as the THREE.MeshBasicMaterial however the result was the same - just a flashing screen. I was curious so I cut down the code even more. Here's the minimum viable to get the screen to flash in the way it has been:

const gl = new CanvasRenderingContextHolographic();

const canvasShim = {
    addEventListener: () => { }
};

var camera, scene, renderer;
init();
animate();
function init() {
    camera = new THREE.Camera();
    scene = new THREE.Scene();
    renderer = new THREE.WebGLRenderer({
        canvas: canvasShim,
        context: gl
    });
}

function animate() {
    requestAnimationFrame(animate);
    renderer.render(scene, camera);
}

Thus it would appear there is some incompatibility with the renderer itself. Thoughts?

[carstenschwede]

I am going to have a Windows PC setup on Monday - I'll report back when I got the emulator running.

[lwansbrough]

@carstenschwede Awesome!

[carstenschwede]

Sorry for the delay. It seems like I am encountering every possible error while trying to get the emulator running. Will report back shortly.

[lwansbrough]

@carstenschwede No problem, glad to see you're still trying to work this out. I had a bit of trouble getting the emulator to work as well because I'm on a bootcamp MacBook. It eventually did work once I got Hyper-V working.

[makc]

fyi https://github.com/Microsoft/HoloJS

[lwansbrough]

@makc Not surprising at all. :) May be best work work with these guys on the way forward seeing as it's the official solution.

[Mugen87]

Closing. There is a basic three.js support for HoloJS, see https://github.com/Microsoft/HoloJS/pull/23