Cesium: Integrate with other WebGL engines

[pjcozzi]

There's been some interest in integrating Cesium with other WebGL engines.

On the Cesium side, this should be pretty straightforward. The changes will require Cesium to only own the context during a call to Scene.render, not forever. This is already mostly there since objects only access the context in update functions called from within Scene.render, but we also keep some state between frames that we'll need to change:

• Render State #638
• Clear states (if I don't remove them sooner)

Also, when we do the post processing framework, we need to be mindful of when we are rendering to texture vs. the main framebuffer.

[m-schuetz]

Hi,

I've done a first test with cesium + three.js here:
http://potree.org/demo/experimental/2015.08.19_cesium_test/examples/cesium_test.html

The basic idea is this:

• Copy "cesium.camera.viewMatrix" and "cesium.camera.inverseViewMatrix" to "three.camera.matrixWorldInverse" and "three.camera.matrixWorld"
• The potree point cloud is given in EPSG:21781(swiss). This is first transformed to WGS84(degrees) and from that to Cesium.Cartesian3.fromDegrees().
• pointcloud.position is set to the Cartesian3 value of the bounding box minimum
• pointcloud.lookAt(bottomLeftHigh) is called to align the point cloud to align the point cloud on the ground
• potree.pointcloud.up.copy(latDir) is used to rotate the point cloud to align with longitude and latitude

Issues:

• This example is specific to a point cloud in the EPSG:21781 coordinate system
• three.js is rendered in its own canvas over the cesium canvas. Idealy, both rendering engines should use the same context and render into the same buffers, if possible.
• Precision issues with the three.js camera, especially noticeable when the camera is moving close to the point cloud. Possibly because three.js uses floats? Does cesium use double precision values for transformations?
• Only works if the aspect ratio is 1, therefore the fixed canvas size.

[m-schuetz]

Thanks, I'll take a look at it!

[m-schuetz]

Found out how to render into the same canvas with the same context:

First, create a three.js renderer and pass the canvas as argument. The canvas/context was already initialized by cesium. Three.js will call getContext() for the same canvas and therefore, according to the spec, receive the same context as cesium.

three.renderer = new THREE.WebGLRenderer({alpha: true, canvas: cesium.viewer.canvas});

In the render loop, cesium has to be rendered first.

During the rendering with three.js, WebGLRenderer._currentProgram has to be set to null before using the three.js render function. Since _currentProgram is a private variable, this can only be done through WebGLRenderer.resetGLState(). resetGLState(), however, also calls state.reset which causes errors. This workaround therefore temporarily changes the state.reset function to an empty function

var resetFunc = three.renderer.state.reset;
three.renderer.state.reset = function(){};
three.renderer.resetGLState();
three.renderer.state.reset = resetFunc;

Now it's probably not that important for most cases but it would still be interesting to use the same depth buffer in threejs as well. Do you think that will be possible with the way cesium handles depth buffers? At https://cesiumjs.org/massiveworlds/ there are slides about rendering with multiple frustums in which case reusing the depth buffer is probably out of question.

[pjcozzi]

Thanks for the info.

Now it's probably not that important for most cases but it would still be interesting to use the same depth buffer in threejs as well.

I expect most apps will want this, e.g., depth testing points vs. Cesium's terrain. I expect this can be done by setting Three.js' camera near and far planes and explicitly rendering the Three.js scene for each frustum in Cesium. See executeCommands in Scene.js.

The jittering issue in Three.js we discussed on twitter would also need to be addressed.

[brucefuqiming]

We're doing a project to mix Cesium and Three, like potree. Cesium is rendered firstly,then Three's elements rendered on the Cesium's canvas.But the problem is Three's elements always block Cesium's. e.g.: there is a geometry created by Three in front of the Cesium‘s earth,it can block the Cesium‘s earth and other elements correctly, but when the scene is rotated to the backward, Three's object still block the Cesium,maybe depth buffer is the main reason,but we cannot find the solution to solve. Do you have any methods?We wanna make Cesium the part of the Three’s scene,like an ordinary geometry. Is it possible?

[pjcozzi]

@brucefuqiming this is because Cesium and Three have two different depth buffers and Three is being rendered second. To merge the scenes correctly, this will require core engine support in Cesium as described in this issue.

[pjcozzi]

For a prototype of integrating Cesium and three.js, see Section 6.2 here: https://www.cg.tuwien.ac.at/research/publications/2015/Adorjan-2015/Adorjan-2015-thesis.pdf

[m-schuetz]

One possibility could be to render color and depth buffers in one lib and then use the depth buffer(s) in the other lib to compose the renderings of both.

e.g.:

• User renders his stuff in three.js and provides color buffer, depth buffer, and projection parameters to cesium.
• Cesium does a screen pass (or one for each frustum) with the three.js color and depth map as input and discards all fragments that do not pass a custom depth test. The custom depth test would consist of transforming the three.js depth values to cesium depth values.

Vice-versa, cesium could provide a color buffer and all of its depth maps and someone can use this to do a custom compose step to merge the cesium renderings with his three.js or other renderings.

[gaojunxiaowwww8888]

Hello sir, I would like to ask you about the cesium+three.js how to load the.Gltf model, and let the three.js camera with cesium camera rotation, there is no good examples of reference, thank you.

[pjcozzi]

@gaojunxiaowwww8888 you might want to ask the three.js community or try the Cesium forum: http://cesiumjs.org/forum.html. Just a heads up that the Cesium team doesn't yet have any three.js experience.

[gaojunxiaowwww8888]

Hello sir, I want to ask is whether in the development of cesium+three by three cesium light to improve the illumination effect, three light can affect cesium light; or what is the way to change the cesium in the light effect?

[gaojunxiaowwww8888]

@m-schuetz Hello sir, I want to ask is whether in the development of cesium+three by three cesium light to improve the illumination effect, three light can affect cesium light; or what is the way to change the cesium in the light effect?

[gaojunxiaowwww8888]

Hello, sir. I don't know how to improve the lighting effect of cesium. Is there a good example for reference?

[pjcozzi]

@gaojunxiaowwww8888 please do not post the same question over and over again in this issue. As explained in CONTRIBUTING.md, the Cesium forum is the preferred place to ask question; we use GitHub issues for bugs and feature requests.

[gaojunxiaowwww8888]

For the cesium+three.js development process, the three model jitter problem is solved? Now encountered this problem, I hope you can help, thank you

[pjcozzi]

@gaojunxiaowwww8888 please do not post the same question to the Cesium forum and this repo. Your question was answered in https://groups.google.com/forum/#!topic/cesium-dev/gCWi3BpT47U.

For jitter in three.js, your best bet is to ask the three.js community as I mentioned to you above.

[pjcozzi]

Related blog post: https://cesium.com/blog/2017/10/23/integrating-cesium-with-threejs/

[pjcozzi]

For discussion, see https://groups.google.com/forum/#!topic/cesium-dev/GX7xLXrPDsE via #6094

[pjcozzi]

Just adding a note to myself that we would also consider down-scoping this if it makes it easier to implement, e.g., support three.js geometries but not post-processing, etc.

[pjcozzi]

Related: luma.gl's WebGL state tracker