Three.js: scene.background support for equirectangular textures

[mkeblx]

Currently the helper background property on a Scene can be set to either a cubemap (rendered on a cube geometry) or a texture on a fullscreen orthographic plane. There are a lot of equirectangular images (a lot more coming from 360/VR cameras soon) might be nice to also support as another type and get same benefits.

Potential API:

scene.background = new THREE.TextureLoader().load( "textures/360-cam.jpg" );
scene.backgroundSphere = true; 

// OR : more flexible can define mesh
scene.background = new THREE.TextureLoader().load( "textures/equirect.jpg" );
scene.backgroundMesh = new THREE.Mesh(..., ...);

[mrdoob]

How aboud doing a Equirectangular to Cubemap helper instead?

/ping @spite

[spite]

I've got half working a helper to convert equirectangular panoramas to THREE.WebGLRenderTargetCube (based on https://github.com/spite/THREE.CubemapToEquirectangular)

It makes more sense to use cube mapping rather than spherical mapping, since there's cases in which the seams in the wrapping can't be fixed.

[spite]

Pick your poison:

• https://github.com/spite/THREE.EquirectangularToCubemap
• https://github.com/mrdoob/three.js/pull/9746

🍰🍰🍰

[mkeblx]

Awesome thanks @spite. These should be sufficient. I did want to check one thing with #9746.

[mrdoob]

I'm thinking we may want to add THREE.EquirectangularTexture().

[makc]

It makes more sense to use cube mapping rather than spherical mapping, since there's cases in which the seams in the wrapping can't be fixed.

I had someone telling me that they get seams in cubemaps, quoting https://github.com/KhronosGroup/WebGL/issues/1528 this would not happen with the equirectangular image, although you need a bit heavier shader

[spite]

@makc https://github.com/mrdoob/three.js/issues/1621

[makc]

@spite check this 4 tap trick article edit actually nvm, that would require altering the texture any way

[WestLangley]

bump.

I'm thinking that if the scene background isTexture, then we should assume the texture is an equirectangular one, and support that. Currently, the texture is assumed to be flat.

If users have just a flat texture, they can set it as the CSS background.

[spite]

That will make capturing trickier, since the CSS background won't be part of the Canvas. It might be better to have a backdrop/background object that can work with equi/cubemap/flat textures (sphere/cube/tri geos)?

[WestLangley]

That will make capturing trickier, since the CSS background won't be part of the Canvas.

@spite Good point.

Maybe an API that captures these features...

renderer.clearColor
renderer.backgroundTexture // image or procedural texture, backdrop
scene.background //equirectangular or cube

[mrdoob]

@WestLangley We could also add a new mapping... if( scene.background.isTexture && scene.background.mapping === EquirectangularMapping ).

[WestLangley]

@mrdoob Yes, I think that is a good idea. We would have to make sure it still works if the user sets the mapping in the loader callback -- and hence, potentially after the first call to render.

[DavidPeicho]

Has the mapping been abandoned in favor of the PMREM conversion?

I imagine having the mapping directly done in the shader is still nice for the background. What do you think? @mrdoob @WestLangley

[mrdoob]

I imagine having the mapping directly done in the shader is still nice for the background. What do you think?

I considered this recently...

Something like a THREE.EquirectangularTexture which the renderer could internally do a PMREMGenerator pass when used as scene.environment.

🤔

/cc @elalish

[DavidPeicho]

Do we need to use the PMREMGenerator, or should we just handle a new mapping like you said?

if( scene.background.isTexture && scene.background.mapping === EquirectangularMapping ) {

        // Set shader define to `ENVMAP_TYPE_EQUIREC `

}

And fetch the texture using spherical coordinates in the shader.

[WestLangley]

@DavidPeicho

var options = {
    generateMipmaps: true,
    minFilter: THREE.LinearMipmapLinearFilter,
    magFilter: THREE.LinearFilter
};

scene.background = new THREE.WebGLCubeRenderTarget( 512, options ).fromEquirectangularTexture( renderer, texture );

[DavidPeicho]

@WestLangley Interesting! I missed that, really nice and done on the GPU.

It would still be nice to support equirectangular like any other texture as a background with no conversion, but it's not a priority.

[marcofugaro]

thank you @WestLangley, that was useful.

How should the WebGLCubeRenderTarget size be chosen in order to get the best resolution from the image?

I am doing something like this example, with a really hi-res equirectangular image.
Right now I'm getting the closest power of two from the height, but I agree that this should be done in a better way.

const size = highestPowerOfTwo(texture.image.naturalHeight)

[elalish]

@marcofugaro I generally use half the height (assuming powers of two). So for a 1024x512 equirect, I use a 256x256 cubemap. That gives identical pixel sizes at the equator, and the cube map gets 25% fewer pixels overall, since the stretched poles get collapsed.

[Mugen87]

Fixed via #19911 🎉 .