Three.js: EXR Envmap not working on iOS

[krfft]

Following the example on: https://threejs.org/examples/webgl_materials_envmaps_exr.html
and loading a EXR file as environment map with:

var pmremGenerator = new THREE.PMREMGenerator( this.renderer );
pmremGenerator.compileEquirectangularShader();
var exrCubeRenderTarget = pmremGenerator.fromEquirectangular( texture );
this.hdr = exrCubeRenderTarget.texture;

This works perfectly on all desktop browsers but on iOS the HDR is completely black and no lighting is applied to the model. This also applies to the example link on iOS: https://threejs.org/examples/webgl_materials_envmaps_exr.html

Three.js version

• [ ] Dev
• [x] r115
• [ ] ...

Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

OS

• [ ] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [x] Android
• [x] iOS

Hardware Requirements (graphics card, VR Device, ...)

[mrdoob]

Any errors in the console?

[krfft]

[Warning] THREE.WebGLRenderer: EXT_frag_depth extension not supported. (three.module.js, line 16023)
[Warning] THREE.WebGLRenderer: WEBGL_draw_buffers extension not supported.

[krfft]

It seems to be the case for Android too, just tested on Chrome on Galaxy S10

[sciecode]

I'm able to replicate this.

At a quick glance, this looked related with PMREMGenerator not being able to render to FloatType render target, which is a known limitation on mobile. However changing it to HalfFloatType also doesn't work, for whatever reason.

This doesn't look related with EXR per-se, as changing https://threejs.org/examples/webgl_materials_envmaps_hdr.html to HalfFloatTypeor FloatTypealso causes the same issue.

[Mugen87]

It seems to be the case for Android too, just tested on Chrome on Galaxy S10

Can reproduce with a Pixel 1.

[sciecode]

I believe the problem stems from the fact that a DataTexture encoded as Uint16Array can't be correctly read by the GPU on mobile.

PMREMGenerator only copies the texture type to the render targets being used internally.
So for the case where the input HDR is in FloatType, the framebuffers are also FloatType and a mobile can't render due to spec limitation. When the input HDR type is HalfFloatType and a mobile would be able to render, it fails inputting the DataTexture to the GPU.

What I'm not sure is how we'd go about fixing this. Nothing comes to mind.

[mrdoob]

Any ideas @elalish?

[elalish]

The PMREMGenerator is designed to work with RGBE for exactly this reason (since RGBA8 framebuffers are supported everywhere). I think the best solution would be to read in the EXR as an RGBE texture and pass that to PMREMGenerator. Unless there's a reason that would be a problem elsewhere?

[sciecode]

It is possible to output an UnsignedByteType from EXRLoader as we do with HDR. But implies losing information of the alpha channel and some extra time processing data, but given how the texture is being run as an env map, perhaps that isn't too bad.

[WestLangley]

I think PMREM should support the standard HDR formats. That would include .hdr and .exr.

We also want to avoid having to decode RGBE in the shader, so support for floating point data textures is important.

[elalish]

@WestLangley I tried that way originally, but switched to RGBE because while floating point textures are widely available for decode, the same is not true for writing to framebuffers. Samsung devices can't write halffloat buffers and iphones can't write float buffers, so I ended up decoding RGBE in the shaders because that was the only way I could find to make a broadly cross-device solution.

Perhaps we should throw a warning about device compatibility when using non-8-bit formats with PMREM?

[krfft]

Don't know if its of any help but the same file worked with my old implementation on mobile devices:

var cubemapGenerator = new THREE.EquirectangularToCubeGenerator( texture, { resolution: 1024, type: THREE.HalfFloatType } );
var cubeMapTexture = cubemapGenerator.update( renderer );
var pmremGenerator = new THREE.PMREMGenerator( cubeMapTexture );
pmremGenerator.update( renderer );
var pmremCubeUVPacker = new THREE.PMREMCubeUVPacker( pmremGenerator.cubeLods );
pmremCubeUVPacker.update( renderer );
hdr = pmremCubeUVPacker.CubeUVRenderTarget;

[elalish]

@krfft Yes, and the reason for that was that the old PMREMGenerator internally converted everything to RGBM (another 8-bit format). It's a little quicker to decode than RGBE, but comes at the cost of having significant interpolation artifacts at high dynamic range.

[sciecode]

I've pushed #19070 as a temporary fix for the situation. Even if we decide to pursue other solutions in the future, there's no harm in allowing EXRLoader to output UnsignedByteType with RGBEFormat.

That way, at least, we have a viable option for people looking to use EXR envmaps with mobile support.

webgl_materials_envmaps_exr.html

[WestLangley]

If I am not mistaken, PMREMGenerator retains the data type passed in.

That is one reason why Loader.setDataType() is important.

I wonder if it would make sense to add PMREMGenerator.setDataType()?

[sciecode]

I discussed privately with @WestLangley and perhaps allowing the user the ability to set the intermediary and final render target parameters ( Format, Type, Encoding ) might allow us to use regular floating point textures as an input for PMREMGenerator and still render to RGBE 8-bit RTTs correctly on mobile.

As far as I'm aware, most mobiles can read from floating point textures, the problem is usually rendering to them. Thoughts @elalish ? Do you think that would be something we can adapt ?

[elalish]

@sciecode Yes, that fits with my understanding. That sounds like a reasonable path forward; I don't have a strong opinion on the shape of three's API. If we're planning to overhaul PMREMGenerator it might also be a good time to enable different sizes (right now it's fixed to outputting a 256x256 cubemap regardless of input size). It seems like with the right choice of defaults we might even be able to make this change non-breaking, which would be nice.

[WestLangley]

allowing the user the ability to set the intermediary and final render target parameters ( Format, Type, Encoding )

I would allow the users to set only the data type, as I suggested above.