Three.js: GPGPU examples do not work on iPadOS Safari

[jonobr1]

Description of the problem

These Three.js examples do not run on Safari iPadOS 13.5.1. This could possibly be the case on other iOS environments like iPhones, but I don't have one to test it out on. It looks like the GPGPU examples don't run the compute shader and the only warning are these:

THREE.WebGLRenderer: EXT_frag_depth extension not supported.
THREE.WebGLRenderer: WEBGL_draw_buffers extension not supported.

The examples run fine on desktop Safari. At one point, this was due to OES_texture_float as described in this issue. But, it looks like GPUComputeRenderer.js handles this check and doesn't trigger an error message.

Three.js version

• [ ] Dev
• [x] r118
• [ ] ...

Browser

• [ ] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer
• [x] Mobile Safari

OS

• [ ] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [x] iOS

Hardware Requirements (graphics card, VR Device, ...)

This was found on iPad Pro Gen 2 (~2018 model). Running iPad OS 13.5.1

[mrdoob]

I have a iPhone XR for testing and they do work there...
I should be able to try them on an iPad Pro next weekend.

[jonobr1]

Glad to hear this isn't an issue on iPhones! I imagine that's a much larger user base than iPads. Also for context the examples work fine on Chrome for iPad.

[jonobr1]

It's weird because this was working on iPads about a year ago. Reference

[jonobr1]

Further reference, this WebGL Fluid Simulation (not based on Three.js) works: https://paveldogreat.github.io/WebGL-Fluid-Simulation/

I’m going to dig into what might be different on how he’s doing GPGPU style computation to work on iPads.

[jonobr1]

Okay, small update. I made a dead-simple Glitch example to try to isolate the issue. You can run it here (or remix it here).

A square in the middle of the screen should mod from black to red. This works in the environments outlined above along with the Three.js examples that use GPUComputationRenderer.js. From what I understand in @PavelDoGreat's code the frame buffers are all pretty standard. His Data textures are Uint8Arrays instead of Float32Arrays. So, I first tried changing the THREE.DataTexture used in GPUComputationRenderer.js to THREE.Texture with an empty canvas. You can see that commented out code here. Chrome is fine, but Safari on the iPad pro is still black.

This made me think that it wasn't the Data Texture, but the WebGLRenderTargets used in GPUComputationRenderer. Again, differences seemed minimal (linear instead of nearest filtering for instance). One thing that was different that I didn't understand at all (which I think is related to Half Float Type Textures) is that the internal format numbers in @PavelDoGreat's experiment use 36193, a number that isn't in the Three.js source at all. So, I commented out the internal format numbers that Three supplies and replaced it with Pavel's. This errored out this: WebGL: INVALID_OPERATION: texImage2D: ArrayBufferView not TypeUint16. You can see that commented out code here.

At any rate, it's weird that this works on iOS iPhones but not iPads. I got confirmation from a friend of mine who tried this out on his iPad with the same problem. I'd like to think it's a browser issue, but Pavel seems to be able to do the FBO ping-pong without any problems on the iPad. This still makes me think there's a simple tweak that can be made to the GPUComputationRenderer in order to get it working on iPads.

[mrdoob]

Did you tried asking Pavel?

[jonobr1]

I've asked on his Discord channel and hopefully he gets a notification from this thread. Between the two 🤞 he responds.

[jonobr1]

Okay, was able to figure this out! 😌

Pavel is checking if the browser supports WebGL2 for assigning either Float Types or Half Float Types to his textures. The GPUComputationRenderer.js, however checks the userAgent. Not surprisingly the userAgent changed when iPadOS released and so now the check here returns a THREE.FloatType on iPads. This explains why it works for iPhones still.

Unless there is a good extension check to see if Float Type is supported (I say this more as a question cause I don't know WebGL that well), maybe a better solution would be to default to Half Float Type and allow the user to force a different type by passing an argument / calling a function? e.g:

var simulation = new THREE.GPUComputationRenderer( sizeX, sizeY, renderer );
simulation.setTextureType(THREE.FloatType); // Default is THREE.HalfFloatType
// Create your GPU variables, dependencies, initialization, etc.

[mrdoob]

Sounds good to me! 👌

[jonobr1]

I'll make a PR for this then. Thanks for all the input!

[WestLangley]

Maybe follow the .setDataType() API of EXRLoader and RGBELoader, etc.

gpuCompute = new GPUComputationRenderer( WIDTH, WIDTH, renderer )
        .setDataType( THREE.FloatType );

Perhaps leave the default as FloatType and check for the capability in the app itself.

[jonobr1]

Makes sense. Though, FloatType breaks without errors on Mobile Safari (hence the reason for this issue). At least HalfFloat runs everywhere. Happy to do whichever you all want.

[jonobr1]

Sorry to extend this conversation longer than it maybe needs to be, but @spite showed me some feature detection that he developed to do the data type checking: https://github.com/spite/solskogen-2020/blob/master/js/features.js#L184

It would be easy to adopt this into GPUComputationRenderer.js so the detection is based on features and not on browser vendors. This could be in addition to what @WestLangley described. Would this be overkill?

[mrdoob]

@jonobr1 can you extract only the relevant bits from features.js?

[jonobr1]

I believe so, yes. It's the logic for canRenderToFloat

[WestLangley]

It would be easy to adopt this into GPUComputationRenderer.js

IMHO, adding a feature detection utility to the examples is a good idea, but I'd call it from the app, not from GPUComputationRenderer.

[jonobr1]

Sounds good. I'll be sure to update the examples in this way. Thanks for all your input!

[jonobr1]

Soooo, I spent a little time to distill what was necessary from Spite's code. Unfortunately, it's quite a bit of code to do this check. Moved to one function it's:

function cannotRenderToFloat () {

    var canvas = document.createElement("canvas");
    var gl = canvas.getContext("webgl");
    var context = gl;

    if ( !gl.getExtension( "OES_texture_float" ) ) {
        return true;
    }

    var vs = context.createShader(context.VERTEX_SHADER);
    context.shaderSource(vs, `
        attribute vec4 a_position;

        void main() {
            gl_Position = a_position;
        }
    `);
    context.compileShader(vs);
    var fs = context.createShader(context.FRAGMENT_SHADER);
    context.shaderSource(fs, `
        precision mediump float;

        uniform vec4 u_color;
        uniform sampler2D u_texture;

        void main() {
            gl_FragColor = texture2D( u_texture, vec2( 0.5, 0.5 ) ) * u_color;
        }
    `);
    context.compileShader(fs);

    var program = context.createProgram();
    context.attachShader(program, vs);
    context.attachShader(program, fs);

    context.linkProgram(program);

    if (!context.getProgramParameter(program, context.LINK_STATUS)) {
        var info = context.getProgramInfoLog(program);
        throw "Could not compile WebGL program. \n\n" + info;
    }

    gl.useProgram(program);

    // look up where the vertex data needs to go.
    var positionLocation = gl.getAttribLocation(program, "a_position");
    var colorLoc = gl.getUniformLocation(program, "u_color");

    // provide texture coordinates for the rectangle.
    var positionBuffer = gl.createBuffer();
    gl.bindBuffer(gl.ARRAY_BUFFER, positionBuffer);
    gl.bufferData(
        gl.ARRAY_BUFFER,
        new Float32Array([
            -1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
        ]),
        gl.STATIC_DRAW
    );
    gl.enableVertexAttribArray(positionLocation);
    gl.vertexAttribPointer(positionLocation, 2, gl.FLOAT, false, 0, 0);

    var whiteTex = gl.createTexture();
    gl.bindTexture(gl.TEXTURE_2D, whiteTex);
    gl.texImage2D(
        gl.TEXTURE_2D,
        0,
        gl.RGBA,
        1,
        1,
        0,
        gl.RGBA,
        gl.UNSIGNED_BYTE,
        new Uint8Array([255, 255, 255, 255])
    );

    function test(format) {
        var tex = gl.createTexture();
        gl.bindTexture(gl.TEXTURE_2D, tex);
        gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, format, null);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
        gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);

        var fb = gl.createFramebuffer();
        gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
        gl.framebufferTexture2D(
            gl.FRAMEBUFFER,
            gl.COLOR_ATTACHMENT0,
            gl.TEXTURE_2D,
            tex,
            0
        );
        var status = gl.checkFramebufferStatus(gl.FRAMEBUFFER);
        if (status !== gl.FRAMEBUFFER_COMPLETE) {
            //log("can **NOT** render to " + glEnum(gl, format) + " texture");
            return false;
        }

        // Draw the rectangle.
        gl.bindTexture(gl.TEXTURE_2D, whiteTex);
        gl.uniform4fv(colorLoc, [0, 10, 20, 1]);
        gl.drawArrays(gl.TRIANGLES, 0, 6);

        gl.bindTexture(gl.TEXTURE_2D, tex);
        gl.bindFramebuffer(gl.FRAMEBUFFER, null);

        gl.clearColor(1, 0, 0, 1);
        gl.clear(gl.COLOR_BUFFER_BIT);

        gl.uniform4fv(colorLoc, [0, 1 / 10, 1 / 20, 1]);
        gl.drawArrays(gl.TRIANGLES, 0, 6);

        var pixel = new Uint8Array(4);
        gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);

        if (pixel[0] !== 0 || pixel[1] < 248 || pixel[2] < 248 || pixel[3] < 254) {
            // log(
            //   `FAIL!!!: Was not able to actually render to ${glEnum(
            //     gl,
            //     format
            //   )} texture`
            // );
            return false;
        } else {
            //log("succesfully rendered to " + glEnum(gl, format) + " texture");
            return true;
        }
    }

    return !test(gl.FLOAT);

}

It's probably less than ideal to add 140 lines of code to each GPGPU example. Do you have thoughts on how to proceed?

[spite]

The code is to make absolutely sure if there's support for Float or Half Float. Usually you would trust the result of the extensions, that's what they're for, and it's a much more compact code. But apparently there are differences between what you get from checking extensions and actually trying to render. Might be worth checking with the WebGL impl experts to know if that's still the case or not.

[jonobr1]

FWIW, Safari on iPad returns an object for extensions, but is unable to do float type renders on textures...

[jonobr1]

In this case, maybe the best course of action is to do a simple extensions check in the GPGPU examples and file a bug with webkit?

[spite]

SG. Although I don't think a feature detection import is that big of a deal, is it?

[jonobr1]

Yeah, I'm happy to PR something in that adds the checks. Originally WestLangley recommended I add it to each example individually. Now that we know the check involves a majority of Spite's code, should I continue to write in each example or should I write it elsewhere (e.g: GPUComputationRenderer)?

[spite]

I've been thinking of building a third party lib, like Modernizr, that tries to give a clear idea of what the device running your WebGL code can handle. WDYT?

[jonobr1]

Would love that!

[spite]

this would be mostly @greggman code. Gregg, do you have any plan regarding detection, or do you mind if we package most o those functions in a single lib?

[WestLangley]

Originally WestLangley recommended I add it to each example individually.

I think you may have misunderstood... I suggested calling the feature detection utility from the app, not from GPUComputationRenderer.

[greggman]

That check should not be needed in iOS 14? Also though what that code is doing is pretty simple. You could just three.js instead in probably far fewer lines of code.

[jonobr1]

That's awesome, so checking for extensions should suffice?

[greggman]

Well, maybe I spoke too soon. There's another bug related to precision that is supposedly fixed in iOS14

https://github.com/KhronosGroup/WebGL/pull/3061

But I don't think it's related to this one so you might still need the test. It's strange that this stuff doesn't work on iPadOS though. I'm pretty sure it used to work on iOS on the same device or maybe I just have a bad memory.

[jonobr1]

@greggman, I'm fuzzy on the details as well, but this did used to work. Perhaps it's related to iPads moving away from iOS and introducing iPad OS?

@WestLangley, I did misunderstand, sorry about that. When you say "app" do you mean in Three.js src?

[WestLangley]

@jonobr1 By "app" I mean the three.js example that calls GPUComputationRenderer. It could also refer to a user's app.

IMO, the feature detection to determine if float or half-float is supported belongs in the app... but it is possible someone else -- like @mrdoob -- disagrees, in which case you should follow his advice.

gpuCompute = new GPUComputationRenderer( WIDTH, WIDTH, renderer )
    .setDataType( THREE.HalfFloatType );

[jonobr1]

Got it, thanks for the clarification @WestLangley. In that case, I was thinking of the correct place to put the cannotRenderToFloatcode.

@spite, unless you want to make an app I'll update the current examples that use GPUComputationRenderer with the above check.

[spite]

SG. The detection/features lib might take some time :)

[jonobr1]

Okay, sounds good. As we know this is isolated to Safari, I'll have in my PR a one-liner to switch to THREE.HalfFloat for Safari browsers as an interim fix to your library @spite 👍