Three.js: Temporal Anti Aliasing (TAA) rounding errors over accumulation

[throni3git]

Description of the problem

I've discovered there is an implementation of MSAA accumulating over several frames, which is a nice thing. The example to this (webgl_postprocessing_taa) shows the beauty of this.

When i applied it to an application i work on, the surpression of the accumulated rounding errors didn't seem to work, when i use MeshPhongMaterial with it. That is because it is not implemented in the accumulating version of TAARenderPass. So i decided to integrate it myself, but unluckily there is still an issue when the accumulation is going on. The rendered frames go darker inbetween and recover light when the accumulateIndex reaches 32. Please check out my version here.

The inner loop of THREE.TAARenderPass.render in my version is

var baseSampleWeight = 1.0 / jitterOffsets.length;
var roundingRange = 1 / 32;

if( this.accumulateIndex >= 0 && this.accumulateIndex < jitterOffsets.length ) {

    this.copyUniforms[ "opacity" ].value = sampleWeight;
    this.copyUniforms[ "tDiffuse" ].value = writeBuffer.texture;

    // render the scene multiple times, each slightly jitter offset from the last and accumulate the results.
    var numSamplesPerFrame = Math.pow( 2, this.sampleLevel );
    for ( var i = 0; i < numSamplesPerFrame; i ++ ) {

        var j = this.accumulateIndex;
        var jitterOffset = jitterOffsets[j];
        if ( this.camera.setViewOffset ) {
            this.camera.setViewOffset( readBuffer.width, readBuffer.height,
                jitterOffset[ 0 ] * 0.0625*1.25, jitterOffset[ 1 ] * 0.0625*1.25,   // 0.0625 = 1 / 16
                readBuffer.width, readBuffer.height );
        }

        var sampleWeight = baseSampleWeight;
        if( this.unbiased ) {
            // also apply unbiased accumulation here
            var uniformCenteredDistribution = ( -0.5 + ( this.accumulateIndex + 0.5 ) / jitterOffsets.length );
            sampleWeight += roundingRange * uniformCenteredDistribution;
        }
        this.copyUniforms[ "opacity" ].value = sampleWeight;

        renderer.render( this.scene, this.camera, writeBuffer, true );
        renderer.render( this.scene2, this.camera2, this.sampleRenderTarget, ( this.accumulateIndex === 0 ) );

        this.accumulateIndex ++;
        if( this.accumulateIndex >= jitterOffsets.length ) break;
    }

    if ( this.camera.clearViewOffset ) this.camera.clearViewOffset();

}

By the way, can we further surpress the rounding error artifacts? The sphere looks somewhat quantized in the darker areas.
Greetings, Thomas

Three.js version

• [ ] Dev
• [x] r77 as of 2016-05-30

[arose]

By the way, can we further surpress the rounding error artifacts? The sphere looks somewhat quantized in the darker areas.

Using THREE.HalfFloatType or THREE.FloatType (when available via OES_texture_half_float and OES_texture_float) helps with that. As far as I can tell HalfFloatType is sufficient. @bhouston why exactly did you stayed with 8-bit per color channel? Compatibility? Performance?

[bhouston]

Compatibility? Performance?

Yes. Yes. And don't forget lower memory usage (by 2x or 4x compared to half / float.) :)

[bhouston]

BTW the bug that is causing this behavior is because of the way the intermediate TAA accumulated frame is blending with the final frame. It should be using the sampleWeight that is computed from the accumulated weights. I suspect you didn't update this line of code to use actually assumulated weights -- if you blend in the intermediate frame with a weight that isn't representative of the accumulated weights you used to create it, then it will be wrong:

https://github.com/mrdoob/three.js/blob/master/examples/js/postprocessing/TAARenderPass.js#L109

[bhouston]

To be clear, this line here:

https://github.com/mrdoob/three.js/blob/master/examples/js/postprocessing/TAARenderPass.js#L101

Assumes that the accumulatedWeight is not created by weights that vary on a per frame basis. Thus it needs to use what is truely accumulated by the varying weights that are used to avoid the rounding errors.

[throni3git]

I also tried that. I deactivated the weighting in the inner loop therefore since it should also be possible to do it in the "outer" loop, meaning over several frames. My code is

var sampleWeight = baseSampleWeight;

var accumulationWeight = this.accumulateIndex * sampleWeight;

if( this.unbiased ) {
    // also apply unbiased accumulation here
    var uniformCenteredDistribution = ( -0.5 + ( this.accumulateIndex + 1 ) / jitterOffsets.length );
    accumulationWeight += 1 / 32 * uniformCenteredDistribution;
}

if( this.accumulateIndex > 0 ) {
    this.copyUniforms[ "opacity" ].value = 1.0;
    this.copyUniforms[ "tDiffuse" ].value = this.sampleRenderTarget.texture;
    renderer.render( this.scene2, this.camera2, writeBuffer, true );
}
if( this.accumulateIndex < 32 ) {
    this.copyUniforms[ "opacity" ].value = 1.0 - accumulationWeight;
    this.copyUniforms[ "tDiffuse" ].value = this.holdRenderTarget.texture;
    renderer.render( this.scene2, this.camera2, writeBuffer, ( this.accumulateIndex === 0 ) );
}

Note that the conditions are related to the accumulateIndex, rather than relying on accumulationWeight like it was done before...

new Example

So, how to modify the accumulationWeight in an unbiased fashion?

[bhouston]

The rounding happens in the inner loop.

[throni3git]

I reassembled the TAARenderPass to work around the biasing problem. It now copies a new sample with 1/(this.accumulateIndex+1) and 1-1/(this.accumulateIndex+1) of the old image to the writeBuffer, saving it for the next pass.

• background color is determined by CSS, having the ClearColorfully transparent and the alpha for the renderer turned on
• TAARenderPass has a function setAccumulation( doAccumulation ) to set this.accumulate and reset this.accumulationIndex
• also has a function resetAccumulationIndex()
• i modified the example file webgl_processing_taa.html to include an OrbitController, instead of a stop-and-go rotation of the cube and also added a Phong shaded Sphere

Have a look here:
Example

Let me know what you think, this can be turned into a pull request then

[bhouston]

This is nice and I guess you adapted it to the new clearColor, clearAlpha support at I added in ManualMSAARenderPass just recently? Sweet.

[bhouston]

I'm referring to this recent modification of ManualMSAARenderPass: https://github.com/mrdoob/three.js/pull/9124

[throni3git]

Thanks. I tried to incorporate clearColor support, but it doesn't work out as i think it should work. Look at the refreshed example. In CSS, i load an image and the clearColor i've set is red with 0.5 alpha. I think, the meshes should have a color not influenced by clearColor, whilst the parts that are not shaded would turn slightly red. Doesn't work... Full code is here, the essential part is here:

var autoClear = renderer.autoClear;
renderer.autoClear = false;

var oldClearColorHex = renderer.getClearColor().getHex();
var oldClearAlpha = renderer.getClearAlpha();

if( this.accumulateIndex < totalSamples ) {

    // render the scene multiple times, each slightly jitter offset from the last and accumulate the results.
    for ( var i = 0; i < numSamplesPerFrame; i ++ ) {

        var jitterOffset = jitterOffsets[ this.accumulateIndex ];

        if ( this.camera.setViewOffset ) {

            this.camera.setViewOffset( readBuffer.width, readBuffer.height,
                jitterOffset[ 0 ] * 0.0625, jitterOffset[ 1 ] * 0.0625, // 0.0625 = 1 / 16
                readBuffer.width, readBuffer.height );

        }

        // render on transparent rendertarget
        renderer.setClearColor( 0x000000, 0.0 );
        renderer.render( this.scene, this.camera, this.sampleRenderTarget, true );

        this.copyUniforms[ "tDiffuse" ].value = this.sampleRenderTarget.texture;
        this.copyUniforms[ "opacity" ].value = 1.0 / ( this.accumulateIndex + 1 );
        renderer.render( this.scene2, this.camera2, writeBuffer, true );

        this.copyUniforms[ "tDiffuse" ].value = this.holdRenderTarget.texture;
        this.copyUniforms[ "opacity" ].value = 1.0 - 1.0 / ( this.accumulateIndex + 1 );
        renderer.render( this.scene2, this.camera2, writeBuffer, false );

        this.copyUniforms[ "tDiffuse" ].value = writeBuffer.texture;
        this.copyUniforms[ "opacity" ].value = 1.0;         
        renderer.render( this.scene2, this.camera2, this.holdRenderTarget, true );

        this.accumulateIndex ++;

    }

    // now bring up clearColor
    renderer.setClearColor( oldClearColorHex, oldClearAlpha );
    this.copyUniforms[ "tDiffuse" ].value = this.holdRenderTarget.texture;
    this.copyUniforms[ "opacity" ].value = 1.0;
    renderer.render( this.scene2, this.camera2, writeBuffer, true );

    if ( this.camera.clearViewOffset )
        this.camera.clearViewOffset();

    if( this.onRenderedCallback && this.accumulateIndex >= totalSamples ) {

        this.onRenderedCallback();

    }

}

renderer.autoClear = autoClear;
renderer.setClearColor( oldClearColorHex, oldClearAlpha );

[throni3git]

@bhouston any clues?

[throni3git]

Setting blending mode to NormalBlending did the deal for me: now the clearColor and clearAlpha given to the renderer is used.

The refreshed example can be visited here.

    // now blend with clearColor
    this.copyMaterial.blending = THREE.NormalBlending;
    renderer.setClearColor( oldClearColorHex, oldClearAlpha );
    this.copyUniforms[ "tDiffuse" ].value = this.holdRenderTarget.texture;
    this.copyUniforms[ "opacity" ].value = 1.0;
    renderer.render( this.scene2, this.camera2, writeBuffer, true );
    this.copyMaterial.blending = THREE.AdditiveBlending;

[bhouston]

@throni3git Nice! BTW if we want to have unlimited number of sampels or break out of the fixed pre-set number of MSAA samples we are using, we could convert to using Halton sampling: https://github.com/mrdoob/three.js/issues/9256

[Rayfoundry]

This does not seem to work when the Pass is used in a EffectComposer chain with other effects following. Any idea?

[bhouston]

This does not seem to work when the Pass is used in a EffectComposer chain with other effects following. Any idea?

You would have to jitter the camera through the whole effect composer chain and accumulate. I've been meaning to implement this for a while but haven't had the time or a paying client ask for this effort.

[throni3git]

I didn't test it for a more complex effect chain. What effect are you trying to use? Do you have a fiddle for that?

[Rayfoundry]

I didn't test it for a more complex effect chain. What effect are you trying to use? Do you have a fiddle for that?

See https://jsfiddle.net/rayfoundry/jxku52re/

Took me a while to assemble something. It's basically ThreeJS r79 with the stock examples + your TAA implementation. I've added two posteffects after the TAA step. The issue is that irrespective of which effect is first after TAA it "bleeds".

P.S.: The cube rotates after 2 seconds, which deactivates the TAA accumulation. It's still sampling at level 2 though and using TAA. Just without accumulate. The display is correct then.

[ruben3d]

I found that the issue with stacking effects using the TAARenderPass by @throni3git is fixed if the holdRenderTarget is copied always to the writeBuffer, not just when it is accumulating samples, so the other effects in the stack are always applied on top of the clean accumulated image, instead of the result of the prev frame stack.

I have tested it in a complex scene with RGBShift and the new UnrealBloom and works perfectly.

Here is a quick fiddle based on @Rayfoundry 's fiddle and the original TAA example:
https://jsfiddle.net/ruben3d/dtdvkfph/