Pixi.js: Filters in retina display have wrong UV

[andrevenancio]

I'm using pixi 5.1.1

Writting a simple vignette shader I can see that the build in varyings representing the UV's are wrong on the .y component.

If my display is non retina or if it is retina but I initialise the Application with resolution = 1
The shader is as expected. if I'm on a retina display then the .y needs to be divided by 2

const noRetina = true; // manually change the resolution on a retina display (on/off)
const opts = {
    width: 1920,
    height: 1080,
    resolution: noRetina ? 1 : global.devicePixelRatio,
    autoDensity: !noRetina,
    backgroundColor: 0x000000,
    antialias: false,
    view,
};

const app = new Application(opts);

Fragment shader

precision mediump float;

varying vec2 vTextureCoord;
uniform sampler2D uSampler;

uniform float size;
uniform float amount;

void main() {
    vec3 color = texture2D(uSampler, vTextureCoord).rgb;

    float dist = distance(vTextureCoord, vec2(0.5, 0.5));
    color *= smoothstep(0.8, size * 0.799, dist * (1.0 * amount + size));

    gl_FragColor = vec4(color, 1.0);

}

[ivanpopelyshev]

float dist = distance(vTextureCoord, vec2(0.5, 0.5));

That's where its wrong. vTextureCoord is normalized input coord , and input is temporary pow2 texture, its size is not proportional to your filter area, it can be anything. Use convert function

https://github.com/pixijs/pixi.js/wiki/v5-Creating-filters#conversion-functions

vTextureCoord * inputSize.xy / outputFrame.zw // filter (normalized)

Please read the article to know more about filters, before you ask next questions.

Also, in case of RETINA, you might use global constant for filters to specify temporary buffer resolution:

PIXI.settings.FILTER_RESOLUTION = 2;

Another thing is - do you really need filter? Filter works on containers with many elements, first it renders into temp framebuffer, then renders that framebuffer with your shader. In case you need only one element rendered - you can use mesh shader. https://pixijs.io/examples/#/mesh-and-shaders/triangle-textured.js

Oh right, full-screen shader, usually means you need filter.

[ivanpopelyshev]

Actually, it might be that without retina you were in fullscreen mode, that means input coords were equal to filter coords and everything worked fine. With retina enabled, but filter resolution still 1, fullscreen mode switched off and you've got pow2 input coords. If you set PIXI.settings.FILTER_RESOLUTION or that particular filter.resolution, you'll enable fullscreen again.

Basically

filter.resolution = MY_RESOLUTION;
container.filterArea = renderer.screen;

that switches full-screen mode on, temporary buffer size is the same as screen, and input coords = filter coords.

[andrevenancio]

Hi @ivanpopelyshev
thanks for your reply.

Let's do this by parts, I don't want to extend this issue longer than it needs be.

The screenshots above were taken with the exact same screen with the exact same width and height, only the resolution and autoDensity properties changed.

Assuming I was creating a 1920px x 1080px canvas with resolution=1 and autoDensity=false everything seems to be "on scale". The vignette gets "distorted" as its mapping the corners of the screen. So far, so good.

If I then changed resolution=2 and autoDensity=true I would have a canvas with 3840px x 2160px that was scaled down via css to 1920px x 1080px. Shouldn't I have the same looking vignette? What I have now is a perfect circle, not a distorted one. So its either the autoDensity or the resolution enabling such difference no?

Another thing is - do you really need filter?

I think I do, the effect above is just so I can illustrate that the UV coordinates I'm getting aren't the ones I'm expecting. The effect I want to achieve is a mix of what's on the stage, and some post processing going on top. So I do need an input texture of what's on the screen (uniform sampler2D uSampler;). I just covered it red so I'm not sharing clients confidential work :)

For example, applying a filter to the stage, if the content gets drawn into a framebuffer and then used as a texture via uniform sampler2D uSampler; shouldn't I be able to properly see the UV's mapping the corners of the canvas element by doing something like: gl_FragColor = vec4(vTextureCoord, 1.0, 1.0); ?

I do appreciate you took the time to explain a little bit about conversion functions, but reading that article Its not quite clear (for me) what varying to use and when to "convert them". I would be expecting something like what we can do in ShaderToy vec2 uv = fragCoord / iResolution.xy; to map with the corners of my canvas element and It seems like vTextureCoord only does this if autoDensity=false and resolution=1 (first screenshot).

I'm happy to accept that I'm missing something, and all the information in in that article, but maybe just use this thread as a constructive criticism that the article might not be as clear as it could be.

[ivanpopelyshev]

Shouldn't I have the same looking vignette? What I have now is a perfect circle, not a distorted one. So its either the autoDensity or the resolution enabling such difference no?

Temporary buffer is different :) in first case it was screen-sized and vTextureCoord was from 0 to 1. In second it was from 0 to "filterArea.width/input.width".

Two ways of solving:

1. Add a conversion function there before you calculate dist.
2. set FILTER_RESOLUTION or filter.resolution and pixi will automatically enable "temporary buffer of fullscreen-size" again.

I don't know how to make article be more clear, because the topic itself is difficult. Yes, there are topics that take some time from user to understand even if you read multiple explanations. It just takes time and effort.

That's how input filter size is determined: https://github.com/pixijs/pixi.js/blob/dev/packages/core/src/renderTexture/RenderTexturePool.js#L75

If pixel width and height of screen and filter is same, it takes screen-sized buffer. Otherwise pow2-ed, because it takes time to resize texture buffer and its really better to pool them by size.

In your case pixel width and height of filter was twice smaller because filter resolution is not the same as renderer resolution.

Btw, resolution itself is a difficult topic, even without filters.

[ivanpopelyshev]

Another way of solving: https://github.com/pixijs/pixi.js/blob/dev/packages/core/src/filters/defaultFilter.vert - you see, vertex shader already has aVertexPosition which is filter normalized and not input normalized. You can copy vertex shader and pass this thing to fragment through varying - and that's what you have to use for dist argument.

That difference between input coord and filter coord I explained soo many times , i think almost every topic about filters in here (pixijs issues) and https://www.html5gamedevs.com/forum/15-pixijs/ has it.

Maybe that's the way of understanding:

1. Understand that filter actually takes extra framebuffer/texture to render container inside, before your shader is applied
2. Understand that buffer size can be different from the filtered area due to optimizations.

Pixi allows users to not now those things if people use containers that cover full screen. As soon as something is changed - container is partial screen, or screen resolution is different - pixi tries to optimize it and forces user to dig into docs. That's when person comes to forum and asks "filter does whaaaaat?". This pain was introduced in pixi-v4, and at least now we have full documentation on it.

[ivanpopelyshev]

And yes, the big problem is that we still didnt make article for people who come from shadertoy and want to port existing shaders. fragCoord / iResolution.xy wont work here because filter can be applied to parts of screen.

Another pain appears when you use resolution. In that case inputSize and inputPixel uniforms are different, and if you want to take neighbour pixel in your filter shader - you have to look at pixels and not usual CSS size.

Btw, here is the thread back from 2016, where bqvle made his vignettefilter: https://github.com/pixijs/pixi.js/issues/2572

[ivanpopelyshev]

I think "set resolution of filter" is better solution for your case (4K), because it wont use extra 4096x4096 texture. Btw, in case of 4k, its better to disable the antialiasing: if you set "antialias:true" in renderer initialization - turn it off. It kills macs by using MSAAx8, and its obsolete in case of full-screen filters, because, well, filters are not antialiased and any geometries you render inside filter cant use it.

[andrevenancio]

Hey @ivanpopelyshev
thanks so much for your detail explanations. I'm going to digest this information and close the ticket because as I said before its probably me not fully following the article.

You as a core member of the developer team have way more insight on why some things were made the way they were made (like you explain above, for batching for example), and you are right: filters and resolution are quite big topics.

Hopefully coming into this issue will make me dig more into the code and will have a clearer idea as I continue to work on this project.

Enjoy your weekend!

[ivanpopelyshev]

Wait, I remember one more cool thing!

If you need something a vignette , that just "masks" container inside , you can just draw a mesh on top, without filters . Mesh is easier to make and it wont require extra framebuffer: https://pixijs.io/examples/#/mesh-and-shaders/triangle-textured.js