Pixi.js: Custom drag algo miscalculates first touch coordinates only in iOS.

[outisfun]

Hello,

I am building a demo based on this cool example.
Live example here.

I rewrote it in PIXI v5, but an issue that's also present in the original, persists.
The drag and smudge feature works without problems on Android and on desktop, but on IOS, on the very first touch and drag, it takes not the pixels under the brush (finger), but in the top left corner of the screen. On each subsequent touch in a different spot, it seems to carry a "snapshot" of its previous position.

My guess is, this might have something to do with the vTextureCoord it receives, but I don't have much experience with WebGL, so that might be completely wrong.

vTextureCoord is calculated in the vertex shader like this:
{ return aVertexPosition * (outputFrame.zw * inputSize.zw); }

Screenshots on first touch and drag:
iOS
Android

Screenshots on second touch and drag:
iOS
Android

Here is the fragment shader code:

    precision highp float;
    varying vec2 vTextureCoord;
    uniform sampler2D uSampler;

    uniform vec2 brushPosition;
    uniform vec2 brushDrag;
    uniform float resx;
    uniform float resy;
    uniform float brushStrength;
    uniform float brushRadius;
    uniform float brushInverse;
    uniform float brushOpposite;

    void main(void) {
      vec2 uv = vTextureCoord.xy;
      vec2 pixelUnit = vec2(1.0/resx, 1.0/resy);
      vec2 res = vec2(resx, resy);
      vec2 center = uv - (brushPosition / res); 
      center.x *= resx/resy;
      float radius = brushRadius/resy;
      float dist = smoothstep(0.0, radius, length(center));
      dist = mix(step(0.999, dist), dist, 1.0);
      dist = mix(1.0 - dist, dist, brushInverse);
      dist = mix(dist, dist * -1., brushOpposite);
      uv = uv - brushDrag * dist * pixelUnit * brushStrength;

      vec4 color = texture2D(uSampler, uv);
      color = mix(color, color * 1.0, dist * clamp(brushStrength, 0.0, 1.0));

      gl_FragColor = color;
  }

... and the uniforms brushPosition and brushDrag (which are the only ones that change dynamically) are set in the ticker function, before the app renders.

Thanks a lot!

[ivanpopelyshev]

Yes, I can help with that but only in Sunday, I'm attending JS Conference.

Very good report, thank you!

[outisfun]

@ivanpopelyshev , that would be awesome, thank you!

[outisfun]

@ivanpopelyshev , did you by any chance have an opportunity to check the issue? I still can't seem to solve it, so whenever you have time, it would be wonderful if you could look at it, or at least give me a suggestion about what to look at. Thank you!

[ivanpopelyshev]

I'll check it tomorrow or the day after :)

[ivanpopelyshev]

Can you post it in codepen or jsfiddle, somewhere where i can edit it?

[outisfun]

Yes, I'll do it first thing tomorrow morning, I'm traveling this weekend.
Thank you!

On Fri, 15 Nov 2019, 20:37 Ivan Popelyshev [e-mail] wrote:

Can you post it in codepen or jsfiddle, somewhere where i can edit it?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pixijs/pixi.js/issues/6209?email_source=notifications&email_token=ADTUEE235S6TUGKL22OYHDTQT325ZA5CNFSM4JKHQCHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGPUWQ#issuecomment-554498650,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADTUEE5N57NNYRGBV2RC2GLQT325ZANCNFSM4JKHQCHA
.

[outisfun]

Here's a quick and dirty pen:
https://codepen.io/outis/pen/xxxmPNY

I tried to add only the stuff I find relevant, but if anything is unclear or missing, please let me know and I'll add it. Thank you! And have a great week.

[ivanpopelyshev]

I don't understand what's going on yet, I need to make it even more simpler. I'll try to borrow iOS phone , that way i'll be able to reduce example to something tiny, maybe just two "render" operations with even simpler shader.

Its potentially bug in pixi low-level or in webgl on iOS in general (it happened before).

You reproducing this on both safari and chrome, right?

[outisfun]

Thank you! I would be happy to simplify it further, just let me know which parts you want gone or highlighted. It's part of a larger demo so there's a few things that are not relevant, but the PIXI stuff is all in the pen.

And yes, it happens on both Safari and Chrome.

[ivanpopelyshev]

No, your code is good, but I need to cut down number of webgl operations seriously.

It'll be a large number of reductions, and each time i'll check if bug is gone or not. In the end it'll be very small. If bug persists, I will make pure webgl version of it too - that's how its usually goes with low-level bugs.

Alternatively, it can be just bug with interaction, wrong brush uniforms, something like that.

[ivanpopelyshev]

There's also very small chance that its a bug that also affecting security, because manipulation with Framebuffers is tricky, and sudden read from neighbour memory occured before :)

So its not Custom PIXI filter miscalculates first touch coordinates only in iOS. , its Custom drag miscalculates first touch coordinates only in iOS.

[ivanpopelyshev]

Im debugging interaction:

https://codepen.io/ivanpopelyshev/pen/rNNPyyz

if (this.isDragging && !this.failedDrag) {
  if (Math.abs(this.dragDelta.x) > 100 || Math.abs(this.dragDelta.y) > 100) {
    this.failedDrag = true;
    const debugSprite = new   PIXI.Sprite(PIXI.Texture.WHITE);
    debugSprite.scale.set(10);
    this.app.stage.addChild(debugSprite);
  }
}

if i drag mouse too fast - white square is shown. In iOS its shown as soon as i touch it - that means you have interaction bug and not something filter-related. I don't have any guarantees about pixi-viewport nor your dragging algorithms.

For more debugging, I need you to make it direct pixi interaction, that way we can confirm its a pixi bug and not your code (or pixi-viewport)

[outisfun]

Okay, thank you, that's a wonderful catch - I will try resolving it from within my code. It shouldn't be pixi-viewport, because the original code doesn't use it, and it has the same issue.

If the quick drag is the issue, then its absolute value can be clamped? - at least that's what comes to mind first, I'll play around a bit, interactions are much easier to debug. Thank you, that was very helpful! If I come across an issue that seems PIXI-related, I'll post another pen.

[ivanpopelyshev]

Its not quick drag, I made that code to see if FIRST touch in iOS can force it because previous brushPosition is (0,0). That means in iOS in first touch drag is going from point (0,0) to (inputX, inputY) - that's what i caught.

In general fast drag is ok, and you can fix it with doing several flip-flops along the path. In this case its different - its "first coordinate" bug.

Title should be Custom drag algo miscalculates first touch coordinates only in iOS.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.