P5.js: Mouse does not track correctly with WEBGL renderer

[catarak]

Nature of issue?

• [X] Found a bug

Most appropriate sub-area of p5.js?

• [ ] Color
• [ ] Core/Environment/Rendering
• [ ] Data
• [ ] Events
• [ ] Image
• [ ] IO
• [ ] Math
• [ ] Typography
• [ ] Utilities
• [X] WebGL
• [ ] Other (specify if possible)

Which platform were you using when you encountered this?

• [ ] Mobile/Tablet (touch devices)
• [X] Desktop/Laptop
• [ ] Others (specify if possible)

Details about the bug:

As reported by @ajavamind in processing/p5.js-web-editor#1074:

Bug: Mouse tracking with the WEBGL renderer display works differently than P2D.
This code should give same result when using WEBGL or default in createCanvas.

function setup() {
createCanvas(800, 600,WEBGL); // not ok
//createCanvas(800,600); // ok
background(100);
}

function draw() {
ellipse(mouseX, mouseY, 20, 20);
}

Browser Chrome on windows 10 pro
Google Chrome is up to date
Version 74.0.3729.131 (Official Build) (64-bit)

[stalgiag]

If you change the ellipse() use to:

ellipse(mouseX - width/2, mouseY - height/2, 20, 20);

you get expected behavior. This is because the coord system is different in WebGL (ie 0, 0 is at the middle of the screen) which was an early design decision. I think it was done this way to make it possible for people to make smoother transitions from p5 to other WebGL use. I am personally not sure about it but it hasn't been challenged very strongly.

Perhaps though a middle ground would be modifying mouse coords while in WebGL? This might cause more confusion.

[ajavamind]

Thanks for all your help with this issue. Maybe the solution is for the p5.js library to compute mouseX and mouseY based on the renderer chosen for the sketch. The mouse coordinates returned should match the coordinate system for WEBGL.

[stalgiag]

@ajavamind I partially agree that this should be the case but it sort of undermines the sense that mouseX and mouseY are just numbers that correlate to pixels on your screen which is a 2D plane. The GL camera can rotate and this complicates all of this as well. I feel like adjusting some numbers to match the 2d coordinate system and not others becomes a tricky issue.

Like if I am a user and I am using mouseX and mouseY to decide where to put a shape and I decide that it is somewhere near the top left corner. I might be very confused when I swap out the mouse coordinates in my shape method for a number like 10. It becomes a question of what stage we should reveal the coordinate system.

[catarak]

I was looking at the documentation for mouseX and mouseY and I noticed it says that these values are "relative to (0,0) of the canvas." If you apply a translate() before using mouseX and mouseY (the documentation hints that it updates the (0,0) position) for translate, then the values of mouseX and mouseY then are out of sync with the position of (0,0). But it also seems a little crazy to update mouseX and mouseY with every translate()/rotate(). It does feel confusing that by default, in WebGL mode, mouseX and mouseY don't match up with the actual mouse position. Are mouseX and mouseY actually just relative to the upper lefthand corner of the canvas? Should the documentation be more clear about this?

[stalgiag]

I definitely agree that the documentation should be more clear about this. I am not entirely sure about other changes. I have felt kind of displeased with the fact that (0,0) is different in GL mode for a long time but I have usually felt reluctant to advocate changing it because of the number of sketches that would break as a result.

Anyways I am up for discussing that as a possibility. Otherwise, we can just make the documentation change the todo for this issue and call it a day. 🤷‍♀

[catarak]

My vote is for simply updating the documentation, at least for now!

[stalgiag]

So the todo here would be to:
Change the description for

• mouseX
• mouseY
• pmouseX
• pmouseY

to more accurately state that the values are relative to the upper-left corner of the canvas. Potentially there could also be a new sentence that startes that the value at the top-left corner is (0,0) for 2D and (-width/2, -height/2) for WebGL. This might crowd and complicate the description though so I will leave it at the discretion of the person making the changes and if need be that can be discussed in the PR.

[stalgiag]

closed via #3798