React-three-fiber: onPointerOut doesn't fire when moving to an abutting object

[dangrabcad]

I made a sandbox to demonstrate the differences between R3F pointer events and native DOM. (The ts-ignore in the sandbox are there because onPointerEnter and onPointerLeave seem to be missing from the EventHandlers type in types.ts: I can make a PR to add that if that's helpful.)

To see the DOM event behaviour, move the mouse onto the A at the top, then from A directly to B (there's no gap between them). The move from A to B gives these events:

DOM out A 
DOM child leave A 
DOM child enter B 
DOM over B 

You can see the equivalent situation in R3F below. Move your pointer onto the red cube, then directly onto the blue cube. Again, there's no gap between them. When you move from red to blue, you get these events:

R3F child enter bravo 
R3F child leave alpha

Note that the leave and enter are the opposite way around compared to the native events; more on this later. More importantly, note the absence of the "out" event. This comes later, when you move out from the blue cube onto empty space:

R3F out alpha 
R3F leave alpha 
R3F child leave bravo

The top-level leave event is on the wrong element: it should be bravo, not alpha. And the out event only comes now. Bravo (the blue cube) never got an over or out event.

Looking at canvas.tsx, the order is clearly because handlePointerCancel only happens at the end of the event processing, and I think the lack of events on bravo might be because this line is comparing the eventObject instead of the object. Using the eventObject would be correct for enter/leave, where you're not supposed to get events for boundaries inside the element that has the event handler, but you are supposed to get them for over/out, so I think it needs to compare object instead.

It might be worth noting that I only tried using onPointerOver/Out because of the reverse order of onPointerEnter/Leave. In my application, I need to handle these events by setting a state that contains the currently hovered object, or empty if no object is hovered. First I tried to use an onPointerEnter/Leave handler on the child objects, but then I got a bug with overlapping objects, because the late onPointerLeave would clear my hovered object state after it was set to the new object. I checked the W3C specs for pointer and mouse events and couldn't find anything saying which order they should be in, so I don't know if the behaviour is wrong (from the POV of not matching the specified behaviour of native events), but it's definitely inconvenient this way around. I tried using onPointerOver/Out on my root node instead as a workaround, but then hit this bug.

[dangrabcad]

I'm having a look at the code for this right now, with a view to fixing it, but I'm confused by the pointer event emitter in the SharedCanvasContext. This doesn't appear to be used inside the R3F code, and it's not documented in the public API, so what is it for? It's not clear what the behavior of it should be if the event order matched the DOM or if out/leave were treated separately like they should be.

[dangrabcad]

I also noticed pointer capture doesn't work anything like in the DOM. With the DOM, pointer capture means that only the object that captured the pointer gets further events: there are no out/leaves, and no over/enters for other objects until the pointer is released, which happens automatically onPointerUp or onPointerCancel. But in R3F, pointer capture doesn't stop other objects getting events.

[mvp-v]

@dangrabcad I have same issues with OVER/OUT events order. What about adding a new argument to handleIntersects callback? It'll be a function called before intersections loop.
In this case we add opportunity to call handlePointerCancel function before raising MOVE/OVER events.
Also we could try to split handleIntersects into getIntersects + proceedIntersects callbacks and insert handlePointerCancel between

[drcmda]

thanks everyone, so glad this is fixed!

[dangrabcad]

The linked PR is a good improvement - and should let me do what I need with R3F - but it only fixes the order, not the issue in the title, so this issue shouldn't be closed.

[drcmda]

yeah, turns out this isn't fixed, even the order. made a new testcase for that and i can repoduce cases where it still fails. :-S

im on it, but would be happy for any help i can get. events are so hard to get right.

btw @dangrabcad you don't happen to work for the actual grabcad?

[dangrabcad]

Yes, I work for the actual GrabCAD and I'm hoping we can use this library in one of our products. After reporting the bug I've started working on a change to improve the pointer events in general - making over/out and enter/leave different like in DOM, and making pointer capture work more like DOM - but it's nowhere near ready and I hope mvp-v's change will be enough to get me going for now.

[drcmda]

curiously we just bumped into the same problem in our software that bases on r3f. but either way, let me know if you find anything. i'll leave the fix in, but will revert locally just to see if i can work out the root issue. generally i'd be glad to receive upstream fixes for the event system, especially capture which is only a approximation that holds for out-of-target/browser pointerinput but does behave differently otherwise. if you need stakes or some kind security for longevity - i can offer you access as we have done with others that depend on it for business (ueno).

[drcmda]

i think i have a solution @dangrabcad

the order is: red > blue > red > out

R3F child enter alpha
R3F over alpha
R3F enter alpha
R3F child leave alpha
R3F child enter bravo
R3F child leave bravo
R3F child enter alpha
R3F out alpha
R3F leave alpha

im not entirely sure about the difference between enter/over leave/out. i run a few more tests and publish later on but looks promising on my end. our own test is fulfilled as well.

[drcmda]

published a new patch version: 5.1.1

my testcase: https://codesandbox.io/s/pointeroverorder-r3f-forked-nkvii?file=/src/App.js

red > green > red > out, yields ...

Boxred pointerOver
Boxred pointerOut
Boxgreen pointerOver
Boxgreen pointerOut
Boxred pointerOver
Boxred pointerOut

and without stoppropagation:

Boxred pointerOver
Boxgreen pointerOver
Boxred pointerOut
Boxred pointerOver
Boxgreen pointerOut
Boxred pointerOut

both seem correct.

[mvp-v]

If we need a DOM-like behaviour when parent doesn't receive events when pointer is on children, we can use filter callback and return only intersection with nearest object. First object is nearest for meshes, but when we are using Points, we should check other intersections' _distanceToRay_
https://codesandbox.io/s/pointeroverorder-r3f-forked-d1gb1?file=/src/App.js

[dangrabcad]

red > green > red > out, yields ...

Boxred pointerOver
Boxred pointerOut
Boxgreen pointerOver
Boxgreen pointerOut
Boxred pointerOver
Boxred pointerOut

I find this behaviour somewhat surprising. With the mental model that (a) each pointer move does a new hit test, (b) an event is dispatched to all objects intersecting the ray, even if there are other objects in front, until stopPropagation is called, and (c) stopPropagation affects the current event only; in that case, I'd expect that the first time the hit test returns [red, green], when red is already in the hovered set, green would be added to the hovered set and get sent pointerover, regardless of the stopPropagation of any previous pointerover events.

We can't use being DOM-like as a guide to what's correct here, because the case of events going through objects to other events doesn't arise. DOM doesn't send events to elements that are behind others in the z order: propagation only happens in the parent-child dimension. R3F has two dimensions for event propagation: parent-child, and distance along ray.

The behaviour you've described is the more useful behaviour for my use-case, but it only makes sense to me if events don't go through objects i.e. they only go to the first intersection (and its ancestors).

[drcmda]

yes, it happens because of stoppropagation, otherwise you are right, this will follow the natural raycasting order. the latest patch simply follows where @mvp-v has begun. he made it so that cancel will happen before move, but missed stoppropagation, which also calls cancel.

one case where the old behaviour really messed things up is: https://codesandbox.io/s/aesop-bottles-9p4j7?file=/src/Bottles.js:1062-1237

moving over the bottles you notice how it gets stuck sometimes. this is because bottle B hovers, previous bottle A unhovers and overwrites the body cursor. this should behave better now since events are ordered.

i suspect that dealing with events will be a never ending issue. please check it against your work and see if that makes it better. if not we have to iterate.

[dangrabcad]

With this change, if you stopPropagation from onPointerOut, it can cause an infinite recursion of onPointerOut calls. Here's a modification to your sandbox which shows it. Mouse over green box, then red box will trigger it (but red -> green -> red or red -> green -> background is fine).

[dangrabcad]

Also, in my application, if I don't have stopPropagation in onPointerOut, I still get the wrong order for the green -> red boundary, even though it works fine in the CSB. I'm not sure why yet, but it could be because my event handler is changing other state which causes a bunch of objects to re-render.

[drcmda]

stoppropagation in pointerout makes no sense imo, that could leave others events hanging, but it shouldn't cause a loop. will fix. are you sure everything's updated in your local project? if you click the download button in csb and run it locally does it work?

[dangrabcad]

I'm sure everything's updated because I didn't get the infinite recursion before I updated. It's end of day for me here so I'll have to debug it tomorrow.

[stephencorwin]

fwiw, in 5.1.1, I also experienced the onPointerOut recursion error. I am still getting some issues when 2 objects are adjacent to each other. I'm hoping it is some thing to do with my logic, but I feel like it might be what @drcmda mentioned about the bottles getting stuck sometimes.

Hovering over 1 square and then moving to an adjacent one does not change the tooltip.

https://dl.dropboxusercontent.com/s/nscwpxbo3rmcm21/2020-10-21_13-14-05.mp4

[drcmda]

the recursion error is gone in 5.1.3. stupid mistake: https://github.com/pmndrs/react-three-fiber/commit/60e2e84dd313e00e8321197d15235e989ff9f497#diff-b0d797706f31f5a1086042021ee1b1ee216dda744096d693017218ecd38cf2f4

updated csb with stoppropagation in pointerout: https://codesandbox.io/s/pointeroverorder-r3f-forked-1v67p?file=/src/App.js

[dangrabcad]

With a bunch of tracing in the R3F code I figured out why the results are different. handlePointerMove doesn't call onPointerOut at all for the red -> green case: it's actually called inside the stopPropagation of the pointerover event. If you change the CSB like this:

onPointerOver={(e) => {
        console.log(`Box${color} pointerOver`); e.stopPropagation()
      } }

so that the stopPropagation call happens after the log, then they come out in the wrong order. It's because the stopped logic in handlePointerMove's handleIntersects callback doesn't look at the order of the intersections. As far as it's concerned, the red -> green case and the green -> red case are identical even though the red box is in front and the expected behaviour inside the overlap region is different.

I also find it highly surprising that stopPropagation should dispatch events. It's pretty backwards from how the method works on DOM events. This whole over/out business would be a lot simpler if events only went to the first object the ray intersects, not objects behind. This would be a nuisance some cases but there could be an extra prop to make objects transparent to mouse events, like pointer-events: none in CSS.

[drcmda]

here's the reason we decided to implement it that way: https://codesandbox.io/s/r3f-basic-demo-forked-5pkvt?file=/src/App.js

if you go over the right spheres, the central pink one had to shut off the gray one, although gray technically is still underneath. by stopping propagation hover states other than the current one need to be flushed - or else they would stay hovered. this was a critical change that we discussed for weeks until we pulled the trigger. i don't know about the dom, but i do believe the behaviour was similar, at least i've seen comparable demos where these issues never happend - the underlying element would not stay hovered - but if that happens because stoppropagation calls the event or something else that i do not know.

r3f doesn't have to be a 100% close to the dom, it's a cross platform lib anyway and has a different environment to deal with. it just needs to lend itself to real use-cases. with the current implementation you can highlight two overlaying surfaces, or the one that's closer to the camera with stoppropagation, that's a good compromise imo.

[dangrabcad]

i don't know about the dom, but i do believe the behaviour was similar, at least i've seen comparable demos where these issues never happend - the underlying element would not stay hovered - but if that happens because stoppropagation calls the event or something else that i do not know.

Events just don't go to elements behind others in the DOM, unless the front/occluding element completely ignores events with pointer-events: none. stopPropagation controls the way events bubble to the target element's ancestors. If R3F worked the same way, the spheres example would Just Work regardless of whether you use stopPropagation.

I'd be fine with the way the behaviour works now if (a) stopPropagation didn't have the same name as a DOM method that does something completely different, and (b) there were some documentation for the current behaviour so that developers can know how to stop it. As it stands it's a footgun. If it's taken me six weeks to find out the intended behaviour, most developers won't wait that long.

[drcmda]

more documentation would be great. changing sP now would be a hard breaking change. better just explaining what the difference is and how it works in relation to the internal raycaster which it wraps into pointerevents. it does stop propagation after all, shutting off the raycaster even if it would pierce additional objects.

i think developers will come to see that there is otherwise no solution at all since none of this stuff exists in three. they'd have to re-create a bubbling event system from scratch - which in r3f was easily the hardest part. but that's also why we have issues, discussions and discord - to get in touch and sort it out. i'll think about something.

do you want to help put these differences into words? i know too little about how the dom fares.

[dangrabcad]

That might be something I can help with. I can put in something about pointer capture as well - adding the capturing element to the list of intersections rather than having it instead of the hit test is quite surprising, but it kind-of naturally follows on from the hit test going through objects.

[drcmda]

great! if you like open a pr, i'd say we put it directly into api.md, the events section

[dangrabcad]

I'm trying to write up the behaviour of stopPropagation. I was going to say that it blocks pointer events from objects behind it, but it also causes objects in front to be left, as in this fork of the red-green box example. Every pointermove in the red-green overlap region causes the red box to get pointerover and pointerout. This doesn't seem intended. The intended use case here would be that you want the red box to be transparent to pointer events but the green box to block pointer events.

[drcmda]

the current behaviour is that the closest object to the camera "wins", and it will flush all hover states except the winning object. that means closest gets over-state, everything else out-state - IF if was previously hovered.

if it wouldn't flush, then the hover state would be left hanging, and there would be no possibility to exclusively hover objects on top of other objects. with this behaviour both is possible: exclusive and inclusive hover.

Every pointermove in the red-green overlap region causes the red box to get pointerover and pointerout.

the correct behaviour would be red gets over, green out. red should absolutely not get out, if that is the case on your machine there is something broken. i just tested and at least on my machine it works as intented.

[dangrabcad]

I think I didn't save my change to the CSB: try it again now. I changed it so the green box (the one at the back) uses stopPropagation but the red one (at the front) does not.

[drcmda]

@dangrabcad i see it. i'm on it. also have prepared a more in depth api explanation that we can adapt and change later on. will publish it soon.

[dangrabcad]

I can send you a PR for api.md that covers what I have so far if that'll help.

[drcmda]

sure, that's great, send it pls :-)

[drcmda]

i started implementing headless snapshot testing based on the pr from @marcosparagna-eb that i adapted to latest r3f. from now on i think it should always make visual regression tests against changes to the pointer system. and it looks promising. i will push this with the fix when im through.

[drcmda]

@dangrabcad im going back and forth with the last example you posted, but can't make sense out of it. the left, red box doesn't stop propagation, so the pointer hits the green one underneath, which will then trigger hover, which stops propagation and flushes out red, calling pointer-out. this happens every mousemove since red is getting the event first.

is there a real fix to this? i kind of expect it to behave like that.

maybe ........ an object can only flush if it comes .... first in the stack? does that make sense? going into rule salad here if this isn't completely watertight and added to jest testing. what do you think?

[drcmda]

so, indeed, that works. we can define it like so:

• an event handler (for instance pointerOver) can only stop propagation if it is the first in the stack. e.stopPropagation() is a signal that the user wants just the first element to receive events. if an object already has caught the event, an object underneath should not be able to stop it itself.

OR, it should be able to stop it for higher up objects. that could also be a solution.

[drcmda]

it should now be fixed. the solution now makes sense to me: stopPropagation cannot flush out higher-up objects, period. i've also added a test for this so that it won't cause any surprises later on.

[dangrabcad]

As you've said before, the event-handling is a complex area, and I'd like to help arrive at something that works for everyone. I haven't had a chance to look at your new change yet, but I'll try it this week. My main worry is that it adds new tricky cases when the order of objects getting hit changes during a gesture. For example in my application I've got camera controls working by dragging on the background, but while you're moving the camera, foreground objects can become hovered. Though those foreground objects should normally handle pointer events themselves (to manipulate those objects), they shouldn't while a gesture is already in-progress. In 2D DOM I'd just rely on pointer capture to make that work, since it replaces the normal hit test, but will it with this rule? I'll have to try it out and see.

[drcmda]

i think my first idea (only top objs can stop) was flawed, but the current implementation imo is clean. it just means that an object cannot cancel/flush objects that have already gotten the event, only events that follow can be canceled. i've tried all possible variations with three intersecting boxes, wildly adding or omitting stoppropagation calls - and it behaves as i would expect it to. if something comes up let me know.