React-three-fiber: Excluding objects in scene from raycast

[unphased]

We have large (CAD style) models which render great even with millions of triangles. We have elements in the scene which have very simple geometry that we would like to make clickable, however the click handling ties up the main thread iterating through all geometry indiscriminately on the CPU, even though no handlers are placed on the very heavy models.

It would be really nice to be able to exclude them.

[unphased]

somewhat related: if I attach a click or pointerdown/up events to the canvas, click handling appears to no longer work. So probably that's because all the R3F React style mouse picking is implemented internally naively.

I also noticed that initiating a click outside of the bounding box of large heavy models does see an avoidance of sluggishness on the mousedown.

I also noticed that if i override both onclick and onpointerdown to the canvas, it eliminates the sluggishness on mousedown near large models, so it seems to avoid scanning that geometry. That said, there's still dblclick that is hooked and the right click is still triggering IntersectObjects.

Also hovering appears to be well done in such a way that it works on those (simple geometry) elements to respond to hover across them without resulting in any slowing down over large geometry. I would chalk this up to good handling of hover within R3F, as I have no hover events attached on the heavy geometries.

So it does look like I will be able to rather easily override the default mouse handling behavior though it is not well-synchronized with The React Way. Which is okay. Curious if there is a superior approach that already exists.

[drcmda]

it puts events on the wrapper elements, if you override them it does not get events, you can't have any pointerup/down/click/move on the canvas. threejs does not have events, r3f must receive dom events to form event-like raytracing.

it already only raytraces elements that have handlers, it doesn't indiscriminately run through everything. unless you put handlers onto the root group in which case it will run through all of their nested children. if you have handlers on the actual elements you want to pierce it will only test against these, nothing else.

here's something else that might help, you can exchange the raycast method on meshes. the original one goes through a boundscheck first, then checks all vertices - which is expensive. if it doesn't have to be super precise you can use simpler means: https://github.com/pmndrs/drei#meshbounds

btw. we are building cad systems, too, also working with very large models - which sometimes can be tricky. what's your usecase? is this for a product? maybe there's some interest in exchange that can help either side.

[unphased]

Thank you for the clarification! I'm sure now that what's going on is that I have my geometry improperly laid out: We have a large assembly loaded via https://github.com/gkjohnson/urdf-loaders and this is inserted into R3F via a primitive. That's where my handler is attached. I have to make it possible to click on some of the items inside, while other items which are static are the things that could contain a large amount of geometry. Since the data I'm working with doesn't even separate these concerns at the URDF level, it would indeed be absurd of me to expect it to work the way that I kind of need it to work!

Indeed overriding the logic of the raycast will help a lot, but it's also clear that until I can get a more reasonable level of scene graph granularity in place there will not be a clean R3F way to override it. Nevertheless I wonder if it is possible to override the default behavior that does vertex checking with some sort of blanket logic that does go in to do checks on bounding structures (where I also want to somehow force these to be OBBs rather than bounding spheres) on all leaf level Mesh objects within the primitives, but which stops short of actually checking vertices.

Clearly I can fork R3F and proceed by surgically removing that code but we can probably allow for it via configuration. I'll only need to do this until we move over to a superior representation format for the geometries.

I am currently implementing a unified web interface for Realtime Robotics: https://rtr.ai

[drcmda]

that's the problem. you put the entire scene into the graph and put a single handler on it, hence it has to raycast everything. have you seen https://github.com/pmndrs/gltfjsx yet? this is what you should strive for: build an immutable graph where you simply link the geometry and the materials. now you can put event handlers on singled out fractions of it. the gltfjsx tool works with gltf of course, but in principle you can do this by hand, just nest your meshes and link the correct geom/materials.

i don't think this is a forking problem. that r3f can raycast nested groups is a major benefit (threejs can't). you would be trying to work around that but it wouldn't make any sense because you put handlers on a group, what would expect other than raycasting the groups contents.

what also could work for your specific usecase is using layers. you can access the raycaster via

<Canvas
  onCreated={({ camera, raycaster }) => {
    camera.layers.enableAll()
    raycaster.layers.set(1)
  }}
>

and that primes it to objects on layer x only.

here's an example: https://codesandbox.io/s/r3f-basic-demo-forked-niq45?file=/src/App.js

[unphased]

Thanks! TIL about layers, must be a fairly new three.js feature. That will be handy!

I was more thinking that if one were to never want to do raycasting down to the vertices on meshes (e.g. always terminate at the bounding structure raycast), it would be a nice to have that as a global toggle, or say, be able to control that on a primitive. And then I would actually be able to use that. Yes, my primitive is a deep tree of groups and such, and contains large meshes.

I dont think I was saying that r3f implementing event bubbling on the scene tree is a problem (it's both not a problem and definitely useful), it's just that perhaps there are common use cases where raycasting down to the verts kills performance, while it remains clear that it should be the default behavior.

We may even need to back into three.js code to apply such logic, sorry I'm not too familiar with the three.js raycaster implementation.

Yeah obviously I want to use a better representation, anything remotely react-friendly, but it's simply not feasible at the moment.

[drcmda]

always terminate at the bounding structure raycast

you can do that as well: https://github.com/pmndrs/drei#meshbounds

demo: https://codesandbox.io/s/r3f-basic-demo-8fpip

the full source for meshbounds is:

import { Raycaster, Matrix4, Ray, Sphere, Vector3, Intersection } from 'three'

let _inverseMatrix = new Matrix4()
let _ray = new Ray()
let _sphere = new Sphere()
let _vA = new Vector3()

export function meshBounds(raycaster: Raycaster, intersects: Intersection[]) {
  let geometry = this.geometry
  let material = this.material
  let matrixWorld = this.matrixWorld
  if (material === undefined) return
  // Checking boundingSphere distance to ray
  if (geometry.boundingSphere === null) geometry.computeBoundingSphere()
  _sphere.copy(geometry.boundingSphere)
  _sphere.applyMatrix4(matrixWorld)
  if (raycaster.ray.intersectsSphere(_sphere) === false) return
  _inverseMatrix.getInverse(matrixWorld)
  _ray.copy(raycaster.ray).applyMatrix4(_inverseMatrix)
  // Check boundingBox before continuing
  if (geometry.boundingBox !== null && _ray.intersectBox(geometry.boundingBox, _vA) === null) return
  intersects.push({
    distance: _vA.distanceTo(raycaster.ray.origin),
    point: _vA.clone(),
    object: this,
  })
}

but you could replace this with something more precise but still fast, for instance a convex hull check.

[unphased]

Thank you! it seems like it's still just as slow as before after I put raycast={meshBounds} on my primitives so I'll tinker around with a custom raycast override.

[unphased]

• I defined the following raycast override and the slow behavior persists. I was hoping that this would just do nothing but it behaves the same as before so it seems like i've not overridden a thing?

          raycast={function (
            this: any,
            raycaster: THREE.Raycaster,
            _intersects: THREE.Intersection[]
          ) {
            console.log("raycastOverride", this, raycaster);
          }}

removing the onClick handler eliminates teh slowness. I can also confirm via Performance tab that it's getting down and dirty testing vertices.

The console log shows up and actually prints out evenly spread throughout the 1200ms it takes to run the 3 handlers so it seems like i'm tantalizingly close to being able to prevent the default raycast handling..

[unphased]

I'm pretty sure at this point that setting raycast on the primitive is probably bugged and causes the regular handling to happen rather than replacing it. It may very well work right when raycast is set on a mesh or group?

[drcmda]

raycast is individual. it is a prototype method on every mesh. raycaster calls that function to pierce the object. if you have 500 meshes, and you want loose bounds check, all of them must carry that function. you cant place it on a group or a primitive or assume that some top level element will apply to nested children - this isn't how threejs works.

but again, i think there's something else fishy. you say you only have a couple of objects needing raycast and that doesn't add up. 400ms must be caused by immense numbers of objects, heavy ones, too. i am still not sure if you are placing events only on the objects that you want to pierce - and you can't do this by dumping a full primitive into the scene with lots of nested objects - raycast will churn through all of them.

tldr, do not use primitive for your models. build a scene graph similar to gltfjsx. place events on the few elements that need to receive user input. raycaster will ignore everything else and it will be fast.

[unphased]

Great, thanks. I think I definitely have all the info I need now. Yeah, so URDF loader assembles the scene tree (hence why that is inserted as a primitive), and I'm a bit stuck with that for now. Could try to inject suitable raycast behavior at that level.

[drcmda]

it doesnt matter what the loader returns. gltfloader for instance also just returns a scene. i do this in order to create a map of all nodes and materials:

data.nodes = {}
data.materials = {}
data.scene.traverse((obj: any) => {
  if (obj.name) data.nodes[obj.name] = obj
  if (obj.material && !data.materials[obj.material.name])
  data.materials[obj.material.name] = obj.material
})

you could stick this into a hook. or if you tell what urdfloader returns exactly, maybe we can build this into useloader directly. currently it expects an object with a scene property in order to start building these maps.

but once you have it, you can build immutable scene graphs:

const scene = useLoader(URDFLoader, url)
const { nodes, materials } = useSceneMap(scene) // <- build it with the code from above
return (
  <group>
    <mesh geometry={nodes.robotHead.geometry} material={materials.metal} />
    <mesh geometry={nodes.robotArm.geometry} material={materials.steel} />
    <mesh geometry={nodes.robotButton.geometry} material={materials.plastic} onPointerOver={...} onPointerOut={...} />
    ...

[unphased]

Oh wow. Here I was trying to come up with reasons to move away from URDFLoader, but you're definitely onto something with that. Many thanks!!

[drcmda]

i added a new hook, useGraph: https://github.com/pmndrs/react-three-fiber/blob/master/api.md#usegraph

[unphased]

I haven't been able to get into the refactor for this soon as we'll be switching internally to a modification of URDF and probably will build our own loader, so it'd be a bit silly to implement proper integration with URDFLoader only to rip it back out shortly after.

However, prior to any of those undertakings, I've noticed by now that my shoddy way of incorporating the URDFLoader "scene" into the R3F scene via the use of a primitive is failing on cleanup. Even once I stop rendering the primitive, the underlying objects continue to be raycasted for by R3F from mouse events. I found https://github.com/pmndrs/react-three-fiber/issues/815, but I'm not sure I understand the linked fix. Should I try to upgrade to 5.3.7? Should I add dispatch={undefined}? Should I do both?

Thanks.

[unphased]

Oh I think I understand. The fix you made (https://github.com/pmndrs/react-three-fiber/commit/603a1d10851369be93baa96473875d3b224493fa) simply changes it so that without dispatch={undefined} the events are still cleared. this may leave a memory leak but at least the event overhead on those elements won't be a thing anymore. So I should be fine without upgrading and using dispatch={undefined}.

[unphased]

BTW i think it's dispose, not dispatch.