React-three-fiber: React Native Support

[mattblackdev]

First let me say, this project is incredible! It has opened my eyes to Fiber and react-reconciler. Fantastic work!

As an experiment, I got the three reconciler working inside an expo app: https://github.com/mattblackdev/native-three-fiber

It lacks many of the goodies in that awesome Canvas component you built. But, I was surprised how independent the reconciler is and that it's usable on native.

Is there any interest in contributions and/or supporting a React Native/Expo version of the Canvas component and hooks?

[drcmda]

of course! would you be interested in working together on this? we could start a pr, i would help finding ways to make canvas x-platform. if you like you could co-author this lib as im def looking for people wanting to help out.

as for the demo, it's super nice! i knew that using plain render(...) can get us into expo but haven't actually tried, im glad it works ootb! but yeah, the canvas is the problem because it brings all the useful stuff (interaction, hooks). we could break up canvas so that critical parts become exchangeable. the loop and the hooks are cross platform: https://codesandbox.io/s/yq90n32zmx just the interaction layer has to be made more versatile.

[drcmda]

btw, could you make a quick video of some 3d stuff running natively on mobile? i would tweet this, maybe we can get some additional help.

[mattblackdev]

Awesome! I'd definitely be willing to help.

Here's a video: https://youtu.be/_7cMxoL-DCQ

FYI, my motivation is to implement an entity-component-system on top of this for game development.

[drcmda]

i don't have that much experience with RN, what kind of events do you have there, are they similar to pointer events? what is the data that you get?

here's the part where the event data is packaged into a custom event handler (onPointerXYZ): https://github.com/drcmda/react-three-fiber/blob/master/src/canvas.tsx#L308-L317

and here's the part where it reads out clientX/Y to calculate the ray: https://github.com/drcmda/react-three-fiber/blob/master/src/canvas.tsx#L261-L262

it shouldn't be hard to do this in RN, right? we just need to figure out how we'd break the parts that are dom related out of canvas.

could you look into the event thing in RN? i'll worry meanwhile over canvas and breaking it into pieces.

[drcmda]

another question is, in the dom we have the canvas dom element which serves as the foundation for three. how does expo do it?

[mattblackdev]

I think a good way to go would be expo-gl which I’m using in the demo and it exposes touch up, down, etc... I haven’t checked yet but I would bet it provides x, y position.
On Jul 7, 2019, 12:40 PM -0400, Paul Henschel [e-mail], wrote:

another question is, in the dom we have the canvas dom element which serves as the foundation for three. how does expo do it?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[drcmda]

it could be a generic view wrapped around or anything really. if we have these coordinates we could start.

here's a list of all the things that are currently dom related:

• [ ] events: tap, swipe, longpress?
• [ ] the expo canvas element
• [ ] a way to react to and read size in general (i guess on mobile thats when it goes into landscape/portrait)
• [ ] getBoundingClientRect (left, right, top, bottom of the containing element)

[mattblackdev]

I did a little digging on this and learned we have:

• access to native gesture handlers.
• GLView's onTouchDown, onTouchMove, and onTouchUp all pass the x,y via event.nativeEvent.location[X|Y]
• onLayout handler on the GLView could be our hook into resize events and passes the new x, y, width, height.

I see how you update the camera on resize but I'm not sure if the gl object will need updating as well. I see this comment in the GLView.

[mattblackdev]

Okay I got basic resize handling on the GLView figured out for the most part:

onLayout={e => {
  if (!ready) return
  const { width, height } = e.nativeEvent.layout
  const { gl, camera, renderer } = state.current
  const scale = PixelRatio.get()

  gl.viewport(0, 0, width * scale, height * scale)
  renderer.setSize(width, height)
  camera.aspect = width / height
  camera.updateProjectionMatrix()
}}

Only weird part is, it doesn't look like the gl.drawingBuffer[Width|Height] ever gets changed under the hood. This could be related to this comment in expo-gl. Regardless, I haven't noticed any visual distortion yet.

My demo is updated with this code if you want to check it out.

Regarding raycasting, it seems like all the useCallbacks related to raycasting could be reused if we inverted the dependency on getting the x, y from the event. For example, instead of this signature:

useCallback((event) => {
  // do things with event.clientX
}

We do:

useCallback((event, x, y) => {
  // do things with x, y
}

[drcmda]

Nice, that’s all we need. How do we split up canvas.JS to teach it about native events? I know that in package.Json we can target RN and pass an alternative index.Js, which could inject something into the core. That’s how reactspring works, too.

[mattblackdev]

That would definitely work. What are your thoughts about extracting most of the Canvas logic into 1 or more hooks? Or even a hook generator that provides hooks that share state? ie:

import resizeObserver from '...'

const {
  useRenderer,
  useEventHandler,
  useRender,
  useSceneState,
} = makeRenderer({ injectStuffHere })

function Canvas({ children, ...props }) {
  const canvas = React.useRef()
  const state = useSceneState({ canvas, ...props })
  const { handleResize, handleInitialize } = useRenderer({ state, ...props })

  React.useLayoutEffect(() => {
    handleInitialize(stuff)
  })

  React.useEffect(() => {
    resizeObserver(handleResize)
    //...
  })

  const handlePointerUp = useEventHandler('pointerup', props.onPointerUp)
  const handlePointerDown = useEventHandler('pointerdown', props.onPointerDown)

  return (
    <div onPointerUp={handlePointerUp} onPointerDown={handlePointerDown}>
      <canvas ref={canvas} />
   </div>
  )
}

I might be making a leap here. Let me know if you think we should stick to something more orthodox to move forward.

[hashrocketeer]

@drcmda bump

[drcmda]

@mattblackdev sounds great, you wanna go forward with a pr? i was a little dug into work these days so i couldn't work on it, but let's get this rolling.

[birkir]

Here is my progress on RN https://gist.github.com/birkir/5f0724dff3efe5f8681d905fa505468a

I tried many variations on getting raytracing + pointer events to work without success, mostly because lack of https://github.com/jquery/PEP polyfill.

I was investigating to use hooks (like useTouchDown(ref)) to support touches, looks like a neat solution IMO (and universal).

I also added orbit controls, camera settings and so on.

[drcmda]

@birkir wow, you should be co-author by now, github access rights etc, this is the second bomb in a row. you want to be?

i don't think we need PEP. we could move the events part into a vendor specific folder like /targets/web/index.tsx & /targets/native/index.tsx. these index files would fetch the platform independent core part and wrap them into something that receives events. the user-facing api could remain having PE related names.

[birkir]

@drcmda yes! I can't promise anything, but I am very interested in three.js in shared codebases (web+native) these days, so if you can find the use for my work, that would hopefully be beneficial for many others.

I am investigating other repositories that deal with multiple platforms and their architecture (react-spring etc.)

[drcmda]

i've added you, you have write access. also moved the repo to a neutral org so that it isn't attached to my name. we can still work via pr's for bigger changers but you could def add your name to the authors in package.json

[birkir]

Here we go with the refactor, I'll keep discussion in the PR.

https://github.com/react-spring/react-three-fiber/pull/182

[drcmda]

we can continue here: https://github.com/react-spring/react-three-fiber/pull/189

[NielsSkovgaard]

@drcmda yes! I can't promise anything, but I am very interested in three.js in shared codebases (web+native) these days, so if you can find the use for my work, that would hopefully be beneficial for many others.

I am investigating other repositories that deal with multiple platforms and their architecture (react-spring etc.)

@birkir I'm also very interested in this and think it would be really beneficial 😄

Did you manage to get OrbitControls to work in mobile apps with Expo / React Native?

I made a demo app for trying to make it work using different techniques. On one of the screens, I tried with React-Three-Fiber, and as the test results show, it works on web (both touch and mouse input), but doesn't work when running as a mobile app.

See: https://github.com/NielsSkovgaard/expo-three-demos

[robksawyer]

Any new progress on this? Currently, working on an Expo app and I'd like to use react-three-fiber.