Model-viewer: Allow manipulation of model scene graph via "Render Worklet"

[cdata]

Requests to directly manipulate the model scene graph have come up several times, so I'm filing an issue for us to have a canonical place to discuss the feature.

The idea behind this feature is that we would allow users to specify a script that will run as a kind of "render worklet" that has imperative access to the model's underlying scene graph. The goal of the worklet is two-fold:

1. Ensure that user scripts can be transparently moved between the main thread and a worker thread, depending on what set of optimizations <model-viewer> applies in the given environment
2. Ensure that user scripts do not depend on underlying Three.js APIs, in case we decide to offer alternative rendering backends in the future.

The fundamental details of this API are still very much up in the air, but the idea is that a Render Worklet script would have imperative access to a scene graph that is modeled after the glTF JSON schema. Users would be able to change things like PBR properties of materials or vertex information and see the changes reflected in the model in real time.

[cdata]

Some related discussions on this topic include: https://github.com/GoogleWebComponents/model-viewer/issues/186#issuecomment-439303818 and https://github.com/GoogleWebComponents/model-viewer/issues/685#issuecomment-516614906

[fselcukcan]

Plus one on that!

[ataylor09]

Would like to voice my support for this feature.

[hybridherbst]

Same here. Would that also allow adding objects? In our case we're looking at options to have something like "model annotations" (essentially small billboard buttons in 3D that might drive a camera and have a callback to react to).
(I think this proposal would be a "good way" to do this without the hacks described in https://github.com/GoogleWebComponents/model-viewer/issues/654#issuecomment-506754056?)

[cdata]

@soraryu yes, this would eventually include the ability to add objects, although it may take multiple iterations before the worklet has every possible capability.

Although we anticipate that you could implement annotations with the worklet alone, we also wish to offer a high-level API for annotations / hotspots / billboard buttons (see #334 ).

[aiursrage2k]

I would like to request having multiple animations. Imagine something like an ikea furniture, so when you click a button it will move onto the the next step in the animation

[cdata]

@aiursrage2k this is actually possible with our current animation system, I think, although there is probably stuff we could do to make it easier. We already support multiple animations, and we reveal the current time of the playing animation. If you pair that with play and pause methods, you can construct a UX like what you are describing.

The thing that we could do to make it easier would be to let you control the iteration count, like with an animation-iteration-count attribute. Currently, animations play on a loop forever, so you would have to watch currentTime and manually pause() after the first loop.

[aiursrage2k]

Okay I am a unity developer, and I can easily create an app that can import models from google poly, sketchfab etc and allow you to build scenes, export to GLB and then export to usdz. But if I want to make the ar experiences interactive I need to be able to get access to that camera/scene so we can modify how the sceneviewer/quicklook behaves (IE imagine something like an ikea furniture -- where I can step through the animations).

Its kind of baffling that reality format is closed source, it doesnt even seem like its using the z-buffer, it actually duplicates the same models/textures (if you use the same one in a difference scene), and the interactions that are there are very basic (without any api -- so you have to create these experiences manually).

[cdata]

@aiursrage2k I agree with the general sentiment that .reality is not ideal.

My personal background is as a web developer interested in graphics and game dev. I have some experience with Unity as well, and a lot of obsolete expertise with Flash.

In general, I would like to see open formats be developed that are informed by both the web and by game engine technology. In the case of .reality, I think a lot of what they have done would benefit from being an open standard. My hope is that our project can be part of a bigger discussion around how open, interoperable formats might be developed for the web that can feed into the same AR use cases being addressed by Apple with .reality files.

[cdata]

As an artifact of work in #883 , here is a forkable Glitch with a one-off build of model-viewer.js that can be used to play around with a very basic render worklet: https://tidal-mascara.glitch.me/

This proof of concept demonstrates the feasibility of the render worklet in principle. The worklet implementation is currently limited to the following qualities:

• A global model-change event is dispatched when the model scene graph is available
• A global model object is available when model-change is dispatched, and it represents the model scene (much like document represents a DOM document on the browser main thread)
• The scene graph can be traversed manually. Eventually, we will add an API like model.querySelector(material#name), but for now you have to crawl the scene yourself.
• The only thing that you can change at this time is what is shown in the demo: baseColorFactor on a material.

Once again, this is just a proof of concept. We have a much more robust API in mind, and a cleaner implementation as well.

Please let us know if you play around with it or if it gives you new ideas. We would love to hear your feedback 🙇

[cwervo]

@cdata This is so cool! The tidal-mascara example is a great starting point to prove out what's possible.

The eventual model.querySelector('material#name') would be lovely, but will try to play around with this manually sometime next week when I'm back at work!

[hjeldin]

@cdata I've updated your work on render-worklet-proof to reflect the recent changes (monorepo, etc) that happened on master, you can see them here.
I also expanded the API surface to include emissive color, metalness and roughness, with more simple parameters to come.

You mentioned a cleaner implementation incoming, do you have an ETA?

[cdata]

@hjeldin wow that's awesome, thank you for doing that 🍻 do you have any feedback about the experience of expanding the API?

I am currently planning to make a revision to this work that will be landed in master for our current sprint (starting today).

The first pass will be relatively simplified compared to the proof of concept. The first pass will exclude APIs for hierarchy traversal. It will only offer the ability to access and manipulate material properties.

Once we have landed the first pass, we would welcome contributions to expand the scope of the API to include values like the ones you have added in your fork 👍

[hjeldin]

@cdata glad to hear you're about to work on this!

My main architectural issue was trying to find a sane way to load textures that didn't involve serialization/deserialization to/from worker threads (which I currently "solved" by calling TextureLoader.load() inside the worker itself); other than that the api was quite self-explanatory.

I would be more than happy to help expanding the materials API as soon as you push the first pass on master.

[cdata]

@hjeldin yah, that's a good point. One of the architectural constraints for this work is that we must explicitly avoid sending bulk data back and forth between worker and main threads. So, no transferring texture data, vertex buffers etc.

The reason for this constraint is that we want to end up with a content format that is safe and fast enough to be loaded from third-party origins the way, say, <amp-script> is able to be loaded from third-party origins by otherwise trusted contexts.

This constraint leads us towards some unorthodox API choices compared to traditional scene graph APIs. Hopefully we can come up with some creative approaches that allow us to preserve the ability to share these content scripts across origins 👍

[cdata]

Hey folks in this thread. Our first-pass implementation of a scene graph / render "worklet" has landed in master. You can see the details in #992, as well as a page in our docs that will be going online shortly. This feature should be considered both experimental and rapidly evolving. We have a lot of features to add to it based on the feedback y'all have given us, and the API is not considered stable yet.

Please try it out if you have a chance, and let us know what you think!

[cdata]

Live example can be found at: https://modelviewer.dev/examples/scene-graph.html

[deers]

@cdata are we able to change the texture image through the API?

[cdata]

@deers not yet, but that is an intended use case of the API. I just filed https://github.com/google/model-viewer/issues/1096 to track the feature 👍