Model-viewer: Incorporate fidelity testing into third-party renderer projects

[dakom]

After watching @cdata 's talk at GDC (slides here), it seems that the glTF-Sample-Viewer is planned to be integrated into the fidelity tester, and perhaps make it possible to use third-party renderers...

Is there a tracking issue for this?

Fwiw I'm at the very beginning stages of building a rust/wasm/webgl renderer. Therefore I'm particularly interested in being able to hook up the fidelity tests to _any_ renderer since even if I blatently rip off follow the sample viewer today, there's no automated process to make sure my port stays in sync as the reference makes improvements.

Would love to be able to follow a simple interface like (PathToGltf, ViewportWidth, ViewportHeight) -> Promise<ImageBitmap>

[cdata]

Hey @dakom that is awesome. I'm really excited to see your interest in this area.

We don't have a good tracking issue for this yet. In fact, since GDC we have regressed a little and removed Quick Look from our tests. More recently, we decided to focus primarily on renderers that run on the web, and your renderer sounds like it fits the bill. This issue is a great start for expanding our library.

Mostly I would say the pieces are in place if you would like to incorporate your own renderer into our suite. However, it isn't a formally documented system yet, and you would be the first person adding a new renderer who isn't @elalish or me. I would recommend that you follow the model for how we currently integrate Filament. It isn't without bugs (mostly they are bugs that we have traced back to Filament's JS API), but it works well enough for fidelity testing purposes.

The basic architecture we use is to wrap a given renderer in an HTML custom element, and then to craft a basic HTML page containing that elenent. Our test runner then loads the page and uses the content to generate a screenshot.

Here is the custom element we use for Filament: https://github.com/GoogleWebComponents/model-viewer/blob/master/src/test/fidelity/components/renderers/filament-viewer.ts
Here is the HTML page that is used to take the screenshots for Filament (note that it incorporates the custom element above): https://github.com/GoogleWebComponents/model-viewer/blob/master/test/fidelity/renderers/filament/index.html
And finally, here is the source config you would need to amend in order to add your renderer to the test run: https://github.com/GoogleWebComponents/model-viewer/blob/6f4875528cee6db308bfc20037802e58821c1419/test/fidelity/config.json#L7-L19

In addition to what I wrote above, we currently have a very basic facility for running a script to pre-process the IBL. This is needed for Filament, but not for <model-viewer>, and you can see that reflected in the config above. When the script is invoked, there is a $LIGHTING environment variable set that points to an HDR equirectagular image to be used for lighting, and it is expected that by the time the script exits, all preparation work has been done to take the screenshot.

Anyway, that might be more info than you wanted, and we haven't touched on the other concerns you raised, so....

In the long run, what we hope for this process to look like is: you point to any URL and we assume it acts like one of the renderer HTML wrapper documents above. This means that the main configuration interface (the "simple" one, as you called it) would be a URL. As long as the page loaded at that URL sends the appropriate signal (we need to know when the model is rendered and visible), we can use it to take a screenshot.

As for incorporating your renderer into our tests, that is a tricky question that we don't have an answer for yet. There is on-going interest from Khronos to have us bootstrap a first-party fidelity testing effort using this testing infrastructure. In a world where we do that, I imagine we would want a facility for folks to easily get their own renderers running on the test infrastructure without requiring them to be closely coupled to it. This means you would probably be able to install the test runner as a distributed package. Unfortunately, that's just not the case today.

What I will say is that if you get your renderer far enough along, we would be interested in incorporating it here. I'll also say that you can certainly maintain an up-to-date integration if you are willing to keep your git clone up to date and participate in discussion with us here. That's all a lot of work, though, and I'm sure it's not what you hoped for when filing the issue.

Anyway, I'm still very excited to see your interest here. I would like to assure you that there is a lot of energy behind this work on our side, and we plan to make ongoing improvements to the tools and processes. We feel very strongly that this is a critical (and critically underserved) process for 3D renderers. So, I hope you stay interested and maybe help us make all of this stuff better 👍

[dakom]

Wow, awesome feedback!!!

It'll probably be a while until I really have something to show, but it's great to know that these notes are here waiting and that there's active development going on.

Yay!

Feel free to "close" the issue, since you answered it in spades - but I'll probably follow up here when I get to start trying it hands-on.

Thanks again!

[cdata]

No problem, it was good to get it all out there :)

We'll leave the issue open. This user story is part of the long-term arc for our fidelity testing, so I know we'll be referring back to it for some time to come.

Good luck with the renderer!

[aviggiano]

@dakom maybe this is off topic, but where can I find the recording of this talk?

[dakom]

hehe sorry, can't remember anymore!

[elalish]

@aviggiano It's here, starting at 25:30 https://www.youtube.com/watch?list=PLYO7XTAX41FNx5LraG2CE48DBUla3Su_m&time_continue=46&v=mCJoBGOCTwk