Microsoft-ui-xaml: Proposal: 3D ModelViewer Control

[GeorgeS2019]

3D ModelViewer in WinUI

Summary

A XAML control that displays interactive 3D models. This is analogous to existing controls for other types of media (images, motion graphics, videos, etc.).

Rationale

• Allow richer, interactive digital representation of real-world objects, extending the set of visual metaphors (text, vectors, images, videos, etc.) available to app creators today .
• Make 3D easy for Xaml developers. One of the biggest blockers to the use of 3D content in app development is the esoteric nature of existing solutions (game development tools) and the significant technical barrier to entry that comes with them. A control that abstracts away the technicalities and works as easily as \ or \ would be accessible to a wider audience.
• Integrate 3D scenes with the XAML framework, so that customers can leverage the benefits of a mature UI framework (layout, binding, accessibility, styling, input handling, etc.) while using 3D content .

Scope

• Load 3D assets (glTF, etc.).
• Manipulate model via camera controls (TRS).
• [Future] Enumerate embedded animations, if any, and control playback.
• [Future] Modify environment properties (lights, shadows, etc.)
• [Speculative] Smart/automatic contextual changes in MR environments
• Provide options for Fallback and Caching

Important Notes

• Use the Windows.UI.Composition.Scenes APIs for rendering and animations.
• Leverage SceneLoader and other related OSS utilities to modularize the control and the content.

[GeorgeS2019]

In Build 2019 State of the Union - The Windows Presentation Platform - BRK3072 => it is stated that 3D Viewer Control is coming to WinUI 2.x

It would be great if more of the information regarding the 3D viewer control can be shared through this thread as all previous attempts to get 3D support has not arrived as a concrete moving forwards.

[mdtauk]

Has there been any progress on declarative XAML 3D yet, it was talked about the past few Builds

[jesbis]

Yes we can use this issue to track it. @sohchatt can provide the latest info

[GeorgeS2019]

@sohchatt From available information, it is likely that Microsoft will choose JavaScript instead of C# to provide the 3D engine behind the opensource Windows UI control.

Surveying Youtube would reveal that many existing Unity 3D c# developers are moving to Opensource Godot.

I suggest Godot because it has ONE OF THE MOST active community support for providing high fidelity Blender export (advanced vertex/skeletal animation) to Godot 3D. All existing Javascript effort e.g. Microsoft HoloJS, will not offer the open source community this advanced export from Blender to Godot.

[sohchatt]

@GeorgeS2019 @mdtauk Thanks for filing the issue and providing your inputs.

We've added Windows.UI.Composition.Scenes APIs in the May 2019 SDK that allow you to create a 3D scene graph programmatically. We're also creating new open-source libraries and utilities such as SceneLoader to help with constructing this scene graph from 3D file formats such as glTF.

To make these 3D capabilities accessible via markup, let's use this issue to track the development of a XAML Control for loading 3D assets.

From available information, it is likely that Microsoft will choose JavaScript instead of C# to provide the 3D engine behind the opensource Windows UI control

@GeorgeS2019 our current plan it to use the native 3D renderer that ships with the Windows OS (version 18362) for this Xaml Control. The renderer is accessed via the Windows.UI.Composition APIs mentioned above, and allows for visual consistency with other 3D experiences in the Microsoft ecosystem, such as 3D Viewer, BabylonJS, 3D in Office, etc. I'll certainly take a look at the Godot engine and learn more about its capabilities.

If this UWP desktop app is now used in Hololens or Mixed Reality, a more 3D centric user response will be trigger to give a richer experience.

This is certainly a powerful scenario that is especially enabled by having a spatially-aware compositor/renderer, and one that we are eager to enable. We are currently evaluating the feasibility and scope for MR use-cases, but it is not part of our initial staging plans for the 3D WinUI Control. However, I'll be sure to call it out as a future goal as we make more progress on building the control.

[GeorgeS2019]

@sohchatt It would be great if SceneLoaded's gITF support is able to import AND RETAIN (e.g skeletal/vertex animation) FEATURES in gITF exported from Godot

[sohchatt]

@GeorgeS2019 As far as the workflow is concerned, I would expect that, as long as Godot's export conforms to the glTF 2.0 spec, everything should work as you'd expect it to. This approach has the added benefit of making your 3D content portable across the tools and platforms that you're invested in, without having to be dependent on any one renderer. In addition, there is an ongoing effort from the glTF group to standardize renderers as well.

I ought to point out that the Windows.UI.Composition.Scenes APIs in the 19H1 SDK are missing some glTF features (such as skeletal animations, etc.), but it is our goal to make sure that these gaps are addressed in the next update. Would you be willing to file an issue in the SceneLoader repo requesting support for missing features so we may track your request?

[sohchatt]

@GeorgeS2019 I've edited your original post to be more generally applicable to 3D content in all scenarios. Do let me know if you have any questions or concerns. Note that the MR feature you requested is listed above as lower priority because the Windows.UI.Composition.Scenes APIs are not composited holographically at present, but we're evaluating this capability, as mentioned above.

[GeorgeS2019]

@sohchatt This is what I hope to do with the 3D modelviewer control

https://godotengine.org/article/we-should-all-use-gltf-20-export-3d-assets-game-engines

[GeorgeS2019]

Godot receives generious grant from Microsoft! Excellent! C# will be further supported in the next Godot 3.2 version. gITF for skeletal animation import will be further improved in 3.2. Godot export to UWP uses Angle. However, to have a 3D Model Viewer UI control, we still need the work on going with #686

[GeorgeS2019]

@jevansaks It would be a great idea to showcase openai/gym reinforced training in 3D in Hololens 2.0 using SciSharp/Gym.NET and godot(UWP) compiled with Angle (Hololens)

[michael-hawker]

Wanted to link to @JustinXinLiu's preview sample from the Insider Dev Tour: https://github.com/microsoft/InsiderDevTourDemos19/blob/master/Sessions/ui

[GeorgeS2019]

@michael-hawker
Please keep sharing anything related to 3D ModelViewer control for WinUI here.
FYI Skeletal and vertex animation library with potential for porting to WinUI 3.0

As Hololens 2.0 is now available to independent developers, 3D WinUI pioneered in 3.0 is very important.

[myokeeh]

I'm really interested in this.

[GeorgeS2019]

@myokeeh keep pushing awareness of this. I think Microsoft will hopefully allocate more effort into 3D for WinUI 3.0

FYI We need View3D UWP app now appears as WinUI 3.0 control :-)

[Felix-Dev]

You can also find the link to that demo here: https://github.com/microsoft/microsoft-ui-xaml/issues/1688

This post shows me more should be done to bring awareness to that post, which is precisely about developer resources.

[myokeeh]

@GeorgeS2019 does this proposal include the scope to translate/rotate separate objects relative to each other with code? Think about an assembly with moving parts and relationships.

[GeorgeS2019]

@myokeeh, it is important to encourage the team (e.g. @sohchatt) to continue the effort by actively providing feedback to the SceneLoader development

Yes, we are talking of bones :-)

[Noemata]

The WPF 3D Xaml capability was used in hundreds of internal and external facing engineering projects and is still in use. Most of these are corporate style initiatives that have little visibility yet are very strategic. A UWP GLTF based alternative, with Xaml like binding capability cannot come soon enough. Having bindings for a GLTF bone structure and animations would be extremely useful/powerful.

[StephenLPeters]

@GeorgeS2019 did you mean to close this proposal?

[GeorgeS2019]

@StephenLPeters I will reopen it. Closed accidentally.