Monogame: New 3D Model architecture with in built Animation support and PBR

[vpenades]

A while ago, I published an example of how to load glTF models at runtime... since then, this project has evolved into much more, so I would like to introduce the new repository:

https://github.com/vpenades/MonoGame.AnimAndPBR

In it's current state, it's quite usable, and it already supports most of glTF and PBR features.

Since a lot has changed in a while, this issue supersedes #6932

The general plan is to have a new set of effects and model classes that can be used, _out of the box_, as replacement of the very outdated XNA's Model and BasicEffect.

As an example of how the new PBR effects look like, this is a glTF model loaded with the library:

So what's this issue about?

Although a lot has been done, there's still a lot to do. In particular, a missing key point is that right now, the glTF loader only runs at runtime... _there's no content pipeline processing_.

So, first and foremost, I would like big brains around here to review the library and give feedback on how to improve it and ways to solve the current big issues it has to move it forward.

What version of MonoGame does the bug occur on:

• MonoGame 3.8

What operating system are you using:

• All

What MonoGame platform are you using:

• All

[Jjagg]

I'm really excited for this! MGs Model class is outdated by 15 years and this is a long overdue modernisation of it! (Though we'll keep the old model class of course)

Could you be more specific about what kind of review/help you need? Maybe open issues in your repo and link them in this issue.

For the content pipeline:
Off the top of my head, I don't think we want to use any of the existing content classes that exist for Model in the content pipeline.
Instead we can use data only classes that map directly to the runtime classes for the glTF model. So anything that has native resources will need a data-only class for the content pipeline.

I really would love to see this integrated! Lets make it happen! Wish I had more time to put into this!

[vpenades]

First of all, I would like people that might want to be involved, to get familiar with the current basecode, so we can be on the same page. no need to hurry since this is going to be a long trip.

The repository is divided into 3 main parts:

• The PBR shaders
• The new Model classes
• The new importer architecture

Each of these sections has a number of problems of their own.

My experience with monogame shaders is outdated and limited, I essentially transcoded khronos official glsl shaders, but I'm already over my limit on that part... Willmotil is trying to help with IBL lighting, but struggling with monogame's shader limitations already. Also, PBR shaders have a lot of parameters, which has implications in the exploding number of techniques, and the limitations of opengl ES on mobile, which needs to be tested.

So I would need devs with monogame shader experience to take the helm on this one.

The model classes is something I can handle, and more or less they're almost finished.... but since these classes would be the public API everyone would use, I would need devs to review the API and tell if they would be comfortable using it in the way it's been designed, or if the classes naming feels fine, or of we need interfaces here and there, what's missing, what people would like to be included, etc.

On my side, I need to write a document explaining, and to some degree, justifying the architecture.

The third section, the importers, is all about figuring out what would we need to have an intermediate XNB model format without the current content pipeline.

I've already wrote a roadmap / wishlist of things that need to be discussed in this issue.

[SimonDarksideJ]

I'd request for backwards compatibility if:

• The model class could simply be extended, to give the best of both worlds and having an easier transition process.
• Rather than replace Basic Effect, it would just provide more Effect definitions that extend on BasicEffect, like we do with DualTexture and Skinned Effects.

Or this is not feasible. I'd just prefer extending over replacing or having more types alongside the existing to avoid confusion for devs as to "what to choose"

[vpenades]

@SimonDarksideJ

The model class could simply be extended, to give the best of both worlds and having an easier transition process.

Not possible, I tried, I really did, but the architecture needed by glTF, USDZ and to some degree, latest iterations of FBX demands a completely new Model framework.

It is actually possible to load a glTF model into an old MonoGame Model class, provided it is unanimated, and its material setup is relatively basic (no opaque/translucid material mixing, etc) ... but the whole point of developing a completely new Model framework is to support animated models out of the box and other features.

Rather than replace Basic Effect, it would just provide more Effect definitions that extend on BasicEffect, like we do with DualTexture and Skinned Effects.

This is already supported. the current glTF loader/importer has two loading paths: one that loads a glTF model with BasicEffect and SkinnedEffect, and another one that loads the model using the new PBR effects. So it is entirely possible to pour your own effects.

In the repository, there's still a small dependency between the PBR project and the Model project because the Model project requires the IEffectBones interface to declare the bone transforms, I plan to remove that dependency as soon as new monogame packages are published. At that point the PBR and Model projects will be completely independent from each other, and it will be up to the developer which effects to use.

As a side note, it could be possible to write a new SpriteBatch class using the PBR effects, to support sprites with normal maps & other stuff.

MonoGameViewer can load glTF models... and you can switch between PBR and default effects:

This is with PBR:

And this with BasicEffect:

Or this is not feasible. I'd just prefer extending over replacing or having more types alongside the existing to avoid confusion for devs as to "what to choose"

As Jjagg has stated, the old Model classes are not going to be removed... the idea is that both the old and new classes to coexist.

About that coexistence of classes, I agree that adding a new set of classes might be confusing, so I'm open to discuss classes and namespaces naming. (the project names, namespaces and classes names in the repo are totally provisional)... maybe it's about time to set these new classes in namespace MonoGame.Framework.Graphics ?

[SimonDarksideJ]

Thaks for the detailed response @vpenades , that's some great detail and we should endeavour to capture that passion in the documentation to support the new feature, which hopefully be incorporated into the MG base.
Sounds like we simply need PBRModel, PBREffect and PBRSpriteBatch, to make it easier (and applies if the repo already calls them this as I've not been able to invest time looking in to it yet :D )

[willmotil]

So to add a little more info on this.

All in all with Gltf much more is involved in comparison to the old outdated model and effects classes that monogame is currently using.
Gltf is highly tied to being able to load models made specifically with Physically based rendering effects in mind. Since many of these models are built with related material information to PBR the models the loaders and the shaders are somewhat useless without the capability's and presence of the others.
So theses parts are in a manner coupled more tightly naturally.

Essentially just about any Gltf model as it is simply just loads (It's no were near as quirky as assimp and fbx) roughly about 90% of the Khronos Group test models pass, and some of the khr tests pass as well.
However there are some things unfinished as far as the shaders go and some things that can't be implemented due to bugs in MonoGame such as morph targets in the Gltf way and environmental lighting (IBL) on Gl due to the bugged open gl texCubeLoad and tex2d lod calls.
There is a big question mark on if and how to approach dealing with built in shadowing.
There is quite a bit of garbage collection work to do im sure and there are some bugs as well on passing tests which aren't quite right yet but overall most of the tests are passing now.

Atm the models are loaded with the equivalent to loading them FromFile.

So one question is how important is it to start this off with content pipeline editor support ?
My thoughts are just to get it working in MonoGame then worry about refining it as time goes by.

Which leads to another question as there are a lot of classes involved were to put all this stuff in monogame ?

[vpenades]

I've began writing documentation and considerations of the Model architecture here

PBRModel, PBREffect and PBRSpriteBatch

@SimonDarksideJ not quite... the big one is called ModelTemplate (and it's counterpart ModelInstance) ... I don't like the -Template suffix too much, but I can't figure out a better word.... conceptually, you don't draw a ModelTemplate directly... you use it as a _Template_ to produce ModelInstance objects.

I would not prefix the classes with PBR because the model classes can use more effects other than PBR, and also, PBR has much more to do with materials and effects than with models.

Given there's a ModelInstance.... I also considered ModelFactory? ModelResource? ModelPrefab?

@willmotil it is true that glTF _assumes_ PBR materials for rendering... but we also need to have the flexibility to be able to customise the effects of the models, not only because some devs might want to use their own shaders... but also because PBR might be implemented in many ways.... comes to mind PBR in deferred mode, which would require totally different effects that the ones we're implementing.

Also, I don't mind transferring the repo to monogame, or keeping it until it's mature enough for integration.... in fact, when talking about integration, I would consider keeping these projects as separate packages and not included as part of the main MonoGame.Framework package.

[SimonDarksideJ]

"ModelFactory" gets my vote, as what you describe follows the factory pattern

[willmotil]

https://discord.gg/hUM2Nk

https://discord.com/invite/drEYKc

To anyone who is interested or would like to participate on discord concerning this.

[vpenades]

I have a general question for those that use Monogame on smartphones, consoles, and other devices with hardware limitations.

First of all, PBR is not a single shader, but a family of shaders that share a common architecture, the plan is to support the most commonly used ones, which are:

• Metallic Roughness model
  
  
  
  • Normal XYZ
  
  
  • BaseColor RGBA
  
  
  • MetallicRoughness GB
  
  
  • Emissive RGB
  
  
  • Opacity R
  
  
• Specular Glossiness model
  
  
  
  • Normal XYZ
  
  
  • DiffuseColor RGBA
  
  
  • SpecularGlossiness RGBA
  
  
  • Emissive RGB
  
  
  • Opacity R
  
  

Right now we're considering adding Image Based Lighting (IBL) which is the kind of lighting model that makes PBR really "shine". The problem is that IBL lighting requires two additional textures: a Texture2D lookup, and a TextureCubeMap, which would push the texture usage count up to 6 Texture2D and 1 cubemap

The question is... is this acceptable on devices that have a limited number of samplers? I have no clue which is the maximum supported on, say, Android, or Nintendo Switch.

If the 6+1 is acceptable on lower end devices, we would define IBL lighting as an integral part of the PBR effects.... otherwise it would be a derived effect, and only available on hardware with enough juice.

[cpt-max]

OpenGL 3 hardware requires a minimum of 16 textures per shader stage. I'm pretty sure this also applies to OpenGL ES 3. MonoGame still supports OpenGL 2, but that's really old now. I'm not sure if OoenGL 2 defines a minimum texture count as well, but if it does, it should be at least 8. As far as I have seen those numbers are always powers of 2. Since you are already above 4, everything up to 8 should be perfectly fine.

[vpenades]

I have yet another question regarding Monogame's effects architecture...

So far, I am following the pattern used by BasicEffect and SkinnedEffect, which is to compile multiple shader technique permutations, and switch techniques based on the current state of the effect.

The problem is that, with the richness of a PBR effect, the number of technique permutations is insanely high, and we're already reaching compromises and dropping features only because this.

Is there another way around this? what other developers are actually doing when developing their own shaders/effects?

[tomspilman]

@vpenades

The problem is that, with the richness of a PBR effect,
the number of technique permutations is insanely high,

Yeah that is a problem of sorts.

One thing to do is move some of the work to be runtime constant conditionals instead of a different permutation. GPUs are better at conditionals like this than the used to be. As long as it is consistent for a complete draw call and not changed per-vertex or per-pixel it will be rather fast. These are best for things that could vary per-model or per-draw.... as changing a uniform is cheaper than switching shaders.

You can also have multiple PBR Effect classes broken up by a few major categories of usage. Not sure what those would be off the top of my head.

Beyond that... compiling large shaders with lots of permutations in MGFX is super slow. We've never looked at optimizing the shader compiler process, but maybe we should.

Finally we currently load all the shaders into memory when we create an Effect. This means we create GPU resources for tons of permutations that are not ever used. Maybe we should lazy initialize shaders/programs as required instead? This could speed up Effect creation and reduce some GPU resources.

[vpenades]

@tomspilman

One thing to do is move some of the work to be runtime constant conditionals instead of a different permutation.

Already doing that for most minor features...

You can also have multiple PBR Effect classes broken up

PBR is not an effect but a family of effects, being the two most common ones MetallicRoughness and SpecularGlossiness, but there's more in line: ClearCoat, Transmission... So I cannot break the "baseline" PBR effect because the derived ones would be a nightmare to implement. In fact the idea is that the baseline PBR effect to be inherited by developers to expand on it.

compiling large shaders with lots of permutations in MGFX is super slow...

Indeed, given that the main PBR effect has 512 permutations, it takes a while to compile.... I was wondering which is the current state of #7345 , and if using ShaderConductor will bring some benefits over MojoShader on these issues.

I have no clue how other engines handle the shaders stuff... I want to believe that over time, there's better approaches to this issue... I've heard about übershaders, but I don't know how they work. Maybe MonoGame needs some kind of architecture that goes beyond the Effect classes, because the current architecture is reaching its limit with PBR, and we haven't even began talking about adding shadows or any other additional feature.

Maybe we should lazy initialize shaders/programs as required instead?

I also thought about it, and also falls into the "I have no clue how other engines work"... my guess is that effects would be compiled on demand and cached. But then, this would only work on desktop, where the shader compiler is available?

[tomspilman]

"I have no clue how other engines work"

Shader permutations are a fact of life in other engines. Having 1000s of shader permutations is common.

The main thing it hurts is content build times as you have to wait on 1000s of shaders to compile. But it also has a little impact on disk space, load times, and runtime memory usage if you have tons of unused combinations for your game.

It doesn't generally effect frame rate... so that is good.

Epic has some high level switches that let you force some shader features off for mobile to reduce permutations at content build time. This is similar to what i was suggesting by having pre-created variant PBR FX files.

But i could see it work via HLSL #include and having the user make a custom .FX for their game:

// MyPBR.fx File

#define PBR_ENABLE_FEATURE_Blah1
#define PBR_ENABLE_FEATURE_Blah2
//#define PBR_ENABLE_FEATURE_Blah3   
#define PBR_ENABLE_FEATURE_Blah4   
#define PBR_ENABLE_FEATURE_Blah5   
//#define PBR_ENABLE_FEATURE_Blah6

#include <pbr.fx>

Or use the shader macro feature of the Effect compiler or MGCB tool.

If GPU become fast enough eventually more settings can become dynamic uniforms and less build time permutations will be needed. But for now it is a fact of life to have 1000s of permutations in advanced renderers.

[tomspilman]

I've heard about übershaders, but I don't know how they work.

One big shader file you compile with different #defines to toggle features. So you're basically already doing this.

Maybe MonoGame needs some kind of architecture that goes beyond
the Effect classes, because the current architecture is reaching its limit with PBR

Our effect architecture is not much different than material architectures in other engines.

I think at most we may want to optimize the things i mentioned:

• Optimize our MGFXC shader compiler performance.
• Lazy load/create/clone GPU shaders and uniform buffers as used at runtime.

Those things would help a ton with having tons of permutations.

[cpt-max]

compiling large shaders with lots of permutations in MGFX is super slow...

I was wondering which is the current state of #7345 , and if using ShaderConductor will bring some benefits over MojoShader on these issues.

I ported enough shaders to ShaderConductor by now to be confident about it. It's a big change, so a few things might pop up once more people are using it, but I don't expect any serious problems.

Whenever you have the same shader in multiple techniques, MGFXC is indeed very wasteful with compilation time. It compiles the same shader over and over again for every technique. It then compares the compilation result to previously compiled shaders, notices that an equivalent shader is already present, and then discards the result.

The ShaderConductor branch still does it this way. I thought about doing it more efficiently, but decided against it. The idea being to rock the boat as little as possible, making the transition easier. Optimizations like this are better done in a separate commit.

It should be a super easy optimization though, unless I'm missing something. If a shader with the same name and shader profile has already been compiled, don't compile it again, done.

Another compile time optimization possibility with ShaderConductor would be to disable shader optimizations on demand. That's probably where most of the compilation time is spent. That could also help with the annoying problem where shader parameters are getting optimized away.

* Lazy load/create/clone GPU shaders and uniform buffers as used at runtime.

On the OpenGL side this is already happening for shaders. Only DX creates the GPU resources during effect creation.

[tomspilman]

If a shader with the same name and shader profile has already been compiled, don't compile it again, done.

Yeah I think that is it. I don't think we can pass any #defines in per-shader or anything. So maybe that is all we need to check.

That would be a good separate PR.

[vpenades]

Right now, with the most basic PBR effect, the number of available/expected permutation On/Off bits are:

• Vertex Skinning attributes
• Vertex Morphing attributes (reserved for when monogame actually supports it)
• Vertex Tangent Space attributes
• Vertex Color attribute
• Vertex TexCoord attributes

So for the vertex format alone, that gives 5 bits, which yields 32 techniques.

On the Pixel shader, the techniques are limited to the texture samplers, with most other details defined with shader constants:

• Normal Mapping
• Primary color texture (RGBA)
• Secondary PBR texture (metallic roughness)
• Emissive
• Occlusion
• IBL light (optional, but very desirable in order to have good looking PBR)

So it's 6 more bits, 64 more techniques for every vertex shader type.

In total: 32 * 64 = 2048 techniques

Pixel Normal Mapping only makes sense with Vertex Tangent Space, so that bit can be coalesced into a single bit, so we can cut it down to 1024 techniques...

Notice that, In the shader's I'm working with, the structure I'm passing from the vertex to the pixel shader -is always the same- , so in the end, it's going to be only 32 vertex shaders and 64 pixels shaders. so maybe things could go easier if I could set the vertex and pixel shaders independently.

Epic has some high level switches that let you force some shader features off for mobile to reduce permutations at content build time. This is similar to what i was suggesting by having pre-created variant PBR FX files.

This is something that could be done with this shader, by leaving IBL and maybe occlusion out of the equation for mobile. That would ensure that the shader has a maximum of 4 sampler states simultaneously.

[tomspilman]

so maybe things could go easier if I could set the vertex and pixel shaders independently.

I would look at how BasicEffect is setup...

https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Platform/Graphics/Effect/Resources/BasicEffect.fx#L456

With all the different combinations of pixel and vertex shaders pre-defined in techniques.

The C# side then just needs to decide which one is needed.

https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Graphics/Effect/BasicEffect.cs#L469

Could be for 2048 permutations this is way too much of a chore? Some helper macros could solve it however.

We could also expose enough in the C# Effect classes that it could create a new technique at runtime programmatically combining the right shaders.

Still the MGFX compiler will require all shaders to be compiled ahead of time.

[vpenades]

@tomspilman knowing that having thousands of techniques is fine is reassuring, because I was worried having so many techniques would cause problems or would be seen as an abuse of the technology.

I'm already using helper macros to build the techniques... in this case I am "abusing" the T4 templating files to make the FX entry points. The T4 are originally intended to templatize c# code... but they work just fine with fx code.

The techniques T4 file is here.

[tomspilman]

I was worried having so many techniques would cause problems or would be seen as an abuse of the technology.

It could be possible you run into bugs with that many techniques, but we should look to fix that.

Lots of permutations is something to expect now.

[cpt-max]

One thing to do is move some of the work to be runtime constant conditionals instead of a different permutation.

Already doing that for most minor features...

That's what I usually do even for bigger features. Fewer shaders to manage, less code duplication. Performance is fine. It looks like those scenarios get optimized by the driver to the point where it's pretty much like the if statement isn't even there, as long as the condition is based on a constant parameter.
I usually don't start a new shader unless I need different inputs or outputs, or the changes are so big, it would result in an if-else-djungle.

[tomspilman]

@cpt-max @vpenades

Yeah i know that having conditionals in your shaders is less of a performance hit than it used to be.

In particular i believe the GPU optimizes for cases where a conditional is fixed for the entire draw call and doesn't change per-vertex or per-pixel. I expect the conditional is not totally free of cost in these cases, but tremendously better than a conditional that can change per-pixel.

I think we would have to profile and test to know for sure. I have had trouble finding definitive information on this.

[vpenades]

@tomspilman as soon as it's been said... I've hit the first one #7391