Monogame: Add Geometry Shader Support

[tomspilman]

Extend the Effect system and pipeline to support geometry shaders (possibly hull and domain shaders as well). We need to see what support for these other shader stages is under various platforms and decide what platform we can support.

[cdhanna]

As a monogame user, I would love to see geometry shaders implemented.

[tomspilman]

I'm going to try to tackle this in the next few weeks. What I need to do:

• Extend the MGFX parser to support the extra stages.
• Compile the geometry shader and extend the MGFX binary format.
• Add runtime support for the geometry shader stage.
• Add docs showing how GS works under MG.

I am going to skip on OpenGL support for the first pass and just focus on the DX platforms which I know best. In theory this should be supportable on most GL platforms via extensions.

We will have to figure out some way to signal when this feature doesn't work so games can fall back to a non GS path.

[tomspilman]

Actually GL support for geometry shaders will be dependent on supporting GLSL directly (see https://github.com/mono/MonoGame/issues/2167) as MojoShader doesn't support GS in any way.

[cdhanna]

Okay. I shall eagerly await :)

[stoffemi]

Any news on this?

[SliNGz]

Definitely a needed feature. Would really like to see this get done 😃

[georg-eckert-zeiss]

Really hope for this : )

[cpt-max]

I made a MonoGame branch that contains hull, domain and geometry shader support for WindowsDX
https://github.com/cpt-max/MonoGame/tree/shader
It's based on this fork from JSandusky:
https://github.com/JSandusky/MonoGame

Here's a sample MonoGame project with all three shader stages:
https://github.com/cpt-max/ShaderTest

Still want to do some more testing etc., but I'm planning to make a pull request soon.

[tomspilman]

@cpt-max - I assume this is all for DirectX only.

What would be the plan for OpenGL? I suspect we have to kill or replace the old MojoShader path to support it there?

I know Vulkan will support these new stages, but does Metal support it?

[cpt-max]

Yes, it's DirectX only.

To me ShaderConductor looks like the most promising replacement for MojoShader.
https://github.com/microsoft/ShaderConductor
It's currently missing reflection features. There's an open pull request for this though:
https://github.com/microsoft/ShaderConductor/pull/45

Metal currently doesn't have geometry shaders, but it does have tesselation. It seems to be functionally equivalent to hull and domain shaders:
https://developer.apple.com/library/archive/documentation/Miscellaneous/Conceptual/MetalProgrammingGuide/Tessellation/Tessellation.html
There's a section on the bottom of this page that could be of interest: Porting DirectX 11-Style Tessellation Shaders to Metal

[jnoyola]

It's currently missing reflection features. There's an open pull request for this though

Side-note about ShaderConductor: The maintainer had originally given me a super rough estimate that reflection would be supported by the end of 2019, but that was obviously dependent on this PR. Unfortunately the author of that PR hasn't been active on github, let alone this PR, for over a year.

[tomspilman]

To me ShaderConductor looks like the most promising replacement for MojoShader.

Does it generate shader source code for us to compile separately? Or does it generate final shader binaries?

I ask because we need to support console with whatever system we decide on. Consoles generally have proprietary shader binary formats, but generally work from existing shader languages (glsl, SPIR-V, hlsl, etc).

Note pretty much all the consoles support some sort of HLSL cross compiler to their native format. So we could still support that path if we need to.

Unfortunately the author of that PR hasn't been active on github

Well that PR isn't very complex. If anyone finds time we should try to take it over and resolve the fixes they requested in the PR.

[cpt-max]

Does it generate shader source code for us to compile separately? Or does it generate final shader binaries?

It generates readable code. First it uses DirectXShaderCompiler to turn HLSL into SPIR-V, then it uses SPIRV-Cross to go from SPIR-V to GLSL, ESSL and MSL.

[cpt-max]

I made a first attempt at replacing MojoShader with ShaderConductor in MGFXC. Translating a very simple test shader to GLSL and then running it in DesktopGL already works. There's still a lot missing (no samplers, no constant buffers, etc).

I ran into a caveat though. MonoGame uses D3DCompiler for compiling HLSL, while ShaderConductor uses DirectXShaderCompiler. There are some differences between them in the HLSL they accept, more details here:
https://github.com/microsoft/DirectXShaderCompiler/wiki/Porting-shaders-from-FXC-to-DXC
For example DX9 style texture lookups like tex2D don't work anymore, it has to be texture.Sample()

If we want to go down that path, it's probably best to still keep the MojoShader path alive for shader model 2 and 3 in order to not break existing projects, and only use ShaderConductor when SM4 or higher is used.
Of course that means a SM4 shader might compile fine for DX, but then fail for GL. I see two options to deal with that situation:

• Fix up the HLSL as much as possible in MGFXC before handing it off to ShaderConductor, e.g. turn tex2D into texture.Sample()
• Switch to DirectXShaderCompiler also for DX. Existing HLSL might need to be modified, but at least the behaviour is consistent between DX and GL. To not break existing projects this option could be activated on demand only, either by detecting that SM4 is used for GL, or by manually adding something to the HLSL code to mark it as cross platform.

[tomspilman]

For example DX9 style texture lookups like tex2D don't work anymore, it has to be texture.Sample()

That isn't the end of the world... we could live with making that required on all platforms moving forward if that was best.

DirectX 9 is now 18 years old.

Fix up the HLSL as much as possible in MGFXC before handing it off to ShaderConductor,
e.g. turn tex2D into texture.Sample()

That sounds like an interesting idea. If that works then maybe we don't need to keep MojoShader.

[peon501]

If you are going to change OpenGL please don't above OpenGL 3.30. Direct3D SM 4.0 is equivalent to GLSL version 3.30 so I think I would be ok, but above you will alienate a lot of user base as me.

[peon501]

and it has to run on Linux via wine or natively.

[tomspilman]

If you are going to change OpenGL please don't above OpenGL 3.30.

Sure... for MG 3.x we should stay as compatible as possible with older hardware. We want to support SM 3/4 hardware.

I don't think would change that until MG 4.x and make bigger changes to the APIs. And at that point i expect we'll be more focused on Vulkan, DX12, and Metal. Depending on how compatible Vulkan is we may or may not need OpenGL after that.

[cpt-max]

I made some progress on the Mojo->ShaderConductor switch. I forked ShaderConductor to add the needed reflection data:
https://github.com/cpt-max/ShaderConductor/tree/reflection.
Here is the MonoGame branch for the ShaderConductor switch:
https://github.com/cpt-max/MonoGame/tree/shaderconductor

It's based on my pull request for hull, domain and geometry shaders. Here you can see the ShaderConductor changes only.

It's actually pretty close to complete. The main things missing are array parameters and GLSL ES output for mobile platforms. It already passes all DesktopGL unit tests except for one. The Invert.fx shader fails because one line gets ignored. Still need to figure out why that's happening.

if (!any(color)) return color;

There's a couple of open questions:

- Target OpenGL version:
I set the target now to OpenGL 3.30 as @peon501 suggested. I think that's a good choice, at least for the Desktop. It's the SM 4.0 equivalent, ShaderConductor doesn't like deprecated SM 3.0 HLSL anyways. Also I'm using uniform buffers for updating parameters (equivalent to DX constant buffers) which require at least OpenGL 3.10. It's possible to go lower, all the way down to OpenGL 1.10 (used by Mojo), but then shader parameter locations have to be queried at runtime and the parameters need to be updated individually. MojoShader solved this by packing all parameters into a giant float4 array. ShaderConductor has no such option.

Optionally you should be able to go higher though. After all I want to enable the new shader stages also for OpenGL. I would suggest to always pick the equivalent OpenGL version based on the chosen shader model:
http://renderingpipeline.com/2012/03/shader-model-and-glsl-versions/

- 32bit support
I noticed that there is a 64 and 32 bit version for Mojo, does that mean we also need a 32 bit version for ShaderConductor?

- std140 layout
That's probably the biggest annoyance in the current version. All global shader parameters are packed into a uniform buffer using std140 alignment rules. That basically means floats have to be 4-byte aligned, float2's are 8-byte aligned, float3 and float4 are 16-byte aligned. The best strategy is to have all float4's first, pad float3 to float4, then have all float2's, and regular floats go last.

I haven't found an option in ShaderConductor to automate this rearranging of parameters, it just keeps the order in which they are declared. This means if you have a float2 followed by a float everything is fine, but if you switch them around it violates std140, because the float2 is now at a 4 byte offset.
Miracously not a single one of the stock/unit test shaders ran into this issue.

The options are:
1.) Leave it up to the shader programmer to rearrange parameters.
2.) Postprocess the generated GLSL in MGFXC to rearrange the parameters.
3.) Somehow get ShaderConductor to do that, which I think it should.

- Deprecated SM 3 HLSL
I mentioned this before, there are some older HLSL features not accepted by ShaderConductor.
Now I have some practical experience with this. There are 3 things I had to change so far:
1.) COLOR output semantic on the pixel shader needs to be changed to SV_TARGET
2.) tex2d, tex2dlod etc. needs to be replaced with texture.Sample. That also means you can't just have a sampler without also declaring a texture in the shader.
3.) Negative of a matrix doesn't work, so -mat needs to be changed to mat*-1. I think this is a bug so I opened an issue for this.

- OpenGL uses column major matrices
OpenGL matrices need to be transposed compared to DX. With Mojo that wasn't necessary as it doesn't use matrix parameters at all, it uses multiple vectors instead. I'm now doing the transposing in EffectParameter.cs, whenever the matrix parameter changes, so the

internal object Data { get; private set; }

byte array already stores the transposed version. The current implementation is not efficient performance-wise, I just wanted to make it work for now. I'd like to know first if that's the right place. The alternative would be to keep EffectParameter.Data the same as DX and do the transposing when the ConstantBuffer gets updated.

[cpt-max]

Most of the problems have been solved. PR #7345.

[cpt-max]

Forgot to mention this here, PR #7352 is finally up.

[willmotil]

_So VTF (vertex texture fetch) will now also work in OpenGL, before it only worked in DirectX._

That's awesome max that was a sorely missed feature.