Godot: Stencil support: List of use cases and ideas

[Windfisch]

I think Godot should offer support for stencil buffers to avoid rendering overhead or do fancy effects. However, there are many ways to use them, and it might be difficult to add this feature so it's both easy to use and flexible enough.

The following should become a collection of ideas what the stencil buffer can/will be used for, and what things must be respected in its implementation. Please feel free to add ideas and to discuss :).

---

I'll start here:

• Support for portal-like games: One camera A for the player and another camera B inside a viewport with "the environment on the other side of the portal". That "other" camera B should only render the scene where A can actually see a portal.
• This only requires one bit depth in the stencil, the ability to render objects into the stencil only (setting or clearing bits there), and the ability to do a simple on/off stencil test in the camera.

Possible problem with any implementation: The Godot engine itself may want to use the stencil buffer for its own effects in the future (?). Maybe we can only expose a part of the buffer?

[reduz]

My take on Stencil is that no matter if we eventually add low level support for it, we should support the high level use cases out of the box, then decide whether its worth supporting a low level API.

[reduz]

Use cases that come to mind:

• Removing silhouettes of some objects (clipping them out of geometry using masks).
• Shadow volumes (may not be worth, but some modern games like Mario Odyssey support them for the character)

[antoniotorresm]

The most basic use case is outlining meshes. The current way of outlining meshes in Godot is quite hacky and doesn't get good results (even the example in the docs shows seams between the mesh and the outline).

Another cool use case is to do x-ray vision, like in Overwatch. (Note how back faces are also shown).

Of course, you can use it for "masking" objects. With this you can apply an scene-wide post-process shader, but only to certain objects (those that write to stencil).

These are the main use cases that come to my mind, but there are a lot more. UE4 even has an extension for it called 'Custom depth' which is super useful (you can find some effects there).

[CptPotato]

It's also useful for VR rendering to mask out areas that are not seen on the HMD. For example the OpenVR api can retrieve a stencil mask (if the device/manufacturer provides one).

[Windfisch]

• Removing silhouettes of some objects (clipping them out of geometry using masks).
  you mean like this? https://www.youtube.com/watch?v=9pWFCji7StY

• Shadow volumes (may not be worth, but some modern games like Mario Odyssey support them for the character)
  I think that shadow volumes might be a very special case; I'd rather not support them than to have no, or a too complex, stencil support exposed to the user. (Of course, if there's a nice way to support them, that's great).
  But let's keep them in mind.

Masking objects sounds great; I assume we'll need to set single bits in the stencil for this, at least if we want to support multiple post-proc-effect types.

As for outlining objects, there's a short article here: https://rdmilligan.wordpress.com/2017/08/27/outlining-objects-with-opengl-stencil-testing/. tl;dr: Draw the object into the scene and also into the stencil buffer. Then draw the mesh again, but 1.2x larger, in the desired border color, but only where the stencil hasn't been set (i.e. don't overlap the actual object). This would require a strict sorting of objects, does Godot already support this (for non-transparent objects, and other than "the order in which they appear in the scene tree")?

[reduz]

Outlining and X-Ray using Stencil is overkill IMO and most of these effects mentioned can be done much better adding custom g-buffer support (plannig that for the Vulkan renderer).

This is why I mean, that Stencil makes the most sense to me for masking objects or creating shadow volumes, and not much else.. or effects like Planar Reflections or Portals. Other than that, I think their use is very limited.

[io12]

I am working on a seamless portal Godot asset and stencil buffering would be very helpful for performance!

[avencherus]

Not familiar stencil buffers, but it is looking like a way to do many of the effects I'd be puzzling over for a long while. There is a lot of situations where I have a 2D puppet character made of dozens of small sprites, but I want to put a directional gradient over them, or create a selection highlight.

I would need the final combined texture just for those nodes to do this, otherwise a material on all those sprites would only apply to itself, with undesired results.

[clayjohn]

@avencherus It sounds like you need a Viewport, not a stencil. Stencil buffers are used for masking effects. If you want to apply a gradient or an outline to a group of Sprite you should use a Viewport.

[Banduck]

@clayjohn
But the problem with viewports is: when you scale the window, viewports become pixelated. This can't be the best solution.

[avencherus]

@clayjohn Thanks, I've heard this to be quite expensive, have not had a chance to profile something like that on dozens of characters. And as Banduck suggests it probably has other quality side effects.

The best thing I've thought of trying is having a script keep track of the sprites, then frame by frame using the VisualServer to draw them onto a new canvas using their textures, ordering, and transform information. Then attach the canvas material to that.

Haven't yet gotten to a point where I will need to experiment with it yet. No idea what the performance would be, but all solutions so far seem like they're doing the work the engine has already done all over again.

[Calinou]

@Banduck This can be solved by resizing the viewport and downscaling the ViewportContainer or Sprite that displays the viewport to compensate for the increased size. I've done this in the "3D in 2D demo" in https://github.com/godotengine/godot-demo-projects/pull/341, but it hasn't been merged yet.

[clayjohn]

@Banduck in addition to what Calinou has said. Stencil buffers will have the same issue. If you alter the window size you would have to change the size of both your stencil buffers and your frame buffers (Viewports)

[avencherus]

@Calinou I looked at the 2D/3D example in your branch using 3.1. I was getting 1-5 FPS while running the demo.

This is outside of the topic, and wondering if I should open an separate issue.

I noticed it relieved when I switched to an empty scene in the editor, so the editor must be having some issues with resources. Does this PR maybe address this? https://github.com/godotengine/godot/pull/29297

I can understand the feature as an optimization decision for a large 3D project, but not as a requirement in situations like this, where there are a few basic pieces of geometry.

[Banduck]

@Banduck This can be solved by resizing the viewport and downscaling the ViewportContainer or Sprite that displays the viewport to compensate for the increased size. I've done this in the "3D in 2D demo" in godotengine/godot-demo-projects#341, but it hasn't been merged yet.

@Calinou I tried something like that, but the result wasn't satisfying. Maybe I did something wrong.
But besides that, there are other problems.
What if I only want certain objects to be inside the viewport node? The problem is, objects outside the viewport behave differently than objects inside the viewport.
For example Parallax Scrolling doesn't work anymore, because I need it outside the viewport, so that only certain objects have an outline effect.
Viewports are simply not a good solution... unless I somehow get it all wrong.

[puthre]

I was thinking on how would I add an inner shadow on a text and the stencil buffer came to mind. I don't have any idea on how it can be done otherwise.

[Wavesonics]

Even for 2D this would be really useful. Being able to render see-through effects when a character is behind another thing.

[io12]

There is a branch here that adds stencil support https://github.com/MillionOstrich/godot/tree/stencil-test and an example project using it https://github.com/MillionOstrich/godot-stencil-test. Could this be merged into master?

[Jummit]

Probably not until after 4.0.

[clayjohn]

@io12 @Jummit the GLES3 backend is being removed in 4.0. So there is no chance of stencil support being added to the GLES3 backend.

Stencil support is planned for the Vulkan backend for 4.0

[clayjohn]

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!