Postprocessing: Efficient way to share composer render state

[jpweeks]

I think it would be useful to have an efficient api for composing input / output between EffectComposer chains. This could be used for creating a texture feedback effect, for instance (my current use case).

Currently, the only way to save render state for use as input in another effect chain is to use the SavePass, which runs a full-screen render pass to copy pixels to a render target, in combination with a TextureEffect. This is pretty inefficient, and in many cases preserving the composer render target state isn't necessary.

I'm currently using a SaveSwapPass as an initial attempt to address this need. Any thoughts on other ways to address this use case?

[vanruesc]

Hm, I don't fully understand your use case. Why do you want to use multiple composers? Wouldn't it be easier to create a custom Pass that encapsulates other passes and effects to render something into a dedicated render target for further processing?

If that doesn't work in your setup, could you please provide a concrete example of your pass chains? I'll take a closer look then.

[jpweeks]

This is the render flow I'm using:

I couldn't see any way to implement this with a single EffectComposer since its effect chain only supports a linear progression of effects (as far as I can tell).
I can try and put together an example implementation of this when I have a bit more time, just wanted to get your thoughts on implementing a non-linear effect chain.

[vanruesc]

Sorry for the delay!

I'm afraid I still don't understand your render setup 😅 I'm probably missing something.. an example would help a lot. Right now I only have a vague idea of what you might want to accomplish and that's not enough to provide a meaningful opinion.

[vanruesc]

Just a heads-up: there hasn't been much activity in this issue lately, so I'll close it in a few days unless you'd like to revisit it.

[vanruesc]

[From #145] Yeah, you can reach into an effect and grab render target textures, which is what I currently do, but this feels a bit hacky, and can break if the effect or composer swaps a render target.

You could make it clean by defining a getter on your custom effect called texture which returns renderTarget0.texture while declaring other class members as private via documentation. renderTarget0 will always point to the most up-to-date image data because of the swapping in FeedbackEffect.update().

Swapping shouldn't interfere with your rendering when you're working with effects; it's true that the EffectComposer swaps buffers, but it only swaps the main input and output buffers at certain times between passes to avoid reading from and writing to the same buffer. Once you're in effect land, there's only the input buffer.

[From #145] Is there an API for branching? The only way I could find to do it with the current API is to have multiple composers.

If by branching you mean if/else, then you'd need to create a thin custom Pass that holds other passes and calls their render method depending on some condition.

If by branching you mean parallel rendering, then no, there's no API for that.

If by branching you mean feeding the result of some effect or pass to one or multiple other effects or passes, then no, there's no API for that. There is also no automatic buffer management for passes and effects. I haven't really explored this yet, but I think that it would be too error-prone.

[From #145] it'd be nice to be able to write an effect graph similar to how Blender's material editor works

I'm not too familiar with Blender, but an effect graph would basically be a post processing setup through data, right? Like a JSON description of effect relationships and settings. It should be possible to build such a system on top of this library.

[jpweeks]

Yeah, by branching I mean parallel rendering; not literally running at the same time, but effects using the same render target as input and being used together in some effect further down the chain.

I don't have an API proposal in mind yet, but am thinking providing a more imperative flow similar to Wagner could be a start to making feedback and possibly branching work more easily.

[vanruesc]

effects using the same render target as input

That shouldn't be a problem. I think what you mean is that you want to somehow preserve and use the internal main buffers of the EffectComposer without copying them via the SavePass. I don't think that's possible. Trying to track one of those buffers to predict when its contents will be lost seems unfeasible.

and [effects] being used together in some effect further down the chain.

This should also not be a problem, or at least I don't see the problem that you're hinting at.

I don't have an API proposal in mind yet, but am thinking providing a more imperative flow similar to Wagner could be a start to making feedback and possibly branching work more easily.

Sounds interesting! If you have any suggestions on how the current API could be improved, I'd like to hear them. Wagner, however, is severely outdated. It was great at the time, but it's 38 versions behind three. The uniform reflection logic that it uses is obsolete, it swaps read/write buffers just like the original EffectComposer from three and it also doesn't seem to have any form of buffer management.

[jpweeks]

Wagner, however, is severely outdated.

Agreed, I just meant the API design where each pass is a function call rather than an internal array of EffectComposer.

[vanruesc]

Looks like this issue stagnated again.

Unfortunately, it is still not clear to me what you're trying to implement. I understand that you'd like to feed a render result back into your effect chain but I don't understand why you need two EffectComposers and a SaveSwapPass for that.

It would be great if you could provide an example setup that demonstrates the kind of effect chain you're trying to create. I'd like to see an actual problematic setup in action.

If there's no activity, I'll close this issue in a few days. If you don't have enough time right now, just re-open this issue later.

[jpweeks]

I'll work on putting together an example.
In the meantime, here's an example of the effect chain I've described above, implemented with a lower-level library: https://jsfiddle.net/jpweeks/yLv9mdge/

[vanruesc]

Thanks for the example!

This is how I would implement it with postprocessing:
https://codesandbox.io/s/postprocessing-feedback-k7s2p

The pass chain looks like this:

composer.addPass(feedbackConsumerPass); // Custom Pass
composer.addPass(renderPass);
composer.addPass(smaaPass); // Optional
composer.addPass(feedbackProducerPass); // FeedbackEffect (EffectPass)

The FeedbackConsumerPass draws the result of the FeedbackEffect and implements scaling/translation. The subsequent RenderPass only clears depth and renders the scene on top of the feedback colors because it always renders into the input buffer. Even if that wasn't the case, the FeedbackConsumerPass could easily be modified to put the feedback colors into the right buffer.

The FeedbackEffect is primarily a TextureEffect that writes the input colors to screen. In its update method, it blurs the result of the previous pass and writes the result to a buffer which serves as input for the FeedbackConsumerPass.

This is an advanced effect chain which is more complicated to implement than most other setups. However, it is possible to implement it with the tools that postprocessing currently provides.

[jpweeks]

Ah interesting, thanks!
I think this is a good solution to my presented case, and a good example of working with the update and render hooks in creative ways. Thanks for your patience and the great library 😀