Filament: Offscreen rendering

[molysgaard]

Is your feature request related to a problem? Please describe.
I am using Filament in a game-like project.
In it there is a free moving camera, controlled by the user. This camera is rendered on screen. This works great.
In addition, some of the entities in the game have their own cameras, eyes. The rendered image stream from these cameras will be used in computer vision algorithms, which again is feed into the AI controlling these entities in the game.

What would be the path of least resistance to implement this using the current architecture of Fillament?

Describe the solution you'd like

• What the entities sees with their "eyes" does not need to be rendered on screen.
• The rendered views would preferably be available as a buffer of pixel values in user process memory space. Given my limited understanding of the intricacies of GLX/OpenGL programming I imagine rendering to a GPU texture instead of the GPU framebuffer, and then copying this texture back into process memory using some driver call?

I have been trying to understand the current architecture of the rendering system. To my understanding a solution would be to write a custom SwapChain. Looking into this it seemed like there was a tight coupling between SwapChain and, in my case, GLX. Does this mean that everything you render has to go on screen?

I do have time available to implement this feature if given the right directions on how to do it.

Describe alternatives you've considered
It would also be usable to render all the views on screen, and then copy the screen framebuffer back into the user process, but it feels awkward if there are 10s or 100s of entities, each which own their own little box of the physical screen to render their camera.

OS and backend
I am using Linux with the OpenGL backend for my project.
To my understanding usage of the finished feature is OS and backend agnostic, but the actual implementation could require different implementations per OS and backend.

[bejado]

I don't think we currently support rendering a camera to an arbitrary texture, so that's something that needs to be looked into. For the "copying back into process memory" part, take a look at what we're doing here for FilamentPanel

[molysgaard]

If I would like to make this happen, where should I start looking in the code? What components would have to be changed?

[pixelflinger]

@molysgaard this is a feature we will be working on very soon.
Note that internally this is already completely supported. You could look at ShadowMap.cpp to see how to set-up a texture as a render target. Another place to look for examples is PostProcessManager.cpp.

Basically you need to create a render target from a texture:
auto handle = engine.getDriverApi().createRenderTarget(..., textureHandle, ...);

Then, in FRenderer::renderJob() it uses the render target it gets from getRenderTarget(), instead you'd need to modify the code so it uses yours. It could be passed as a parameter or through some other way.

The real implementation and user facing API would be more involved, but that should be enough to unblock you in the meantime.

[molysgaard]

For information when you get around to implementing the public API I leave the details of how we ended up making a temporary solution.

Changes to filament for plumbing a custom rendertarget here: https://github.com/google/filament/compare/master...staaker:offscreen-rendering-staaker

To create the texture and the rendering target the following was done from the application code:

static int OFFSCREEN_DIM = 400;
_offscreenTextureHandle = static_cast<filament::details::FEngine*>(_engine)->getDriverApi().createTexture(
        filament::driver::SamplerType::SAMPLER_2D, 1, filament::driver::TextureFormat::RGBA8, 1, OFFSCREEN_DIM, OFFSCREEN_DIM, 1,
        filament::driver::TextureUsage::COLOR_ATTACHMENT);

_offscreenTextureRenderTarget = static_cast<filament::details::FEngine*>(_engine)->getDriverApi().createRenderTarget(
        filament::driver::TargetBufferFlags::COLOR, OFFSCREEN_DIM, OFFSCREEN_DIM, 1, filament::driver::TextureFormat::RGBA8,
        {}, { _offscreenTextureHandle }, {});

In the rendering loop, the following was added to render and read out from the GPU texture memory:

std::array<uint8_t, OFFSCREEN_DIM*OFFSCREEN_DIM*4> pixelData;
if (_renderer->beginFrame(_swapChain)) {
    // render the screen
    _renderer->render(_view);


    //  TODO DEBUG RENDER TO CUSTOM TEXTURE CONTEXT AND READ BACK FROM GPU MEMORY USING readPixels
    _renderer->render(_offscreenView, _offscreenTextureRenderTarget);

    static_cast<filament::details::FEngine*>(_engine)->getDriverApi().readPixels(_offscreenTextureRenderTarget,
            0,      //xoffset
            0,      //yoffset
            OFFSCREEN_DIM,    //width
            OFFSCREEN_DIM,    //height
            filament::driver::PixelBufferDescriptor(
                    reinterpret_cast<void*>(pixelData.data()),   //Pixel buffer
                    pixelData.size(),                               //Byte size
                    filament::driver::PixelDataFormat::RGBA,        //Bit depth
                    filament::driver::PixelDataType::UBYTE          //Alignment
            )
    );

    _renderer->endFrame();
}

Seems to be working well.

[shartte]

Just for the sake of adding more uses for this:

• Draw 3D scenes seamlessly into a user interface (in my case a 3D preview of the character in an RPG character sheet)
• Do custom post-processing effects like outline shaders (this might be doable with multiple-render-targets more efficiently)
• Mouse picking with multiple render targets

[bhack]

An EGL headless solution it is useful.
GLX is not available everywere.
See https://github.com/nvidia/nvidia-docker/wiki/Frequently-Asked-Questions#is-opengl-supported