Sokol: Read contents of offscreen render target?

[danhambleton]

Is it possible to access the pixels of an offscreen render target texture? Issue #171 seems to imply that reading the contents of the screen is not supported by sokol. Does this apply to render targets? My setup is similar to the one in the offscreen example:

    /* a render pass with one color- and one depth-attachment image */
    sg_image_desc img_desc = {
        .render_target = true,
        .width = 256,
        .height = 256,
        .pixel_format = SG_PIXELFORMAT_RGBA8,
        .min_filter = SG_FILTER_LINEAR,
        .mag_filter = SG_FILTER_LINEAR,
        .sample_count = MSAA_SAMPLES,
        .label = "color-image"
    };
    sg_image color_img = sg_make_image(&img_desc);

[floooh]

No, reading resource content back to the CPU side is generally not supported in sokol_gfx.h, that's also true for render targets. I'd suggest using the extension function approach described in #171 and calling the backend 3D-API specific functions required to get the data back directly.

[danhambleton]

Yep, makes sense (it is indeed very slow). I ended up implementing this little function:

/* extensions impl*/
SOKOL_API_IMPL void sg_read_texture_data(sg_image img_id, void* pixels) {
#if defined(_SOKOL_ANY_GL)
    _sg_image_t* img = _sg_lookup_image(&_sg.pools, img_id.id);
    const GLenum gl_img_format = _sg_gl_teximage_format(img->cmn.pixel_format);
    const GLenum gl_img_type = _sg_gl_teximage_type(img->cmn.pixel_format);
    GLenum gl_img_target = img->gl.target;
    GLuint gl_img_level = 0;
    glGetTexImage(gl_img_target, abc, gl_img_format, gl_img_type, pixels);
#endif

}

I found that you have to call this function after the second onscreen pass. That is, the first pass renders the texture off-screen, then the second pass renders it to a full-screen quad. Right before the sg_end_pass of that second step is when I needed to read the texture (otherwise all pixels were black).

[edubart]

How to do this on other backends (Metal/DX11/WebGPU), does anyone have this extension implemented for those other backends?

Reading pixels from render target textures is useful for my use case where I need to cache heavy rendering output to the disk and also to save screenshots to disk.

[Fra-Ktus]

I needed that and ended implementing a DownloadTexture function for DirectX 11 and Metal.

You can't read the texture directly and have to copy it first.

Here is the extract for Metal:

id<MTLTexture> tex = 0;

// get the texture from the sokol internals here...

if (tex)
{
    id<MTLTexture> temp_texture = 0;
    if (_sg_mtl_cmd_queue && tex)
    {
        const MTLPixelFormat format = [tex pixelFormat];
        MTLTextureDescriptor* textureDescriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:format
                                                           width:(width)
                                                          height:(height)
                                                       mipmapped:NO];

        textureDescriptor.storageMode = MTLStorageModeManaged;
        textureDescriptor.resourceOptions = MTLResourceStorageModeManaged;
        textureDescriptor.usage = MTLTextureUsageShaderRead + MTLTextureUsageShaderWrite;
        temp_texture = [_sg_mtl_device newTextureWithDescriptor:textureDescriptor];
        if (temp_texture)
        {
            id<MTLCommandBuffer> cmdbuffer = [_sg_mtl_cmd_queue commandBuffer];
            id<MTLBlitCommandEncoder> blitcmd = [cmdbuffer blitCommandEncoder];

            [blitcmd copyFromTexture:tex
                         sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(0,0,0) sourceSize:MTLSizeMake(width,height,1)
                    toTexture:temp_texture
                        destinationSlice:0 destinationLevel:0 destinationOrigin:MTLOriginMake(0,0,0)];

            [blitcmd synchronizeTexture:temp_texture slice:0 level:0];

            [blitcmd endEncoding];

            [cmdbuffer commit];

            [cmdbuffer waitUntilCompleted];
        }
    }
    if (temp_texture)
    {
        MTLRegion region = MTLRegionMake2D(0, 0, width, height);
        NSUInteger rowbyte = width*4;
        [temp_texture getBytes:pixels bytesPerRow:rowbyte fromRegion:region mipmapLevel:0];
        result = 1;
    }
}

[edubart]

@Fra-Ktus Thanks for the code, I've ended up with some similar code for Metal.

@floooh What's your opnion on this being officially supported by sokol_gfx.h? I've working functions for D3D11/Metal/OpenGL called sg_query_image_pixels(sg_image img_id, void* pixels, int size) to retrieve pixel data from any sg_image including render targets and sg_query_pixels(int x, int y, int w, int h, bool origin_top_left, void *pixels, int size) to retrieve pixel data from the current framebuffer, that I could make a PR.

[floooh]

What's your opnion on this being officially supported by

No objections, a PR would be welcome. I guess it's better to have any way to extract pixel data to the CPU then none at all, even if it's slow ;)

[Fra-Ktus]

@edubart Here is my DirectX11 code to download the bitmaps:

const ID3D11ShaderResourceView * res_view = 0;

// get resource view from sokol internals here

if (res_view)
{
    ID3D11Resource* tex = 0;
    ID3D11Texture2D* texture_copy = NULL;
    ID3D11ShaderResourceView_GetResource((ID3D11ShaderResourceView *) res_view, &tex);
    if (tex)
    {
        D3D11_TEXTURE2D_DESC description = { 0 };
        ID3D11Texture2D_GetDesc((ID3D11Texture2D*) tex, &description);
        description.BindFlags = 0;
        description.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
        description.Usage = D3D11_USAGE_STAGING;
        HRESULT h = ID3D11Device_CreateTexture2D(_sg.d3d11.dev, &description, NULL, &texture_copy);
        if (h == S_OK)
        {
            ID3D11DeviceContext_CopyResource(_sg.d3d11.ctx, (ID3D11Resource*) texture_copy, tex);
        }
        else
        {
            texture_copy = NULL;
        }
    }

    if (texture_copy)
    {
        D3D11_MAPPED_SUBRESOURCE mappedResource;
        HRESULT h = ID3D11DeviceContext_Map(_sg.d3d11.ctx, (ID3D11Resource*) texture_copy, 0, D3D11_MAP_READ, 0, &mappedResource);
        if (h == S_OK)
        {
            char * src_line_ptr = mappedResource.pData;
            char * dst_line_ptr = pixels;
            size_t s = width * 4;
            for (uint32_t v = 0; v < height; v++)
            {
                memcpy(dst_line_ptr, src_line_ptr, s);
                dst_line_ptr += s;
                src_line_ptr += mappedResource.RowPitch;
            }
            result = 1;
        }
        ID3D11Texture2D_Release(texture_copy);
    }
}

[danhambleton]

FWIW, we have started to collect some of these methods in a separate sokol-ext repo. It would be great to add the metal and DirectX implementations.

[floooh]

To be honest I'm a bit undecided again on whether this should go into the core API after sleeping over it...

Maybe it would indeed make sense to put it into an extension header after all (there could be an "exts" directory in the sokol repository similar to the "utils" directory.

(the point being that reading out pixels "naively" is a blocking operation on all APIs, and a proper async version is either a lot more effort, or impossible (e.g. GL))...

[iryont]

It is not like it would cause any harm, but in some cases it is heavily important to have such ability (rendering expensive stuff which is cached within a texture).

Asynchronous reading would be difficult, indeed, but I think that overall it is not needed as the caller should know exactly how expensive such request is and that it needs to be used with care.

Anyway, @Fra-Ktus - you are forgetting about one important thing. Sokol gfx has an internal buffer which needs to be committed before attempting to read back, otherwise you might end up with gibberish data like I did experience myself (rendering to an offscreen target to capture it and cache right away after ending pass for a framebuffer).

Also, capturing cannot work inside an on-going pass in APIs with command buffers, so before capturing following example should be included:

SOKOL_ASSERT(!_sg.mtl.in_pass);
if(_sg_mtl_cmd_buffer) {
    #if defined(_SG_TARGET_MACOS)
    [_sg_mtl_uniform_buffers[_sg.mtl.cur_frame_rotate_index] didModifyRange:NSMakeRange(0, _sg.mtl.cur_ub_offset)];
    #endif
    [_sg_mtl_cmd_buffer commit];
    [_sg_mtl_cmd_buffer waitUntilCompleted];
    _sg_mtl_cmd_buffer = [_sg_mtl_cmd_queue commandBufferWithUnretainedReferences];
}

It's a piece of code from _sg_mtl_commit without encoding presentDrawable because we don't want to do that yet.

[Fra-Ktus]

@iryont - Yes, I am doing that.
For Metal I had to write a custom CommitRenderToTexture function and for DirectX, it's enough to call sg_commit() before reading the content of the buffer.

See the shader browser I build using those functions to read back the content of rendered shaders...
https://youtu.be/RsO5wQhZ3JM
I grab 30 frames for each shader and loop it on the interface to give a preview to the user.
Browsing that many shaders in parallels is not possible on most standard systems.
And by default the Sokol gfx does not have that many slots.

[iryont]

Yea, you did show a good example. It is precisely my point why reading back is an important feature in some cases since real-time rendering would be rather impossible due to required computation power.

[freneticmonkey]

Sorry for digging this post up. I'm trying to use this implementation for picking for my editor that I'm currently working on but I'm seeing a bunch of strange problems that I now suspect are due to the ordering of the calls to glgetteximage() and other operations. I suspect the function above is missing a call to bind the texture before reading it.

[MikeHart66]

Would be super cool to have something in the official distro for reading pixel from a rendertarget texture. +2000 to that

[floooh]

For now I'd prefer this in some sort of optional "extension header". I'm planning an overhaul of the entire resource management area for copying data into and between resources (might be a while though). Maybe that's an opportunity to tackle the "copy data out of resources" problem, but any solution that's created before would be broken anyway.