Imagesharp: Easy to use API for inline pixel-shader-like processors

Created on 12 Nov 2019  路  3Comments  路  Source: SixLabors/ImageSharp

This issue follows the conversation with @antonfirsov and @JimBobSquarePants on Gitter.

API description

An easy to use API to apply custom pixel-shader-like effects to images of any type. By pixel-shader-like I mean an effect that applies a given transformation to all pixels in an image independently, with no need to have information about surrounding pixels.

Such an API could be structured like this:

public delegate void PixelProcessor(Span<Vector4> pixelSpan);

public static IImageProcessingContext ApplyProcessor(this IImageProcessingContext image, PixelProcessor processor);

Behind the scenes, ImageSharp would create an instance of a dedicated processor that takes the PixelProcessor instance as parameter, then use ParallelHelper.IterateRowsWithTempBuffer to go over the pixels of the target image. On each batch of pixels, it'd use the optimized PixelOperations<TPixel>.ToVector4 API to write the target pixels as Vector4 in the target buffer, and then pass it as parameter when invoking the input PixelProcessor. It'd then use PixelOperations<TPixel>.FromVector4 to copy the processed pixels back into the target image.

Additional overloads would be needed to also take a Rectangle to specify the area to apply the processor on, and a PixelConversionModifiers to indicate the type of pixel conversion to use, if needed.

Rationale

With the introduction of the pixel-format agnostic Image class, writing custom processors has become rather complicated for new users and for developers not familiar with the architecture of ImageSharp - a processor needs a non-generic class that implements IImageProcessor, with a method that creates a generic type of the processor (another user-defined class) that then applies the desired effect on the target image. Many developers also have difficulties understanding how to properly apply effects to an image of an arbitrary pixel-format, often making mistakes such as converting each pixel into a "well known" format like Rgba32, and then operating on each color channel separately, etc. Improper parallelization is also a common issue for custom processors.

With the proposed API, all the pixel-format operations, rows batching etc. would be completely transparent for the end user. Also, this would have the advantage of helping users leverage the SIMD features of the Vector4 class, as they'd be able to use classic operators on an entire pixel with ease, as opposed to having to handle each color channel separately.

Use case example

This entire processor implemented in this PR could be implemented using the proposed API with just this code, as opposed to having to create two custom classes for the processor, and having to deal with the parallelization and pixel-format conversions manually:

image.Mutate(c => c.ApplyProcessor(span =>
{
    var factors = new Vector4(0.2125f, 0.7154f, 0.0721f, 0);
    for (int i = 0; i < span.Length; i++)
    {
        ref Vector4 v = ref span[i];
        float w = v.W;
        float luminance = 1 - (Vector4.Dot(v, factors) * 0.6f);
        v *= luminance;
        v.W = w;
    }
}));
enhancement feature request

Most helpful comment

@Sergio0694 we need a better (probably more verbose) name than ApplyProcessor, but otherwise, feel free to grab it!

All 3 comments

@Sergio0694 we need a better (probably more verbose) name than ApplyProcessor, but otherwise, feel free to grab it!

@antonfirsov Awesome, great! And yeah I agree, that name for the API was just a temporary placeholder, we might want to pick a better and more self descriptive name there. I'll get started with some groundwork then and open a draft PR we can use to track this then! 馃槉

Fixed with #1065

Was this page helpful?
0 / 5 - 0 ratings