Monogame: Better imaging API

[TechnologicalPizza]

I'm working on a pretty simple imaging library for MonoGame that uses slightly optimized StbSharp libraries for reading/writing/resizing images.
Runtime font rendering is also planned by using StbTrueTypeSharp.
Inspiration for the API comes from SixLabors's ImageSharp.

(I don't know how smart it is to use a heavily modified fork of StbSharp, the performance improvements and features were too great to give up on for my project at least, but the original StbSharp can be used for this imaging library with some minor tweaks).

The current API already has some nice features:

• Progress reporting supplied with the currently processed frame, giving you access to decoded pixels in real-time and cancellation through CancellationToken.

• Image formats are trivial to add and the built-in codecs can be overwritten by user-defined ones. Both custom formats and codecs can be added at runtime (e.g. you could use Android or Windows Imaging Component codecs which may be better than the built-in ones).

var customFormat = new ImageFormat(
    name: "The Foo Format",
    primaryMimeType: "image/foo",
    primaryExtension: ".foo",
    supportsAnimation: false,
    associatedMimeTypes: new HashSet<string>("image/foo"),
    associatedExtensions: new HashSet<string>(".foo")
);
ImageFormat.AddFormat(customFormat);

// Both IImageEncoder and IImageDecoder are interfaces that derive from IImageCoder.
Image.AddEncoder(customFormat, new IImageEncoder()); // insert custom encoder instance here
Image.AddDecoder(customFormat, new IImageDecoder()); // insert custom decoder instance here

• The API is easy to use:

FrameCollection<Color> framesFromStream = Image<Color>.LoadFrames(stream);

var colorData = new Color[20 * 20];
var sourceRect = new Rectangle(5, 5, 10, 10);
Image<Color> imageFromPixels = Image<Color>.LoadPixels(colorData, sourceRect);

Where Color can be replaced by any type that implements the IPixel interface (currently implemented on all of the packed vector types courtesy of ImageSharp code).

• Simple API for image manipulation/processing:

image.Mutate(x => x.Crop(x, y, width, height));
image.Mutate(x => x.Resize(width, height));

I don't really plan on creating/implementing more than Crop and Resize, maybe some color stuff like Sepia or Grayscale and maybe some kind of simple Blur.

It can not be merged with the develop branch though as the library is using C#7, .Net472 and the System.Memory package. I thought about .Net5 and if MonoGame actually was .Net5 compatible, I would be able to turn this imaging library into a pull-request.
A pull-request would theoretically be possible as soon as MonoGame supported C#7 (and C#7 support is supposedly in the works).

I suppose that this addition slightly future-proofs the imaging part of MonoGame.
Comments and suggestions about this would be great.

[TechnologicalPizza]

This library would be a seperate project. If #6879 goes through, this library could even be used in projects that only use the MonoGame.Framework.Core library, not needing platform-specific versions.

[vpenades]

I would suggest to look into binomial basis...

[TechnologicalPizza]

I would suggest to look into binomial basis...

I'm not exactly sure what you mean/what I'm supposed to do... ;/

[vpenades]

I would suggest to look into binomial basis...

I'm not exactly sure what you mean/what I'm supposed to do... ;/

It's a new texture format optimized for GPU

[TechnologicalPizza]

Ok the Github repository looks promising, but this issue is mostly about improving what we got, not really adding new stuff together and a better API. :wink:

[rds1983]

As a maintainer of StbSharp I would like to share my opinion.
Generally I think it's great idea to make new pure C# imaging library, based on StbSharp code.
It might become quite popular, taking into account that ImageSharp is going to switch to nonpermissive licenses: https://www.reddit.com/r/csharp/comments/ajsbzx/imagesharp_are_switching_to_nonpermissive_licenses/

I would like to suggest following:

• Split gargantuan StbImage.Generated.cs(and correspondenly StbImageWrite.Generated.cs) into smaller files. One file for each image format (i.e. StbImage.Jpeg.cs, StbImage.Png.cs, etc).
• Copy comments from original stb_image.h to the cs. This is very boring task, however it would raise the value of the library. As I know some devs refuse to use StbImageSharp, because they afraid of how naked generated code looks. Unfortunately I dont see a way to transfer comments automatically.
• Make the code safe. That is the hardest and the longest task. It requires carefully replacing all unsafe pointer staff with safe one. However it would also significally raise the library value. As firstly it'll make its code look less scary. Secondly it'll allow the library to be used in environments where only safe code is allowed (i.e. Web).

[TechnologicalPizza]

Awesome to see some opinions from you.
And yes, the reasoning behind this "MonoGame.Imaging" library's creation was that ImageSharp was changing their licensing model (I used ImageSharp before and liked the API).

1. I've already split up StbImageWrite.Generated into 11 files and will split StbImage.Generated soon. This yields better readability and way better IntelliSense performance and feature set (some features are apparently disabled on large files).

2. Most functions will utilize a CancellationToken. I chose to add this because .NET Core doesn't and won't have Thread.Abort for the foreseeable future. A CancellationToken is also trivial to implement/use.

3. Most functions report progress. This is probably something that will rarely be used but I really like progress bars 😄.
   I've also glanced over the PNG encoder and decoder and I thought about rewriting them so they write the file progressively instead of buffering all the data in memory and then writing/reading it in one chunk. Improving this will make the library slightly more mobile-friendly (and progress reporting and cancellation will be slightly more accurate).

4. I did look at stb_image.h and judging by the amount of documentation, I may write some "smart" converter (maybe I'll try to use Sichem for the token extraction) so I don't manually have to copy-paste everything.

5. I cut out zlib (from both Image and ImageWrite) because .NET's DeflateStream uses zlib internally. Only a header and footer was needed to make it a valid deflate stream for PNG. Using DeflateStream was a ++ scenario; we get a relatively up-to-date zlib which gives better performance and compression ratios and we get smaller result binaries. The zlib implementation can even be changed with a simple delegate field.

/// <summary>
/// Delegate for a zlib deflate (RFC 1951) compression implementation.
/// </summary>
public delegate IMemoryResult DeflateCompressDelegate(
    ReadOnlySpan<byte> data, CompressionLevel level,
    CancellationToken cancellation, WriteProgressCallback onProgress);

/// <summary>
/// Custom zlib deflate (RFC 1951) compression implementation
/// that replaces the default <see cref="DeflateCompress"/>.
/// </summary>
public static DeflateCompressDelegate CustomDeflateCompress;

This API is for writing. There is also one for decompression.
It will change when I get around to rewriting PNG encoding/decoding.

1. I've wrote some safe code using Span<T> for various tasks such as IO and other API that didn't use intense pointer magic :smiley:.

public delegate int WriteCallback(in WriteContext context, ReadOnlySpan<byte> data);

public static int DefaultWrite(in WriteContext context, ReadOnlySpan<byte> data)
{
    if (data.IsEmpty)
        return 0;

    byte[] buffer = context.WriteBuffer;
    int left = data.Length;
    int offset = 0;
    while (left > 0)
    {
        int sliceLength = Math.Min(left, buffer.Length);
        for (int i = 0; i < sliceLength; i++)
            buffer[i] = data[i + offset];
        context.Output.Write(buffer, 0, sliceLength);

        left -= sliceLength;
        offset += sliceLength;
    }
    return data.Length;
}

1. The pixels are actually not supplied by pointers in my StbImage/ImageWrite forks. I've created a special delegate gimmick for that purpose instead.

public delegate void ReadBytePixelsCallback(Span<byte> destination, int dataOffset);
public delegate void ReadFloatPixelsCallback(Span<float> destination, int dataOffset);

Formats that support +16-bit quality read floats, other formats read bytes. The performance penalty is minimal and the most important part of this is the possibility of on-the-fly format conversion.
Code that MonoGame.Imaging is currently using: https://pastebin.com/J89qKV5j
Making all of this safe code is also not hard.

I hope you get to see how large project scope is. 👍

[rds1983]

Yeah, MonoGame.Imaging looks cool.
I dont know when MG will support Span.
However I think it might be good idea to release your imaging lib as standalone lib.
I've recently compared performance of StbImageSharp vs ImageSharp: https://github.com/StbSharp/StbImageSharp#reliability--performance
It turns out that StbImageSharp loads jpg like 8 times faster than ImageSharp.
Yours is even more faster than StbImageSharp as far as I understand.
So it could become quite popular and useful.

[TechnologicalPizza]

Yeah I've spent some time optimizing Stb Image, ImageWrite, and ImageResize. I've also made the StbImage API more consistent, so it should be easy to use even without the main library.
There are some improvements/additions too but I can't bother mentioning them here.
Running your test suite over my lib would also be a smart idea (I wouldn't be suprised if there are inconsistencies between my forks and the originals).

The elephant in the room is that MonoGame.Utilities is used heavily, so it would be a required dependency for the library, and I don't know how to manage that.

[rds1983]

The elephant in the room is that MonoGame.Utilities is used heavily, so it would be a required dependency for the library, and I don't know how to manage that.

Well, considering MG license, probably you could just borrow that code.
Just don't forget to credit MonoGame in the README.md, add MonoGame license in the repo and indicate in the comments that code had been borrowed.
I am pretty sure MonoGame team wouldn't mind that.

[Jjagg]

Anything goes as long as you adhere to the MG license and the respective license of any third party dependency that you indirectly depend on. So yeah, you can pull code out of MG, just make sure you include the copyright notice and a copy of the license.