Imageprocessor: Blend modes

[JimBobSquarePants]

It would be great to have the ability to blend colors based on the W3c specification.

http://www.w3.org/TR/compositing-1/#blending

Additional blend modes can be added as static methods on ColorSpacialTransforms in ColorSpacialTransforms.cs

Blend Modes:

• [ ] Separable blend modes
  
  
  
  • [ ] normal blend mode
  
  
  • [x] multiply blend mode
  
  
  • [ ] screen blend mode
  
  
  • [ ] overlay blend mode
  
  
  • [ ] darken blend mode (committed awaiting review)
  
  
  • [ ] lighten blend mode (committed awaiting review)
  
  
  • [ ] color-dodge blend mode
  
  
  • [ ] color-burn blend mode
  
  
  • [ ] hard-light blend mode
  
  
  • [ ] soft-light blend mode
  
  
  • [ ] difference blend mode
  
  
  • [ ] exclusion blend mode
  
  
• [ ] Non-separable blend modes
  
  
  
  • [ ] hue blend mode
  
  
  • [ ] saturation blend mode
  
  
  • [ ] color blend mode
  
  
  • [ ] luminosity blend mode
  
  

[christopherbauer]

Hey Jim, I was eyeing for something like this, but my initial thought seemed like this might be too simple. I don't have internet atm but the pseudo code I came up with was:

public Color multiply (Color destination) {
    if(destination == Black)
        return Black;
    if(destination == White)
         return this;
    return new Color(this.R * destination.R, this.G * destination.G, this.B * destination.B);
}

I'm going to need to do a bit more work to get backdrop alpha correct per:

Cr = (1 - αb) x Cs + αb x B(Cb, Cs)

Cr: the result color
B: the formula that does the blending
Cs: the source color
Cb: the backdrop color
αb: the backdrop alpha

And I am missing clamping here, but it seems to conform to the multiply section of the w3 blending spec. Could it be so simple? If so when I get home I'll start right on it with tests happily.

[JimBobSquarePants]

Hi Christopher,

The only complexity involved that I see is that the color component values in the Color class are premultiplied. I'm not sure what issues that would cause with the formulas given.

I think we should move Lerp, Multiply and any other transforms into a new ColorTransforms.cspartial struct just to make it easier to see what is going on.

Once we have factored alpha into equations we can look at utilizing the backing vectors in the calculations also.

[christopherbauer]

Yes I'm sure you've found this already, but if anyone else also takes a look a good link for some reading is https://developer.nvidia.com/content/alpha-blending-pre-or-not-pre.

I'll look more into better tests for this branch and doing the work without multiplication via some of the library functions. I'll validate against the examples from the links as well.

[JimBobSquarePants]

Yeah chum, read that, then promptly forgot about it. I'd like to double check Lerp against it, I was having issues when mixing alpha and non alpha colors.

[christopherbauer]

I've been reading up on the premultiplied thing, it seems pretty random really, with whether or not something is actually premultiplied being up to whether or not the application that created the file following the spec for the file type. Like pngs are supposed to be non-premultiplied by default. Do we have a way of detecting this or a way of guaranteeing we have the type we're expecting?

[JimBobSquarePants]

I premultiply the values when I read from pngs then translate back when I'm saving.

https://github.com/JimBobSquarePants/ImageProcessor/blob/V3/src/ImageProcessor/Formats/Png/TrueColorReader.cs#L53

Whether I should be doing this by default... Who know's. I could do with some advice from someone who knows more than I do.

[christopherbauer]

Yeah I am going to start on trying to determine how to do this "the right way". Using the example from https://developer.nvidia.com/content/alpha-blending-pre-or-not-pre I'll be trying to determine how to get the multiply blending to return a "greenish red" instead of (what I expect will be) the "drab green" in the implementation we merged today. I'll also shift Lerp to the new partial later. After that I can start attacking the other blend modes.

I've also cleaned the comments up in this issue so that if anyone else wants to grab one of the blend modes they don't have to read our back-and-forth. Hope you don't mind.

[Quintinon]

I'm looking into helping some with this as I've been wanting to get (back) into OSS. Most of these seem straight forward to implement. I'm interested in attempting to do some performance improvements later as those are always "fun". I've got some ideas on performance that might help but I need to learn the code base first, so for now I'll start work on the screen blend.

[Quintinon]

I just noticed this was assigned to someone, if I should be working on a different item let me know, I won't be able to start working on it until this weekend anyways.

[JimBobSquarePants]

Hi @Quintinon, Love that you want to get involved! @christopherbauer do you want to work together on this?

Would love to see what you could ad performance wise also!

[JimBobSquarePants]

Hey @Quintinon, @christopherbauer have you had a chance to chat about this?

[christopherbauer]

Oh gosh I'm sorry I started a new job and have had no time to even think on this (actually work I've done for ImageProcessor is really making me an all-star over there! :)). I'd love for @Quintinon to take on some of the work here if they are still interested, and if you've already started on screen mode blending give us a heads up! :+1:

[christopherbauer]

Ho @JimBobSquarePants. I've been quite lax in my open source contributions because of the new job. I want to get started on the things we discussed in issue #329, are you still okay with what we discussed back in Feb or is there something more pressing I can look into?

[JimBobSquarePants]

Hi @christopherbauer that's no problem. Work always takes first priority.

Yeah, I'm happy for you to continue with that. On that note though, I want to write it all in a way that assumes all colors are premultiplied as that's how the class is intended to be used. If a developer ignores what I will be putting in the documentation then that will be their issue 😜

[voidstar69]

Is anyone working on adding the remaining blending modes? If not, I would like to work on adding them.

[JimBobSquarePants]

@voidstar69 if @christopherbauer is happy for you to take over that then definitely go for it. All the original complexity is now gone as Color is now no longer premultiplied. It should be a case of simply plugging through them as per the described formulae.

[CVertex]

Hey dudes,

Years and years ago (when I was in uni) I wrote a multiply/add/exclude blend modes for a project. I remember being using some bitwise tricks for doing fast multiply to save on ops.

I did some googling yesterday and found better techniques employed in https://github.com/teichgraf/WriteableBitmapEx/blob/master/Source/WriteableBitmapEx/WriteableBitmapBlitExtensions.cs

Maybe worth using for inspiration? (when u start optimizing of course)

Cheers

[JimBobSquarePants]

@CVertex Man! Code like that hurts my eyes! I much prefer hiding everything behind my PixelAcessor!

Which reminds me, I need to optimise that to return a whole column or row.

Thanks for the link though, useful!

[CVertex]

What's the status of this? Blend modes are fun, I might help if you're not finished..
If you can layout a rough structure and pattern, it'll make it easier.

From what I can see, I'd make blend modes part of the Overlay op, but maybe we want it in more places?

[JimBobSquarePants]

Dunno what the progress is just now.

I think we should actually implement them as extension methods for Vector4 that way we can use them for all IPackedVector types. After we can add some syntactic sugar by wrapping the methods in the Color struct.

e.g.

Vector4 Lighten(Vector4 first Vector4 second) {
    // Do your thang...
}

Color Lighten(Color source Color background) {
    return new Color(Vector4.Lighten(source.ToVector4(), background.ToVector4());
}

[CVertex]

@JimBobSquarePants Okay cool.

Defining these ops at a pixel level will work for multiply and many of the separable operations should be fine. However, you could miss some perf opportunities by not running these on regions or whole images.

For region based operations, like saturation and blend, how do you want those to be structured?
I think you can run these in multiple passes if you want to leverage ParallelFx. Prob need to do some more reading.

Thoughts?

[JimBobSquarePants]

@CVertex To be honest, I'm not sure. I've struggled with the granularity requirements of the API at several points.

My gut feeling is to define at pixel level and then look for opportunities to enhance performance when they occur. Most operations internally are by pixel anyway and to allow the possibility of manipulation outside of the current processor architecture would always be a bonus.

[CVertex]

@JimBobSquarePants I hear ya dude..

Let's start with the pixel level ops and see how that feels for accomplishing some things..

[JimBobSquarePants]

This issue was moved to https://github.com/JimBobSquarePants/ImageSharp/issues/16