Webrender: Add capabilities required for SVG filters

[mstange]

https://bugzilla.mozilla.org/show_bug.cgi?id=1409486

This isn't needed in the short term but it's good to keep in mind for the longer term.

SVG filters allow a bunch of things that CSS filters don't:

• Instead of building up a single chain of filters, you build up a graph of filter nodes ("filter primitives"). Each node can have zero, one or two input nodes.
• There are primitive types which don't take an input: Flood, Turbulence, and Image.
• There are primitive types which take two inputs: Composite, ArithmeticCombine, Blend, and DisplacementMap.
• You can clip, move, and tile (repeat) things.
• In SVG filters, each primitive has a color space assigned in which it's supposed to operate. I suggest handling this mostly in the caller, by inserting appropriate color space conversion filters between nodes where the color space changes.

I sat down today and sketched out an API. The biggest problem I see here are the dynamically-sized arrays. Those are needed in three places: Two of the component transfer filters accept a color table with a dynamic length, and the convolve matrix filter accepts a convolution matrix of dynamic size. I'm not sure if they are going to be a problem for serialization, but the way we handle pushing separate iterators (e.g. for the filter list itself) makes me think that they are.

References:

• Spec
• Loosely-typed Firefox-internal FilterDescription API
• Somewhat more strictly-typed Moz2D FilterNode API

Proposed WebRender API:

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub enum FilterPrimitiveInput {
    Original,
    TransparentBlack,
    OutputOfPrimitiveIndex(usize),
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct ImagePrimitive {
    pub image: ImageKey,
    pub image_rendering: ImageRendering,
    pub rect: LayoutRect,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct TurbulencePrimitive {
    pub base_frequency: f32,
    pub num_octaves: u32,
    pub stitchable: bool,
    pub seed: u32,
    pub output_rect: LayoutRect,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct ClipPrimitive {
    pub input: FilterPrimitiveInput,
    pub clip_rect: LayoutRect,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct TilePrimitive {
    pub input: FilterPrimitiveInput,
    pub source_rect: LayoutRect,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub struct LinearTransferFunction {
    pub slope: f32,
    pub intercept: f32,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub struct GammaTransferFunction {
    pub offset: f32,
    pub amplitude: f32,
    pub exponent: f32,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub enum ComponentTransferFunction {
    Identity,
    Linear(LinearTransferFunction),
    Gamma(GammaTransferFunction),
    Discrete(Vec<f32>),
    Table(Vec<f32>),
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct ComponentTransferPrimitive {
    pub input: FilterPrimitiveInput,
    pub red_function: ComponentTransferFunction,
    pub green_function: ComponentTransferFunction,
    pub blue_function: ComponentTransferFunction,
    pub alpha_function: ComponentTransferFunction,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub enum ColorMatrixDetails {
    Brightness(f32),
    Contrast(f32),
    Grayscale(f32),
    HueRotate(f32),
    Invert(f32),
    Saturate(f32),
    Sepia(f32),
    Matrix([f32; 20]),
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct ColorMatrixPrimitive {
    pub input: FilterPrimitiveInput,
    pub details: ColorMatrixDetails,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub enum MorphologyOp {
    Erode,
    Dilate,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct MorphologyPrimitive {
    pub input: FilterPrimitiveInput,
    pub horizontal_radius: f32,
    pub vertical_radius: f32,
    pub op: MorphologyOp,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub enum FilterPrimitiveEdgeMode {
    /// Pixels outside the input are treated as transparent black.
    None,
    /// Pixels outside the given rect are treated as having the same color as
    /// the closest pixel along the edge of the rect. Conceptually, the
    /// edge pixel rows and columns are repeated outwards.
    /// Pixels outside the input but inside the given rect are treated as
    /// transparent black.
    RepeatEdgesOfRect(LayoutRect),
    /// Sampling wraps around to the other side of the given rect.
    /// Pixels outside the input but inside the given rect are treated as
    /// transparent black.
    Wrap(LayoutRect),
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct BlurFilterPrimitive {
    pub input: FilterPrimitiveInput,
    pub edge_mode: FilterPrimitiveEdgeMode,
    pub horizontal_stdev: f32,
    pub vertical_stdev: f32,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct ConvolvePrimitive {
    pub input: FilterPrimitiveInput,
    pub edge_mode: FilterPrimitiveEdgeMode,
    pub kernel_size: (u32, u32),
    pub kernel_matrix: Vec<f32>,
    pub divisor: f32,
    pub bias: f32,
    pub target: (u32, u32),
    pub kernel_unit_length: LayoutSize,
    pub preserve_alpha: bool,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub enum LightingDetails {
    /// Diffuse lighting with the given diffuse constant.
    Diffuse(f32),
    /// Specular lighting with the given specular constant and specular exponent.
    Specular(f32, f32),
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub struct PointLight {
    pub position: (LayoutPixel, LayoutPixel, LayoutPixel),
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub struct SpotLight {
    pub position: (LayoutPixel, LayoutPixel, LayoutPixel),
    pub points_at: (LayoutPixel, LayoutPixel, LayoutPixel),
    pub focus: f32,
    pub limiting_cone_angle: f32,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub struct DistantLight {
    pub azimuth: f32,
    pub elevation: f32,
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub enum Light {
    Point(PointLight),
    Spot(SpotLight),
    Distant(DistantLight),
}

#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub struct LightingPrimitive {
    pub lighting: LightingDetails,
    pub light: Light,
    pub surface_scale: f32,
    pub kernel_unit_length: LayoutSize,
    pub color: ColorF,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub enum CompositePrimitiveOp {
    Over,
    Atop,
    Out,
    In,
    Xor,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct CompositePrimitive {
    pub inputs: (FilterPrimitiveInput, FilterPrimitiveInput),
    pub op: CompositePrimitiveOp,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct ArithmeticCombinePrimitive {
    pub inputs: (FilterPrimitiveInput, FilterPrimitiveInput),
    pub k1: f32,
    pub k2: f32,
    pub k3: f32,
    pub k4: f32,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct BlendPrimitive {
    pub inputs: (FilterPrimitiveInput, FilterPrimitiveInput),
    pub mode: MixBlendMode,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub enum ColorChannel {
    Red,
    Green,
    Blue,
    Alpha,
}

#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize)]
pub struct DisplacementMapPrimitive {
    pub input: FilterPrimitiveInput,
    pub map: FilterPrimitiveInput,
    pub x_channel: ColorChannel,
    pub y_channel: ColorChannel,
    pub scale: f32,
}

/// The filter graph is stored as a Vec<FilterPrimitive>. Primitives that take
/// inputs refer to other primitives via their index in this Vec.
/// All coordinates are given in LayoutPixels, in the local space of the
/// stacking context that this filter is applied to, *inside* that stacking
/// context's transform. (In the CSS model, filters apply before transforms,
/// e.g. if you blur and skew an element, the result will have an uneven blur.)
#[derive(Clone, Copy, Debug, PartialEq, Deserialize, Serialize)]
pub enum FilterPrimitive {
    // The first three primitive types aren't really filters, they're
    // generators. They render something without needing an input.
    Flood(ColorF),
    Image(ImagePrimitive),
    Turbulence(TurbulencePrimitive),
    // The following primitive types affect the geometry but not the colors.
    Offset(LayoutPoint),
    Clip(ClipPrimitive),
    Tile(TilePrimitive),
    // These two types handle sRGB <-> linearRGB conversion which is required
    // for SVG filters.
    LinearToSRGB(FilterPrimitiveInput),
    SRGBToLinear(FilterPrimitiveInput),
    // This primitive is a wrapper that discards the RGB channels of its inputs
    // and treats them as black.
    ToAlpha(FilterPrimitiveInput)
    // The next primitive types need a single input.
    Opacity(PropertyBinding<f32>),
    ComponentTransfer(ComponentTransferPrimitive),
    ColorMatrix(ColorMatrixPrimitive),
    Morphology(MorphologyPrimitive),
    Blur(BlurFilterPrimitive),
    Convolve(ConvolvePrimitive),
    Lighting(LightingPrimitive),
    // These primitive types combine two inputs.
    Composite(CompositePrimitive),
    ArithmeticCombine(ArithmeticCombinePrimitive),
    Blend(BlendPrimitive),
    DisplacementMap(DisplacementMapPrimitive),
}

[mstange]

Also, holy cow, Rust makes writing these kinds of APIs so much easier than C++. In C++ I had to resort to generic property bags and runtime property type checks.

[glennw]

We've recently introduced the Picture type. As part of the general brush changes, I'd like to eventually see all composite operations (blends, mix-blend-mode etc) ported over to run through these Picture primitives. Once that is done, the tree formed by Pictures including PicturePrimitives should make it quite simple to extend to support the graph required for this.

[mattwoodrow]

Thanks for writing this up Markus!

Looking at the gecko code, it appears that all SVG filter primitives output a 'PrimitiveSubregion' and expect/require that we should clip to that. It looks like we also have this for CSS filters, but it's effectively a visible region computed using the current bounds of the content (which can change asynchronously), and we don't want to clip to that.

There also are a lot of different coordinate spaces involved in the gecko code, and most operations are defined used coordinates in filter space. I think we need a way to translate from input coordinates into filter space, some filters change that space (like the offset filter), and then we need to convert back into destination coordinate space at the end of the filter chain.