Skiasharp: GPU-accelerated WPF without WindowsFormsHost

[freezy]

After reading through many issues mentioning the drawbacks of WindowsFormsHost (no transparency, no event bubbling, no borderless windows), I've tried to implement another approach: Do the expensive computing off-screen and copy the result into a (non-GPU-accelerated) SKElement.

As @mattleibow mentioned in #717, the unit tests' WglContext should be able to deliver a GPU-accelerated off-screen context.

My main view with SKElement:

<Window x:Class="WPF.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:wpf="clr-namespace:SkiaSharp.Views.WPF;assembly=SkiaSharp.Views.WPF"
        mc:Ignorable="d"
        Title="MainWindow" Height="450" Width="800">
    <Grid HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
        <wpf:SKElement x:Name="BitmapHost" PaintSurface="OnPaintCanvas" />
    </Grid>
</Window>

Code behind:

```c#
public partial class MainWindow : Window
{
private SKSurface _surface;
private GRContext _grContext;
private SKSize _screenCanvasSize;

public MainWindow()
{
    InitializeComponent();

    var glContext = new WglContext();
    glContext.MakeCurrent();
}

private void OnPaintCanvas(object sender, SKPaintSurfaceEventArgs e)
{
    OnPaintSurface(e.Surface.Canvas, e.Info.Width, e.Info.Height);
}

private void OnPaintSurface(SKCanvas canvas, int width, int height)
{
    var canvasSize = new SKSize(width, height);

    // check if we need to recreate the off-screen surface
    if (_screenCanvasSize != canvasSize) {
        _surface?.Dispose();
        _grContext?.Dispose();
        _grContext = GRContext.Create(GRBackend.OpenGL);
        _surface = SKSurface.Create(_grContext, true, new SKImageInfo(width, height));
        _screenCanvasSize = canvasSize;
    }

    // draw onto off-screen gl context
    DrawOffscreen(_surface.Canvas, width, height);

    // draw offscreen surface onto screen
    canvas.DrawSurface(_surface, new SKPoint(0f, 0f));
}

private void DrawOffscreen(SKCanvas canvas, int width, int height)
{
    // will be more expensive in the real world
    using (var paint = new SKPaint()) {
        paint.TextSize = 64.0f;
        paint.IsAntialias = true;
        paint.Color = 0xFF4281A4;
        paint.IsStroke = false;
        canvas.DrawText("SkiaSharp", width / 2f, 64.0f, paint);
    }
}

}
`` TheWglContext` class is copied from this repo's test directory.

The problem I'm having is when running this, I'm getting first this:

Managed Debugging Assistant 'CallbackOnCollectedDelegate' : 'A callback was made on a garbage collected delegate of type 'WPF!SkiaSharp.Tests.WNDPROC::Invoke'. This may cause application crashes, corruption and data loss. When passing delegates to unmanaged code, they must be kept alive by the managed application until it is guaranteed that they will never be called.'

This is followed by a System.NullReferenceException without any stack trace. So it's somewhere in a native call, but I can't figure out where. Any hints or suggestions would be appreciated!

I've created a repo to reproduce: freezy/wpf-skia-opengl.

Related: #213, #622, #688

VS bug #776804

[freezy]

Okay, after some more debugging, I figured out what caused the NullReferenceException: The WNDCLASS instance was garbage collected when it shouldn't, moving it to a static property solved it.

However, when closing the window, I'm now getting an System.AccessViolationException, again within some native code:

System.AccessViolationException
  HResult=0x80004003
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

If anyone could shed some light on that, that would be great. I've updated the sample repo, just launch it and close the window to reproduce.

[freezy]

Turns out that Unloaded for disposal is too late, Closing seems to do the trick.

@mattleibow if you think that SkiaSharp would benefit from a WPF view implementing this approach, let me know, otherwise feel free to close!

[mattleibow]

Also related #764 and #755

With regards to drawing offscreen for WPF - this seems like a good way (you only pay for the final draw). I may hook some things up.

I am going to leave this open as a feature request so we can track/prioritize this.

[mattleibow]

From @Mikolaytis in #819:

Hi, I'm using SkiaSharp in my WPF app with WGL rendering as described here.

The FPS of this approach is low (all performance is thrown into copy bytes to WritableBitmap), so I'm in search for another solution and found an perfect example, how to render in WPF D3DImage over OpenGL via SharpDX, Angle.

https://github.com/l3m/wpf-gles

Performance is on another level, but I'm failing to connect this example to SkiaSharp.
Is this possible at all?
Did you ever considered this type of approach for WPF apps?
If this approach will connect to skia - it will be awesome!

[mattleibow]

From @Mikolaytis:

Here we go! Done: https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK

[mattleibow]

From @Mikolaytis:

Turns out text/line/etc. draw do not render anything, only images are rendering.

[mattleibow]

From @john-cullen:

@Mikolaytis did you ever get text / lines rendering for this?

[mattleibow]

@john-cullen, I can't be sure at this point, but it might be that the stencil buffers aren't set up right. I see this is 0: https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK/blob/f41433c75701519991eb861aa063653293d1781f/src/WpfGlesDemo/SkiaRenderer.cs#L78

Hopefully @Mikolaytis has got a working solution that we can benefit from. 🤞

[mattleibow]

Just merging some issues and it appears that @freezy has some code in a repo:

@ibgorton in case you're still using a CPU-based surface, I've created a proof of concept using an accelerated off-screen surface without OpenTK or ANGLE that seems to work well. More info and code here: https://github.com/freezy/wpf-skia-opengl.

[john-cullen]

@mattleibow thanks for responding immediately!

I have a couple of questions. First of all, I apologise if these are obvious, but I've never done any graphics work before now.

How does the approach in this issue / the linked repository differ from using a WriteableBitmap as done here https://github.com/8/SkiaSharp-Wpf-Example (which I got to work on dotnetcore).

Does being accelerated imply that the rendering to the bitmap buffer is calculated on the graphics card instead of the cpu?

Does this translate to noticeably better performance for Skia?

If I'm just issuing API calls to Skia, from a development perspective should the experience be transparent? ie, could I develop using the mechanism I've got to work and later on switch to an accelerated backend without any change in the Skia code? (obviously I'd expect some change in that code that wires up the bitmap to my WPF control).

Basically, I'm evaluating a method to create a high performance chart as there does not seem to be an existing general-purpose implementation that both fits our use-case well enough and has the required performance. SkiaSharp seems to be the nicest way to accomplish that on dotnetcore.

Thank you for your time.

edit: both approaches seem to have about the same performance if I call InvalidateVisual on the bitmaphost in freezy's example from CompositionTarget.Rendering event.

[Mikolaytis]

@mattleibow We are using @freezy approach for a 6 month already. Issue is - we are still using WriteableBitmap to draw canvas on a WPF window. WriteableBitmap is stored in the RAM so - on GPU rendering we are having next pipeline:
1) Rendering surface on GPU
2) Converting surface to bitmap pixels
3) Copying pixels to the Writeablebitmap handle (from GPU to the RAM)
4) WPF will push buffer from RAM to it's texture buffer on GPU
5) Image will be rendered on a WPF window

2-4 steps are terrific performance downgrade.

I want to have next pipeline:
1) Rendering surface on GPU
2) Copying it to the D3DSurface
3) Image will be rendered on a WPF window

I imagine this pipeline should work at least 10x faster than first one.

I've tried a lot of options over a 6 month and did not found a working solution.
The best solution I found I posted here: https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK
All we need is to try to fix Skia rendering proplems in it.
Changing stencilbuffers options do not help.

[Mikolaytis]

So, the fail of my code was to use Avalonia EGL libs. Today I've built angle myself and everything is working. Wow! https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK - here is an example with updated libs.
Now I will try to integrate this thing into our project.

[mattleibow]

@Mikolaytis Thanks for keeping this thread updated. I just haven't had the time to work on this. But, due to changes and things, I will be having to build the ANGLE libraries for UWP from source. And, since vcpkg will be doing the work, it seems to be super trivial to get a Win32 version out as well. As a result, I will be including the ANGLE bits in the main Views package.

With all this, I will be having a closer look at implementing true native rendering for WPF. I will look at what you have got in that repo and hopefully get something going. I do look forward to the day when we can have a DX view in WPF.

Thanks again for your work.

[Mikolaytis]

@mattleibow Maybe let me try to integrate this into SkiaSharp.Views.WPF. I can do this in 2-6 hours and make a PR. We are already using my solution for a half of a year in production.

What will be added:

1. OpenTK nuget package 3.2.0 for GLES launch
2. SharpDX.Direct3D9 nuget package 4.2.0 for DirectX texture creation
3. 4 dlls prebuilt libs of Angle (2 x64 and 2 x86) latest master - you can make an automatic build from source later.
4. Fallback logic to the CPU rendering if target PC do not have GPU.
5. (optional) Rendering in separate thread - huge perf boost and no UI thread freezes
6. (optional) (will take additional time) We can get rid of SKElement and provide the output as an ImageSource (D3DImage) that can be used as brush or source of an Image Element.

Unresolved issues that will be added too :)

1. App crash on PC Sleep and other Device Lost events. I did not found a solution yet to solve this issue. Somehow D3dImage are breaking the WPF window rendering and I did not find a way to recover it. I hope maybe someone more experienced in DirectX than me will be able to help us to resolve this someday.

[mattleibow]

Sounds like a good plan. I can get ANGLE building very quickly as all the bits are there. I just use VCPKG.

EDIT

For step 4, we will need to chat to the Avalonia team. I know they also distribute a custom build of ANGLE, so we need to make sure we work with them to override our version that we distribute. If e are using .targets files, this would be easy because we can just exclude. With the new automatic runtimes folder, I am not sure how to exclude conditionally.

@Mikolaytis, we can include ANGLE dlls in the SkiaSharp.Views.WPF package, but what happens if we manage to add ANGLE support in WinForms? Also, what happens if you are working on a server and there is no UI... I'm thinking of maybe a new package SkiaSharp.NativeAssets.ANGLE.Desktop or something that WPF can reference. If we add WinForms, then we just add a new dependency. And, if you have no UI, you can just manually pull that in. That was my first thought, let me know what you think.

@kekekeks might have some words on this.

[mattleibow]

Just referencing this issue as it is very much related: https://github.com/mono/SkiaSharp/issues/243

[Mikolaytis]

Is Avalonia using SkiaSharp.Views.WPF? I think they are using only SkiaSharp itself. So there will be no dll overwrite conflicts because I want to add ANGLE libs only into SkiaSharp.Views.WPF.
I don't think that ANGLE is needed for WinForms because in WinForms you can use openGL without any problem.
About separate package for ANGLE libs idea - I'm in, but I have 0 experience with nuget.

[mattleibow]

OK, cool. Go ahead with the WPF work, I'll see what needs to be done with the packages. I'll try a few things and see what I come up with.

But, packages are just the final step, no need to look at that right now.

[kekekeks]

I have a somewhat working setup for one of my WPF apps (closed-source, unfortunately) which doesn't involve OpenTK.

Note that
1) GPU->CPU->GPU is terribly show, FPS will be abysmal
2) For some reason D3DImage is tricky to get right, you either get lost frames or flicker, my current approach is: glFinish, lock the image, do the blit, glFinish, unlock the image.

[kekekeks]

Regarding ANGLE binaries, for now you can use ones packaged for Avalonia, they generally work well

[Mikolaytis]

@kekekeks I've solved flickers in my closed source app by using 2 surfaces. 1 surface(just a background surface) to draw everything, and in the end just DrawSurface() the surface to another surface(that is connected to the ANGLE output).

2 surfaces will use 2x of memory, but will significantly reduce the lock state time amounth of the D3DImage.

pipeline:
1) draw to the 1 surface in another thread (Render thread)
2) drawSurface to second surface (Render thread)
3) secondSurface.Flush() or GrContext.Flush() (Render thread)
4) GL.Finish or GL.FLush (can be invoked in any thread with MakeCurrent before call)
5) trylock, setdirty and unlock image in (ui thread)

[Mikolaytis]

@kekekeks did you handle DEVICE_LOST in your closed source WPF app? If so can you provide any useful article in the web that can guide me in the right direction?