Skiasharp: [BUG] Crashes in sk_canvas_restore_to_count on Mac 10.15

[praeclarum]

Apps using SKGLView sporadically abort() when SkiaSharp calls sk_canvas_restore_to_count with this typical stack trace:

Thread 0 Crashed:: tid_307  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib          0x00007fff6f20e7fa __pthread_kill + 10
1   libsystem_pthread.dylib         0x00007fff6f2cbbc1 pthread_kill + 432
2   libsystem_c.dylib               0x00007fff6f195a1c abort + 120
3   com.kruegersystems.circuitmac   0x0000000108fd3518 altstack_handle_and_restore.cold.1 + 40
4   com.kruegersystems.circuitmac   0x0000000108ce490c altstack_handle_and_restore + 188
5   libSkiaSharp.dylib              0x000000010927d784 0x10924c000 + 202628
6   ???                             0x000000011062618e 0 + 4569850254
7   ???                             0x000000010f7e8b4e 0 + 4554918734     at <unknown> <0xffffffff>

      at SkiaSharp.SkiaApi:sk_canvas_restore_to_count <0x000ad>
      at SkiaSharp.SKCanvas:RestoreToCount <0x0005a>
      at SkiaSharp.SKAutoCanvasRestore:Restore <0x00052>
      at SkiaSharp.SKAutoCanvasRestore:Dispose <0x00042>
      at SkiaSharp.Views.Mac.MySKGLView:DrawRect <0x005d0>
      at <Module>:runtime_invoke_void__this___CGRect <0x000fd>
      at <unknown> <0xffffffff>
      at AppKit.NSApplication:NSApplicationMain <0x001a4>
      at AppKit.NSApplication:Main <0x00112>
      at Circuit.Mac.MainClass:Main <0x00052>
      at <Module>:runtime_invoke_void_object <0x000b0>

All crashes are here: https://gist.github.com/praeclarum/d5dbcda0782daa0ae81b494969b4d93f

Code

c# new SKGLView();

Expected Behavior

No crash

Actual Behavior

Crash after some period of time. It always crashes, but can take awhile.

Basic Information

• Version with issue: 1.68.2.preview21
• IDE: Visual Studio for Mac
• Platform Target Frameworks:
  
  
  
  • macOS: 10.15
  
  

[mattleibow]

It seems to be crashing with a bad access. Do you have any info as to what was happening before the crash?

This looks like it might be that the canvas for some reason got cleaned up. Maybe you did a refresh. Maybe you have done some navigation or hidden/shown some views. Something might be living too long, and then it tries to run code on an old canvas.

[praeclarum]

@mattleibow Unfortunately, it happens with 0 user interaction (maybe the GC is collecting something?). I can start the app, walk away, come back and it's crashed.

It's 100% repro though - it just takes 5 seconds to a few minutes depending on the phase of the moon. I just ran again to verify (rebooted machine, unplugged eGPU) and it crashed in 2 minutes with no user interaction.

I should note that I have 2 canvases on screen at a time.

[praeclarum]

@mattleibow I am getting a similar crash but with a different stack. This time it's SkiaSharp.SkiaApi:sk_canvas_set_matrix calling gr_backendrendertarget_get_gl_framebufferinfo.

=================================================================
    Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
    Native stacktrace:
=================================================================
    0x106548f98 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_dump_native_crash_info
    0x10653c7b5 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_handle_native_crash
    0x1067a9513 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : altstack_handle_and_restore.cold.1
    0x1064ba90c - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : altstack_handle_and_restore
    0x106a59830 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MonoBundle/libSkiaSharp.dylib : gr_backendrendertarget_get_gl_framebufferinfo
    0x106a282ed - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MonoBundle/libSkiaSharp.dylib : sk_canvas_set_matrix
    0x10cfc789e - Unknown
    0x10cfc5c73 - Unknown
    0x10cfc2890 - Unknown
    0x10cfc2b9e - Unknown
    0x106553807 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_jit_runtime_invoke
    0x10668f2e7 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_runtime_invoke_checked
    0x106693c2e - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_runtime_invoke
    0x10679db2b - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : _ZL31native_to_managed_trampoline_52P11objc_objectP13objc_selectorPP11_MonoMethod6CGRectj
    0x1067a00ed - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : -[MySKGLView drawRect:]
    0x7fff2aed705d - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : -[_NSOpenGLViewBackingLayer display]
    0x7fff38e3a9f1 - /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore : _ZN2CA5Layer17display_if_neededEPNS_11TransactionE
    0x7fff38e193da - /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore : _ZN2CA7Context18commit_transactionEPNS_11TransactionEd
    0x7fff38e18002 - /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore : _ZN2CA11Transaction6commitEv
    0x7fff2a979e87 - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : __62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke
    0x7fff2b09577d - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : ___NSRunLoopObserverCreateWithHandler_block_invoke
    0x7fff2d6120ee - /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation : __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__
    0x7fff2d612014 - /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation : __CFRunLoopDoObservers
    0x7fff2d61170b - /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation : __CFRunLoopRun
    0x7fff2d610bd3 - /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation : CFRunLoopRunSpecific
    0x7fff2c16765d - /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox : RunCurrentEventLoopInMode
    0x7fff2c16739d - /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox : ReceiveNextEventCommon
    0x7fff2c167127 - /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox : _BlockUntilNextEventMatchingListInModeWithFilter
    0x7fff2a7d8eb4 - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : _DPSNextEvent
    0x7fff2a7d7690 - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]
    0x7fff2a7c93ae - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : -[NSApplication run]
    0x7fff2a79b775 - /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit : NSApplicationMain
    0x10a67a0f5 - Unknown
    0x10a679f13 - Unknown
    0x106553807 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_jit_runtime_invoke
    0x10668f2e7 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_runtime_invoke_checked
    0x1066966ec - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_runtime_exec_main_checked
    0x1064b0b32 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_jit_exec
    0x1064b3c24 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : mono_main
    0x1064691b8 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : xamarin_main
    0x10646a014 - /Users/fak/Dropbox/Projects/Circuit/CircuitMac/bin/Release/iCircuit.app/Contents/MacOS/iCircuit : main
    0x7fff64c6d7fd - /usr/lib/system/libdyld.dylib : start

=================================================================
    Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x106a59830):0x106a59820  55 48 89 e5 41 57 41 56 53 50 49 89 f6 49 89 ff  UH..AWAVSPI..I..
0x106a59830  48 8b 47 40 83 78 58 00 0f 8e ad 00 00 00 49 8b  [email protected].
0x106a59840  07 4c 89 ff ff 50 68 49 8b 47 40 ff 48 58 49 8d  [email protected].
0x106a59850  7f 08 e8 39 09 02 00 49 8b 4f 40 48 8b 51 28 48  [email protected](H

=================================================================
    Managed Stacktrace:
=================================================================
      at <unknown> <0xffffffff>
      at SkiaSharp.SkiaApi:sk_canvas_set_matrix <0x000ad>
      at SkiaSharp.SKCanvas:SetMatrix <0x00052>
      at Circuit.Mac.CircuitView:OnPaintSurface <0x00422>
      at SkiaSharp.Views.Mac.MySKGLView:DrawRect <0x0053f>
      at <Module>:runtime_invoke_void__this___CGRect <0x000fd>
      at <unknown> <0xffffffff>
      at AppKit.NSApplication:NSApplicationMain <0x001a4>
      at AppKit.NSApplication:Main <0x00112>
      at Circuit.Mac.MainClass:Main <0x00052>
      at <Module>:runtime_invoke_void_object <0x000b0>
=================================================================

[mattleibow]

That is very weird... the sk_canvas_set_matrix is the C API, which will call into C++ and not ever need to call back out to the C layer. In fact the C++ API knows nothing of the C API. The gr_backendrendertarget_get_gl_framebufferinfo. Are you calling GetGlFramebufferInfo at all in your code? For those two to be together is weird enough for me to think that maybe there is something wrong with the symbols.

Do you have some render loop going? Or is this a pure, "just put it on the screen and it breaks" issue?

Could you attach a sample that you can confirm crashes, maybe the source and the even the fully compiled app. Then I'll run it and try find out exactly what is going on.

I looked at the other crashes in that gist, and there is a surprising number of gr_backendrendertarget_get_gl_framebufferinfo items on the stack. Which only comes up if YOU call GetGlFramebufferInfo, which is often not the case.

[praeclarum]

No I'm just using your view. I often have 2 to 3 of them on screen at a time though. You may notice it called "MyGLView" but that's just because I copied and pasted your code in to get a better stack trace. It crashes whether I use the nuget version or my copy pasted version.

Since the other Skia bugs, I've been careful to delete bin/obj before reporting to you.

I'm sorry, I don't have a small repro. But this is in a released app that's crashing. Are you testing on Catalina? Is the GLView stable for you?

[praeclarum]

@mattleibow I did it! I made a minimal repro! :-D

Steps:

1. Run the attached project under the debugger.
2. Create a few more windows with Cmd+N
3. Drink some coffee, read Twitter
4. Observe crash

In the following screenshot, it crashed after 1500 frames, or about 25 seconds. I put a GC.Collect thread in it just to speed things up. I'm guessing a finalizer is doing something naughty.

I should also note that I updated macOS to the latest 10.15.3 and have the same problem so I don't think it's Catalina's fault.

SkiaSharpIssue1121.zip

[mattleibow]

Yes! Thanks for this! I'll have a look and see what is breaking.

[mattleibow]

Just haven't got to this yet. Trying to figure another iOS issue. But, people have been saying that the previous 1.68.0 version is working fine. Have you tried any with that version? Is this a new issue since upgrading to the .1 or .2 versions?

[praeclarum]

@mattleibow I have tried a bunch of versions. Would you give me an exact version to test? I can’t go too far back because there are other crashing bugs in those older versions.

[mattleibow]

Ah, ok. Try 1.68.0. I know some GC fixes were made in 1.68.1 so maybe I broke something. But I also updated the build tools, so maybe that broke something.

But, I'm going to have a look now since I figured out that iOS 13.3.1 is broken when it comes to free accounts... https://github.com/mono/SkiaSharp/issues/1129

[mattleibow]

I managed to repro this in seconds! I just created like a billion windows :) Let me have a closer look now...

[praeclarum]

Hi @mattleibow, every day I'm getting crash reports from paying customers and my reviews are starting to show it. I wonder if you've made any progress?

[gregsn]

I am getting the same issue. canvas.RestoreToCount (saveCount); called in SKAutoCanvasRestore.Dispose() is making my SKLGLControl.OnPaint() crash on ... Windows.

[mattleibow]

Hi folks, sorry about the delay. I am looking into this still, and I am trying to locate exactly what is causing this. I have been collecting issues and thanks to several of you in the community, I have a few samples and more information that will help as I look into this.

Unfortunately, I did get "distracted" by some other issues in this project as well as some other projects that needed my attention. However, I am back on this! And, I will be focusing on finding a solution ASAP this whole week.

@gregsn this Windows thing is interesting and might be a better way to debug as I can work more easily there. Do you have a sample code that you can share? Also, what version of VS, Windows and NuGet are you using?

[mattleibow]

After a day of moving things around, I may have found something... Things are looking good. I have something that works, but I am suspicious that it may just be avoiding some issue. I'll try and get a preview out ASAP with this and look into the core issue.

[mattleibow]

Hey folks, I think I may have found the issue. I have a fix in #1180 (more details as to what, why, and how on that PR soon), and I have published the PR NuGet to the preview feed: https://aka.ms/skiasharp-eap/index.json

Give that a go and let me know if it fixes it for the real world app. So far, the repro by @praeclarum seems to be stable. I created 30+ views and it ran all night. Typically with about 30+ views it would crash at 500

Check out the package version: 1.68.2-pr.1180.1

[praeclarum]

Thanks! Will do.

[ugnivs]

Can this bug be possible on ios?

[mattleibow]

Yes, I think it is and I also fixed it there. Check out https://github.com/mono/SkiaSharp/issues/1144 and https://github.com/mono/SkiaSharp/issues/910

[ugnivs]

Thanks. I'll try to check it in our project

[gregsn]

ok trying these packages for now:

<PackageReference Include="SkiaSharp" Version="1.68.2-pr.1180.6" ExcludeAssets="build" />
<PackageReference Include="SkiaSharp.Svg" Version="1.68.0-preview36" ExcludeAssets="build" />
<PackageReference Include="SkiaSharp.Views" Version="1.68.2-pr.1180.6" ExcludeAssets="build" />

i'll report back

[mattleibow]

@gregsn you can try the new packages (preview 45) on NuGet.org.

[mattleibow]

Hey folks, it has been a long time coming and thanks to all the folks with your repros, samples and stack traces I think we have finally managed to fix those pesky crashes - at least the ones we know about. I have just pushed 1.68.2-preview.50 to NuGet so let me know if this fixes all the crashes you have ever had 😄