Kotlin-native: Obj-C: Crash reports are missing line numbers (and frames in release builds)

[benasher44]

Hi! I'm working on verifying all of the various parts of using kotlin-native mpp to understand the steps to move forward with using this in production. One issue i'm running into on iOS is crash reports. The first issue was #2422, but I got around that by forcing the compiler to always use -g (via freeCompilerArgs), and then for release builds run dsymutil to generate the release build, followed by strip to remove the debug symbols from the original binary (once the good dSYM is made).

In debug builds of the framework, crashes look great (sample crash by doing throw RuntimeException() inside a fun called crash() to test this out):

Instances of kotlin.Error, kotlin.RuntimeException and subclasses aren't propagated from Kotlin to Objective-C/Swift.
Other exceptions can be propagated as NSError if method has or inherits @Throws annotation.
Uncaught Kotlin exception: kotlin.RuntimeException
        at 0   PGFoundation                        0x00000001107adee2 kfun:kotlin.Exception.<init>()kotlin.Exception + 50
        at 1   PGFoundation                        0x00000001107ade12 kfun:kotlin.RuntimeException.<init>()kotlin.RuntimeException + 50
        at 2   PGFoundation                        0x00000001107946c2 kfun:com.plangrid.pgfoundation.Crasher.Companion.crash() + 82
        at 3   PGFoundation                        0x0000000110794603 kfun:com.plangrid.pgfoundation.Crasher.Companion.<init>()com.plangrid.pgfoundation.Crasher.Companion + 179
        at 4   UIKitCore                           0x000000011dc40ecb -[UIApplication sendAction:to:from:forEvent:] + 83
        at 5   UIKitCore                           0x000000011d67c0bd -[UIControl sendAction:to:forEvent:] + 67
        at 6   UIKitCore                           0x000000011d67c3da -[UIControl _sendActionsForEvents:withEvent:] + 450
        at 7   UIKitCore                           0x000000011d67b31e -[UIControl touchesEnded:withEvent:] + 583
        at 8   UIKitCore                           0x000000011d814018 _UIGestureEnvironmentSortAndSendDelayedTouches + 5387
        at 9   UIKitCore                           0x000000011d80dfd1 _UIGestureEnvironmentUpdate + 1506
        at 10  UIKitCore                           0x000000011d80d9ad -[UIGestureEnvironment _deliverEvent:toGestureRecognizers:usingBlock:] + 478
        at 11  UIKitCore                           0x000000011d80d71d -[UIGestureEnvironment _updateForEvent:window:] + 200
        at 12  UIKitCore                           0x000000011dc7d78a -[UIWindow sendEvent:] + 4058
        at 13  PlanGrid                            0x0000000108856097 -[MBFingerTipWindow sendEvent:] + 1783
        at 14  UIKitCore                           0x000000011dc5b394 -[UIApplication sendEvent:] + 352
        at 15  FLEX                                0x000000010ededf80 __35+[FLEXKeyboardShortcutManager load]_block_invoke.108 + 1040
        at 16  UIKitCore                           0x000000011dd305a9 __dispatchPreprocessedEventFromEventQueue + 3054
        at 17  UIKitCore                           0x000000011dd331cb __handleEventQueueInternal + 5948
        at 18  CoreFoundation                      0x0000000115c35721 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
        at 19  CoreFoundation                      0x0000000115c34f93 __CFRunLoopDoSources0 + 243
        at 20  CoreFoundation                      0x0000000115c2f63f __CFRunLoopRun + 1263
        at 21  CoreFoundation                      0x0000000115c2ee11 CFRunLoopRunSpecific + 625
        at 22  GraphicsServices                    0x000000011a8a41dd GSEventRunModal + 62
        at 23  UIKitCore                           0x000000011dc3f81d UIApplicationMain + 140
        at 24  PlanGrid                            0x0000000109569017 main + 71
        at 25  libdyld.dylib                       0x0000000118270575 start + 1
        at 26  ???                                 0x0000000000000001 0x0 + 1

In a release build where both optimizations and debug info (so that a proper dSYM can be generated), the same crash looks like this symbolicated:

SIGABRT: 

0 libsystem_kernel.dylib  ___pthread_kill
1 libsystem_pthread.dylib _pthread_kill
2 PGFoundation            TerminateWithUnhandledException
3 PGFoundation            Kotlin_ObjCExport_trapOnUndeclaredException
4 PGFoundation            kfun:com.plangrid.pgfoundation.Crasher.Companion.<init>()com.plangrid.pgfoundation.Crasher.Companion
5 libobjc.A.dylib         -[NSObject performSelector:withObject:withObject:]
6 UIKitCore               <redacted>
7 libdyld.dylib           _start

For the release build, it looks like it's crashing in the companion object init (possible, but unlikely unless an optimization caused that to happen somehow), but I've verified that it's crashing where I expect in a debug build. Questions:

1. Could release optimizations be making the final trace less legible?
2. Would it be possible to dig into this and fix #2422 as a result to formally have support for tracking production crashes?

Fixing this would go a long way toward making us more confident in our K/N multiplatform releases.

[benasher44]

Also please let me know if there is some kind of test case you'd like me to provide or other info that you'd need to reproduce this. I'm happy to provide it!

[olonho]

There are many facets to this problem. Let me name them:

1. (very hard) Generally, in compiled languages optimized builds do not provide the same level of debugability and post-mortem analysis as debug builds (this is an area where managed runtimes, like JVM does way better). Either optimizations or debugging. Note that many C/C++ libs/apps are distributed in two flavours exactly for that purpose.
2. (medium) The trace you show is printed to stdout by part of Kotlin/Native runtime (https://github.com/JetBrains/kotlin-native/blob/cc26a4c4f3d73d0b6c9df375bb0cac2b587b14c9/runtime/src/launcher/kotlin/konan/start.kt#L25), thus when using it on the actual end user device, it will not be available to developer for analysis (as it's not included in the crashdump).
3. (hard) Exception printed here is thrown and caught by Objective-C<->Kotlin bridge, thus including it into
   Apple crashdump could be non-trivial. Although, if use third-party exception analysis system, such as Firebase Crashlytics, it could be less of an issue, as we could associate arbitrary dictionary with the report.
4. (minor) We do not produce .dSYM bundle for release binaries at the moment, although fixing shalln't be hard.

NB: that current master of Kotlin native does automatic symbolication of stacktraces in debug builds of macOS and simulator, so your experience with debug builds could be even better with Kotlin 1.3.30.

Generally, current suggested scenario is to use debugging facilities on developer's machine, proper on device/AppStore crashdump analysis (including optimized builds backtracing) is considered for inclusion in future versions.

[benasher44]

@olonho thanks for your response! I appreciate the detailed explanation and analysis of the parts of this problem.

1, 3, and 4. For release/optimitized build crash analysis, many of us rely on third-party analysis. The main thing we want to be sure of here is that dSYMs produced by kotlin-native contain enough symbol information for 3rd parties to be able to perform that analysis. Personally, the 3rd party analysis of crashes that we get is usually enough to track down the issue. We rarely analyze the report ourselves.

1. Good point! In general, I've found crash analysis to be quite straightforward in debug builds, so great job there!

proper on device/AppStore crashdump analysis (including optimized builds backtracing) is considered for inclusion in future versions.

I might be wrong, but I don't think most iOS devs will look to you all to provide tools for doing this analysis. As long as dSYMs for release builds have sufficient symbol information, there are already plenty of blog posts and Apple-provided bits of documentation on how to do the analysis. Maybe I'm reading this wrong, but I wanted to clarify that point: good dSYMs > crash analysis tools.

NB: that current master of Kotlin native does automatic symbolication of stacktraces in debug builds of macOS and simulator, so your experience with debug builds could be even better with Kotlin 1.3.30.

🎉 looking forward to it!

Thanks again for y'all's hard work here!

[benasher44]

1. (hard) Exception printed here is thrown and caught by Objective-C<->Kotlin bridge, thus including it into
   Apple crashdump could be non-trivial. Although, if use third-party exception analysis system, such as Firebase Crashlytics, it could be less of an issue, as we could associate arbitrary dictionary with the report.

If a hook could be provided to retrieve this extra debug information (i.e. "arbitrary dictionary with the report") for our crash reporting tools, that would be very welcome!

[olonho]

Unlikely that explicit hooks are needed, just use in top level code smth like

try {
  // app logic here
} catch (t: Throwable) {
  reportToMyTool(t.getStackTrace())
}

[olonho]

We see no good way to create verbose stack trace in release builds, other that decrease optimization level in compiler codegenerator.

[olonho]

See https://github.com/JetBrains/kotlin-native/pull/2604 for more tunable custom exception reporter.

[benasher44]

@olonho thanks for adding a hook! That'll be super helpful (won't have to do try catch finally for every entry point into the library).

Understand about release builds. I think we can close out this issue once we can get dSYMs for release builds. From there, we can just open specific issues if we have trouble with specific crash reports (help determine if future release analysis tools are needed).

[olonho]

Also see https://github.com/JetBrains/kotlin-native/pull/2645 for storing exception report in the crashlog.

[benasher44]

Awesome thanks!!

[benasher44]

We now have a K/N framework out in the wild on iOS. We experienced our first crash and fixed it. However, the trace information, which we reported to our logging tool from the exception hook, was unusable (just addresses; no symbols). Luckily, we were able to debug it with the information from the exception message itself.

It looks like this is only the case with optimizations enabled, so we may have to switch to shipping a debug framework, which is sub-optimal. Going back to the original list of possible options here:

1. Has there been any headway here? In Swift, we commonly will not have line numbers in some cases where code is in-lined, but we will still have some symbol information (i.e. symbol where the code came from). So, we don't expect the same full, rich debug info as we would get from a debug build, but anything over what we have today would be huge for us. We'd also be happy (with some guidance) to submit reports where frames/numbers were missing to help you all debug/improve this.

2. Just putting this number here, so that the next one can be 3 in GH's list formatting

3. For the Apple crashdump thing, I saw that it was implemented, but now that we have crashes, I can't find where these logs should be. Where exactly, should we expect to find them?

Thanks!

[SvyatoslavScherbina]

For the Apple crashdump thing, I saw that it was implemented, but now that we have crashes, I can't find where these logs should be. Where exactly, should we expect to find them?

@benasher44 Which logs?

[benasher44]

What I mean is: what does #2645 mean for crash logging? Since we started using mpp, we've always had an exception handler setup to manually report exceptions just before the app crashes. Is this still the correct the approach? Or does #2645 obviate the need for that?

[SvyatoslavScherbina]

https://github.com/JetBrains/kotlin-native/pull/2645 stores unhandled Kotlin exception stacktraces to standard Xcode crash reports.
If you use Crashlytics or other similar third-party tool, then you still need an exception handler to manually report exceptions.

[benasher44]

These are the ones reported from apps in the App Store back to you through the Xcode organizer? Or is this a debug Xcode log? I'm not sure what those kinds of crash reports are exactly 😬

[SvyatoslavScherbina]

These are the ones reported from apps in the App Store back to you through the Xcode organizer?

Yes.

[benasher44]

Got it. Thanks!

[SvyatoslavScherbina]

Is there anything left? Shouldn't the issue be closed?

[benasher44]

Closing! Will follow up with a new KT issue if something comes up.