Sentry-react-native: Sourcemaps not unobfuscated after Expo over the air update

Created on 6 Oct 2020 · 24Comments · Source: getsentry/sentry-react-native

OS:

[ ] Windows
[x] MacOS
[ ] Linux

Platform:

[x] iOS
[x] Android

SDK:

[x] @sentry/react-native (>= 1.0.0)
[ ] react-native-sentry (<= 0.43.2)

SDK version: 1.8.2

react-native version: 0.0.0

Are you using Expo?

[x] Yes
[ ] No

Are you using sentry.io or on-premise?

[x] sentry.io (SaaS)
[ ] on-premise

If you are using sentry.io, please post a link to your issue so we can take a look:

https://sentry.io/organizations/charlie-cruzan/issues/1940089939/?project=1723562&query=is%3Aunresolved

Configuration:

(@sentry/react-native)

Sentry.init({
  dsn:
    "my-dsn",
  enableInExpoDevelopment: true,
  debug: true,
  dist: "1.0.0",
  release: "myRelease",
});

I have following issue:

The sourcemaps aren't showing up after pushing an OTA update to iOS and Android apps. Normally I would think the issue is with sentry-expo (and it may be), but usually I'd see a "sourcemap not found error" in that case, but instead all i see is
Screen Shot 2020-10-06 at 5 55 01 PM

Steps to reproduce:

follow this guide to use sentry-expo in the bare workflow- https://github.com/expo/sentry-expo#bare-workflow-setup
configure app.json for ota updates like this for sentry- https://github.com/expo/sentry-expo#step-3
build ios release scheme
crash app, error has stacktrace
run expo publish to update app over the air
crash app, stack trace isn't unobfuscated

Awaiting Fix

Source

cruzach

👍6 👀2

Most helpful comment

@cruzach I've made a PR fixing this upstream in @sentry/browser: https://github.com/getsentry/sentry-javascript/pull/3070 and also a fix that would be needed on sentry-expo: https://github.com/expo/sentry-expo/pull/157. I also noticed it was also replacing [native code] frames so I included that fix in too.

jennmueng on 24 Nov 2020

🎉5

All 24 comments

@cruzach Hmm, do the source maps show up under Settings -> {Project} -> Source Maps?

jennmueng on 7 Oct 2020

@jennmueng they do:
Screen Shot 2020-10-07 at 11 45 40 AM

cruzach on 7 Oct 2020

Hi @jennmueng, do you have any update about this? If you need a reproducible demo here is:

Reproducible Demo

Repo: https://github.com/RodolfoGS/sentry-test

git clone [email protected]:RodolfoGS/sentry-test.git
cd sentry-test
yarn install
Configure the following files with your Sentry values (org, project, token and dsn)

 - android/sentry.properties
 - app.json
 - ios/sentry.properties
 - App.js

expo publish (to update EXPO_UPDATE_URL property)
Build APK as release
Install in your device, run and tap on Crash Now button
The crash appears perfectly on Sentry, with their sourcemaps 👍
expo publish
Tap on Update Now button
Tap on Crash Now button
The crash appears on Sentry without their sourcemaps 👎

I'm experimenting this issue on Android and iOS

For more details you can go to the expo repository issue: https://github.com/expo/expo/issues/10571

RodolfoGS on 12 Oct 2020

Hi @jennmueng, any update about this? Thanks!

RodolfoGS on 22 Oct 2020

👍5

maybe worth mentioning- when the stacktrace is shown, I see in_app: true in the event JSON payload. when it's not, i see in_app: false. not familiar with what that attribute means, though

cruzach on 22 Oct 2020

I'm also seeing a similar issue to the above.
Using a bare expo workflow, the source maps work fine on the initial release. But when we push an OTA to our self-hosted jsbundles, and fire off the postExport upload, the sourcemaps result are the same as the above. The sourcemaps are uploaded, the names are valid, the dists/release versions align, but they just show app:///main.jsbundle in forEach etc.

zibs on 23 Oct 2020

I'm also seeing a similar issue to the above.
Using a bare expo workflow, the source maps work fine on the initial release. But when we push an OTA to our self-hosted jsbundles, and fire off the postExport upload, the sourcemaps result are the same as the above. The sourcemaps are uploaded, the names are valid, the dists/release versions align, but they just show app:///main.jsbundle in forEach etc.

I've exact same situation and I'm paying for Sentry... 🙄
This bugs must be fixed ASAP, the basic Sentry functionality it's see where the crash was and it's not working 🙄

F170x on 23 Oct 2020

@RodolfoGS @cruzach @zibs @F170x Ok, noted and I will look into this as soon as I can first thing.

jennmueng on 23 Oct 2020

👍4

@cruzach I've been trying to get the reproduction app from @RodolfoGS to work, it runs fine in dev mode but when I make a release build for either iOS or Android the app always opens to a white screen and nothing happens. I don't have experience with making standalone expo release builds—I've only used it with dev builds before—so I'm not entirely sure I'm doing it correctly.

By building the standalone release build I'm running expo build:android -t apk, downloading the apk from the link it gives then installing it by running adb install {path-to-downloaded-apk} and for iOS expo build:ios -t simulator then dragging and dropping the app after extracting the .tar onto the simulator. Is this correct?

jennmueng on 29 Oct 2020

the expo build:* commands are for managed workflow projects (basically, where you don't have access to the native code). That's working fine as far as I can tell.

This issue is for vanilla RN projects using expo-updates. So instead of building for release with expo build:android or expo build:ios, you'd actually just build through xcode and android studio like any other react native app

cruzach on 29 Oct 2020

❤1

@jennmueng just follow the step by step that I described in my comment.

In the step 2 (_Build APK as release_) you must:

npm start
Open the project (android folder) on Android Studio
Build -> Generate Signed Bundle / APK
Select APK, click Next..., create keystore, Next, select release, click signatures V1 and V2, Finish
Install the APK in your device and follow the steps that I described below

Hi @jennmueng, do you have any update about this? If you need a reproducible demo here is:

Reproducible Demo

Repo: https://github.com/RodolfoGS/sentry-test

git clone [email protected]:RodolfoGS/sentry-test.git

cd sentry-test

yarn install

Configure the following files with your Sentry values (org, project, token and dsn)
 - android/sentry.properties
 - app.json
 - ios/sentry.properties
 - App.js
expo publish (to update EXPO_UPDATE_URL property)

Build APK as release

Install in your device, run and tap on Crash Now button

The crash appears perfectly on Sentry, with their sourcemaps 👍

expo publish

Tap on Update Now button

Tap on Crash Now button

The crash appears on Sentry without their sourcemaps 👎

I'm experimenting this issue on Android and iOS

For more details you can go to the expo repository issue: expo/expo#10571

RodolfoGS on 29 Oct 2020

Ok I am now able to reproduce this. On first look, it isn't a source maps issue, nor an issue on our backend. If you look at the event json of an event sent before the OTA update, you can see that under exception the raw stacktrace, and it seems populated and correct.

However, after the OTA update with the missing stack trace, it seems to be lacking details and only contains these two items:

"exception": {
    "values": [
      {
        "stacktrace": {
          "frames": [
            {
              "function": "value",
              "abs_path": "app:///index.android.bundle",
              "module": "index.android",
              "in_app": false,
              "filename": "app:///index.android.bundle",
              "data": {
                "orig_in_app": 1
              }
            },
            {
              "function": "forEach",
              "abs_path": "app:///index.android.bundle",
              "module": "index.android",
              "in_app": false,
              "filename": "app:///index.android.bundle",
              "data": {
                "orig_in_app": 1
              }
            }
          ]
        },
        "type": "Error",
        "mechanism": {
          "type": "generic",
          "handled": true
        },
        "value": "Test Error"
      }
    ]
  }

Now, this most likely isn't an issue with Sentry at all as we just take the stack trace directly from React Native and don't remove any frames: https://github.com/getsentry/sentry-react-native/blob/5fbd5ba6265431b3fc9a6d859aa36e6f1c1c406e/src/js/sdk.ts#L57-L79

I've also checked and found that event is missing the detailed stack trace frames on the device itself and not removed by our backend by logging the event through beforeSend.

I've seen incorrect/missing stack traces being sent from React Native before through fast reloading on dev builds. Although I haven't run into this issue with OTA CodePush, so I think the most likely cause of this issue could be something related to how expo handles OTA updates with the JS code.

jennmueng on 30 Oct 2020

thanks for investigating @jennmueng ! So the frame rewriting piece of our integrations is really simple: https://github.com/expo/sentry-expo/blob/master/src/sentry.ts#L21-L29 (same with our changes to the global error handler which might be related?), and the only changes sentry-expo makes for handling OTA updates is in how it sets the release and dist (both of which seem to be right), so I'm not sure what could be the culprit here

Any suggestions on how to debug further? the only information I'm getting in the Sentry stacktraces for my errors is :

stacktrace: {
  frames: [
    {
      function: "value",
      abs_path: "app:///main.jsbundle",
      module: "main",
      in_app: false,
      filename: "app:///main.jsbundle",
      data: {
        orig_in_app: 1
      }
    },
    {
      function: "forEach",
      abs_path: "app:///main.jsbundle",
      module: "main",
      in_app: false,
      filename: "app:///main.jsbundle",
      data: {
        orig_in_app: 1
      }
    }
  ]
},

cruzach on 30 Oct 2020

@cruzach Looking at the frame rewriting code, that shouldn't be the issue, so I think the stack trace you're getting is what's coming straight from React Native. I've also made sure that the global error handler changes isn't the issue as errors passed to captureException also lack the frames.

Maybe I would look into how the OTA bundle is registered?

Here's an example of a raw stack trace sent from an OTA CodePush error in case it might help you out.

{
    "function": "callImmediates",
    "abs_path": "app:///main.jsbundle",
    "pre_context": [
        "__d(function(g,r,i,a,m,e,d){\"use strict\";(0,r(d[0]).polyfillGlobal)(\"Promise\",function(){return r(d[1])})},231,[232,139]);",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";var l=r(d[0]);function polyfillObjectProperty(o,t,p){var b=Object.getOwnPropertyDescriptor(o,t),c=b {snip}",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";(0,r(d[0]).polyfillGlobal)(\"regeneratorRuntime\",function(){return delete g.regeneratorRuntime,r(d[1 {snip}",
        "__d(function(g,r,i,a,m,e,d){var t=function(t){\"use strict\";var n,o=Object.prototype,c=o.hasOwnProperty,u=\"function\"==typeof Symbol?Symbol:{} {snip}",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";if(!g.RN$Bridgeless){var l=r(d[0]).polyfillGlobal,t=function defineLazyTimer(t){l(t,function(){retu {snip}"
    ],
    "post_context": [
        "__d(function(g,r,i,a,m,e,d){\"use strict\";var t=r(d[0]);Object.defineProperty(e,\"__esModule\",{value:!0}),e.default=void 0;var u=t(r(d[1])).ge {snip}",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";var n,o=r(d[0]);n=o.now?function performanceNow(){return o.now()}:function performanceNow(){return  {snip}",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";var n;r(d[0]).canUseDOM&amp;&amp;(n=window.performance||window.msPerformance||window.webkitPerformance),m.e {snip}",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";var n=!(\"undefined\"==typeof window||!window.document||!window.document.createElement),t={canUseDOM: {snip}",
        "__d(function(g,r,i,a,m,e,d){\"use strict\";var n=r(d[0]).polyfillGlobal;n(\"XMLHttpRequest\",function(){return r(d[1])}),n(\"FormData\",function() {snip}"
    ],
    "filename": "app:///main.jsbundle",
    "lineno": 242,
    "colno": 3760,
    "in_app": false,
    "context_line": "{snip} es:function callImmediates(){for(b=null;_callImmediatesPass(););b&amp;&amp;b.forEach(function(t){return C.setTimeout(function(){throw t},0)})},emitT {snip}"
}

jennmueng on 31 Oct 2020

👍1

Anything I can do here to help @cruzach?

I did notice when I was initially trying to reproduce this, that when I made a TestFlight release for the app for the first time after initializing the project as an expo "bare minimal typescript" app -- and before integrating sentry-expo -- , the source maps had the same result app:///main.jsbundle in forEach even on the initial release (and of course OTA still didn't work for source maps). Once I added sentry-expo, everything works on that initial release to TestFlight for Sentry, but OTA still fails (the issue we're all dealing with now).

Is there some way that sentry-expo and expo-updates are conflicting with each other somehow in the bare workflow?

zibs on 3 Nov 2020

thanks @zibs but I don't believe so. I have gotten somewhere- it seems that the problem might lie in the where the new JS bundles are being stored since if I just replace the actual minified contents of android.index.bundle with the contents of an Expo OTA bundle, and then install the APK, I see the proper stack trace.

cruzach on 4 Nov 2020

👍1

@jennmueng can you lmk how you investigated the raw stack trace sent from React Native? I checked the error values that the Sentry error handler is receiving in both scenarios, and they're the same (apart from a different filename-

// Remote OTA Bundle
"sourceURL":"/data/user/0/com.example.mobile/files/.expo-internal/bundle-CBB68D5B3785667DE830D3A9EF892C59D0CB74A5674BA0842F3ED01948FAA417"
// Embedded Bundle
"sourceURL":"/data/user/0/com.example.mobile/files/.expo-internal/bundle-1604940884682"

). I'm also not exactly sure what you mean by "how the OTA bundle is registered?", does Sentry expect the bundle to be at a particular filepath (maybe that's why only the embedded bundle is working?)

cruzach on 9 Nov 2020

Ok @cruzach I apologize but I ended up finding out that it could be something with the Sentry stack trace builder that's causing this, as at first I was logging the stack trace after it was processed by the SDK, but if you log the complete raw stack trace of a javascript Error before the event was even passed to Sentry it seems to contain the detailed info while not being present after being processed by the SDK.

The code to get this is:

const error = new Error('Test Error thrown')
console.log((JSON.stringify(error, ["message", "arguments", "type", "name", "stack"])))
Sentry.Native.captureException(error)

And below is the stack trace logged by that console.log statement

onPress@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:595:658
value@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:221:7656
value@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:221:6912
onResponderRelease@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:221:5666
p@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:385
b@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:528
T@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:582
w@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:877
ke@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:12622
forEach@[native code]
_@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:2058
/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:12969
xe@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:89237
Se@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:12420
Re@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:12809
receiveTouches@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:96:13601
value@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:28:3351
/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:28:748
value@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:28:2611
value@/data/user/0/com.sentrytest/files/.expo-internal/bundle-613EDD44F3305B9D75D4679663900F2BCDDDC326F247CA3202A3A4219FD412D3:28:720
value@[native code]

and after being processed it gets reduced to just:

"frames": [
            {
              "function": "value",
              "abs_path": "app:///index.android.bundle",
              "module": "index.android",
              "in_app": false,
              "filename": "app:///index.android.bundle",
              "data": {
                "orig_in_app": 1
              }
            },
            {
              "function": "forEach",
              "abs_path": "app:///index.android.bundle",
              "module": "index.android",
              "in_app": false,
              "filename": "app:///index.android.bundle",
              "data": {
                "orig_in_app": 1
              }
            }
          ]

I'll be raising this issue with my colleagues and investigate this further.

jennmueng on 10 Nov 2020

👍2

thanks for letting me know @jennmueng ! And feel free to lmk if there's anything I can do to help

cruzach on 10 Nov 2020

I'm also in the same situation and also paying for Sentry... We've recently ejected to a bare workflow expo app and have had trouble viewing stack traces and getting source maps so far. I'm currently testing sentry-expo 3.0.3 along with @sentry/react-native 1.7.2 and getting the same results with the likes of app:///main.jsbundle in forEach. @jennmueng please let us know if there is any breakthrough.