Sentry-cli: Can't upload Electron 4.1.4 darwin symbols using sentry-cli

[aguynamedben]

This might be more of a sentry-cli issue, but it's specifically Electron's symbol files so I figured it was more appropriate to ask here.

We upgraded from Electron 3->4 and I'm trying to upload Electron symbols to Sentry for native crash reporting. I'm using the "manual" instructions here.

$ export SENTRY_LOG_LEVEL=debug
$ ./node_modules/.bin/sentry-cli upload-dif -t dsym ~/Downloads/electron-v4.1.4-darwin-x64-dsym.zip --org=my-org --project=my-project

It seems to make progress for a while then it hangs here, for at least 15+ minutes:

It goes from "found 1", to "found 2", etc. but when it gets to "found 4" it just seems to search forever.

Any ideas?

[jan-auer]

Thanks, @aguynamedben. Moved this to the sentry-cli repository since this is in fact an issue with sentry-cli, potentially even symbolic. We've made a couple of internal changes to the DIF upload recently and this might be related.

[jan-auer]

Turns out, sentry-cli is getting very slow when extracting large files from ZIP archives. As a workaround, I would suggest to extract the ZIP archive and invoke sentry-cli upload-dif on the folder instead.

[jan-auer]

One more thing, Electron Framework.dSYM is approaching 2GB, which is Sentry's hard limit for debug files. There will be no way to upload files that are larger. We're currently considering new approaches to allow debugging with files this big, but for now there is no workaround.

At least on Linux, we support zlib compressed debug sections. This helps to reduce the file size. I'm open to inputs if compression of debug information is a thing on other platforms. Note that compressing the entire file is not an option, though.

[jan-auer]

Slow uploads should be fixed with 1.41.2 now.

[aguynamedben]

Another win for @jan-auer. Seemed to work! Thank you. 🙏

Re: the 2GB file size limit in the future... (warning: I'm not an expert at native crash debugging), here's a product idea:

• We're not custom building Electron, and I'd be surprised if a lot of Electron users are custom building. The fact that I'm required to upload these debug files for a stock version of Electron was a little unexpected.
• Could Sentry attain all the .dSYM for the published versions of Electron?
• Could Sentry then cross-reference the already-reported runtime value of Electron 4.1.4 with symbols from electron-v4.1.4-darwin-x64-dsym.zip to provide "built-in" native crash reporting for Electron without users having to upload symbols?

Maybe this is something for down the road when you have more Electron customers. I'm not sure if the 2GB limit is a file transfer issue for Sentry or if it's actually an expensive performance issue that occurs during the analysis of a crash.

Thanks again!

[jan-auer]

All the praise to @mitsuhiko, who provided the fix this time :)

The good news are: We are building infrastructure at the moment that will allow us to do exactly that (you got ahead of us here). For major common frameworks and platforms - Electron being one example - we might start to pull debug files automatically. It will still take us some time to get to Electron, however.

The 2GB file size limit is rooted in how Sentry currently stores debug files internally. Once the new infrastructure is in place, we will start offering options to ingest larger files.

Thanks for the suggestions, such feedback is always much appreciated 👍

[aguynamedben]

Rad.

[jan-auer]

For future travellers, you can activate the "Electron Symbol Server" in _Project Settings > Debug Files_ and don't have to upload Electron's native debug information anymore:

[aguynamedben]

@jan-auer Awesome! We're about to upgrade from Electron 4->5 so we'll try this instead of uploading manually.

[aguynamedben]

Ah, we already did it when the symbols for Electron went above 2GB, an everything works well. Thanks @jan-auer for this feature. 🙏

[jeremyspiegel]

@jan-auer I'm using a custom Electron build, any update on when it will be possible to upload debug files larger than 2GB? Or do you happen to have steps or an example for setting up a custom repository from my Electron build?

[vivekpd15]

For future travellers, you can activate the "Electron Symbol Server" in _Project Settings > Debug Files_ and don't have to upload Electron's native debug information anymore:

I am using the on premise version of sentry 9.1.2

This feature isn't available there. Any workarounds?

[jan-auer]

The workaround for Sentry 9.1.2 is to use the Electron Framework.sym file.
For custom Electron builds with files larger than 2GB, the only way is to push the files to your own symbol server which is supported in Sentry 10. See this page for docs.