Ferdi: Setup Sentry to capture errors

[kytwb]

As more and more users use Ferdi, I think it could be nice to implement Sentry to capture the errors instead of having to ask for console logs. WDYT?
Related: https://github.com/getsentry/sentry-electron

[vantezzen]

I think we'll need to fix #146 first, otherwise we'll get a lot of errors because of this hack.

[kytwb]

That's not a problem, same errors are grouped and can be muted if necessary.

[kytwb]

I just saw you created https://github.com/getferdi/debugger after I opened this ticket; is that related? What did you have in mind with this repository?

[vantezzen]

The debugger is something that I worked on the last few days - its just a better way to view the debug information that Ferdi can output.
My idea with this is that we could create a new menu item "Publish debug information" that would push the debug information to this debugger server to help us get important information without users having to publish screenshots of it to the GitHub issues.

It wasn't really related to this ticket but I thought it was fitting to publish it now that you created it.

What do you think of the idea of this debugger? I would run it on the same server as the API so we wouldn't need any additional server resources.

[Makazzz]

I saw this today https://blog.sentry.io/2019/11/06/relicensing-sentry/. I don't know if this is important.

[kytwb]

@Makazzz all good for us:

The BSL lets us hit our goals in a clean and low-impact way: it won’t change anyone’s ability to run Sentry at their company, but it will ensure that we are protected from our work being used in an anti-competitive fashion.

[bittner]

But ... to be fair, we should:

• Make sure users are informed that Ferdi is sending exception traces to Sentry, and there should be a way to _opt out_ of sending any data to Sentry (i.e. technically, opt out of activating Sentry).
• Actually, Ferdi should even ask for _opt in_ (and not activate Sentry integration at all, by default). It should be more like, "Do you want to help us make Ferdi better?"

We need to be aware of privacy issues. Sending data, even anonymous data, leaves traces.

Let's be a role model for privacy, not piracy.

[vantezzen]

+1 for making it opt-in, asking the user if he wants to contribute on first launch.
@kytwb Can you implement this?

[kytwb]

@bittner @vantezzen understood! 👍

[kytwb]

Related: https://github.com/getsentry/sentry-javascript/issues/2039#issuecomment-486674574

[kytwb]

The way we implement Sentry, I think we will have to prompt to reload Ferdi to make changes effective.

[bittner]

Isn't it possible the way you reference it above?

If for now controlling exception logging is available just as an option in the Settings and activated or deactivated only after restarting the app that's better than nothing w.r.t. privacy. Still better than silently sending data w/o user control. -- BTW, in the end, we also need to make clear that this is not _(intentional) user behavior tracking_ but just logging application error details when exceptions (i.e. "bugs") occur.

Still sounds weird to me that it should be so difficult to turn it off. If everything else fails it should be possible to keep a reference in the sentry.js and act on that to disable or cripple the Sentry instance.

References:

• Commit fc6e64329773f29fc9eee28bc3f4a1a1b01f218d
• Official sentry/electron docs

[bittner]

As suggested by David Cramer we should also make logging anonymous by default. Don't you think?

I'm not sure how this is done, though. There's no mention on it in the docs. :roll_eyes: I asked @dcramer via Twitter about it ... (:duck:_s_ _&_ :running_man:_s_)

[kytwb]

All noted, I will implement as I can and follow up in here. I will follow the Twitter thread to see what can be done at Sentry's level to make logging anonymous (data scrubbing?). Thank you @bittner by the way for your contribution to the OpenCollective 🙏

[dcramer]

You’d want to only set user context (so user = ...) when they opt in to that experience. You also may want to enable the organization setting in Sentry to scrub IP addresses. I’d suggest sending a unique identifier for all users (unique to the install?) and an email contact when they opt in.

[vantezzen]

I just published c2a8dce2 which makes Sentry reporting opt-in only.
I think we don't really get too many insights about errors through Sentry and all important bug reports still come through GitHub. Additionally we hit our rate limit very quickly each month so we don't even get insight to most errors thrown.
The commit added a new "Send telemetry data" toggle to the settings that is off by default.
We could maybe think about adding a screen after login/registration that prompts the user to choose whether (s)he wants to opt-in or not.

Additionally, the Ferdi debugger feature (see https://github.com/getferdi/debugger) is pretty much ready now. The feature allows users to choose "Publish debug information" inside Ferdi and allows us to view them on a Ferdi-like interface in the webbrowser. The server software is ready and the client side code is also pretty much ready in the publish-debug branch. If you think that this is a good idea we would have to setup the debugger server to run on debug.getferdi.com using the nginx proxy we already have in place - maybe you can help me with that.

[kytwb]

LGTM! 🙏 Well done @vantezzen! What you did with the debugger looks interesting! 👀
Opened a new issue to deploy an instance on debug.getferdi.com (see https://github.com/getferdi/debugger/issues/1).

Sharing a visual update on the current implementation before closing:

[kytwb]

@vantezzen I have noticed that we have a lot of old clients still installed with Sentry turned on by default sending a lot of errors and making us reach our quota within a couple of days within the rolling month. I reached out to support regarding this matter and said they could only recommend to look at deploying our own Sentry on-premise; WDYT about setting up our own and updating the DSN in a new release so we only receive data from opted-in/debugging people?

[vantezzen]

Hmm, I don't know if that is necessary as I think that the number of events sent to Sentry will slowly decrease as people update. We can only switch to our self-hosted Sentry on versions that already have it disabled by default so that won't really solve the problem, does it?

[kytwb]

@vantezzen the self-hosted would not be flooded/exceed quota and we could ditch the hosted one that is/does, so I think that would solve the issue.

[vantezzen]

If you think that that's needed then we can set it up.
I just thought that as our number of events will probably go into the range of our quota in the next weeks that there isn't really a reason to set up a self-hosted solution just for that time. As far as I can tell there is no real consequence of us going over the quota until then (unless there is some bill I don't know about).

[kytwb]

Once we're over the quota, we don't receive new events anymore; we exceed our quota in 48h every month, so we're missing most of the events; that's the issue I'm trying to address by ditching the old hosted DSN and use a fresh self-hosted one. Or we can also just create a new hosted one now that it's opt-in only and use that for upcoming releases. WDYT?

[dcramer]

Aside, ya'll qualify for our open source sponsorship, want me to just flip the flag? Probably more than enough quota at that point.

[kytwb]

@dcramer Awesome, let's give it a try! 😄 Thank you!

[dcramer]

:+1: you're set

[bittner]

David, you are awesome! Thank you!

Glad you take care of us, so we didn't even have to apply! :smile: