Sentry: Discussion: reducing email notification noise in Sentry

[benvinegar]

Today the Sentry team had a long conversation in our private Slack channel about reducing email notification noise in Sentry. We're going to open this discussion up to the community here on GitHub, in case anyone wants to chime in with their own experience and what potential solutions they feel might best help their team(s).

The Problem

Sentry can email you a lot, and if you get too much email, this can cause you to subconsciously "tune out" potential problem issues. This noise can be reduced by carefully tuning your project's notification rules, but limitations remain.

Email noise can also be particularly troublesome for client-side JavaScript projects for which stack trace grouping is less reliable, and for which 3rd-party code (console, browser extensions, bookmarklets, malware) can easily trigger one-off errors.

Some Proposed Solutions

New notification rule option based on absolute counts

Today you can configure notifications to occur when an event has been seen more than {times} in {interval}. However, you cannot simply specify more than {times} (without an interval). This could be helpful for client JavaScript projects where it is not uncommon to get errors triggered by a 3rd party that only occur once or twice by the same user.

A new daily/weekly summary email

Sentry sends you a summary email every morning summarizing yesterday's (or last week's) issues. This email could hypothetically:

• highlight issues that didn't get seen (viewed in Sentry) or actioned
• highlight high-priority issues (as defined by you)
• include event counts
• include number of affected users

Most importantly, the goal of the email is to highlight things you may have missed. The thinking here is that, if there is such a summary email, you may feel less obliged to configure aggressive _individual notifications_ for issues. Basically, the goal is to push individual notifications and have them rolled up (with more context) into this daily summary.

See: #647

Release summary emails

Similar to daily/weekly summaries, but instead we summarize issues that have popped up after a release has occurred (30 minutes, 60 minutes, 2 hours ... as configured by you).

The idea here is that issues that may have been triggered by a release are likely to be of more interest and are more actionable than an issue that occurs randomly in the middle of the night. And that if this release summary exists, users will not feel it is as necessary to report as many individual notification emails.

Bring notifications into the UI

Right now email notifications are completely separate from the Sentry application. They appear only in email (or Slack, or Hipchat, etc) – and not in Sentry itself.

If we brought notifications into Sentry, for example, to a dashboard or notification view, potentially users might feel less need to email themselves for lower priority items. They can configure Sentry to report high priority notifications (e.g. issue seen 100 times in a day), while low priority notifications are still visible in this view for regular review.

Improve client JavaScript grouping/whitelisting

As mentioned earlier, email notification noise is felt more in JavaScript projects. There is feeling that if we can improve our JavaScript grouping algorithm, this can reduce the amount of email notification noise as individual events are more likely to be grouped into an existing issue rather than a new one. Most projects are configured to receive emails when a new issue is detected.

Additionally, this could be improved by rejecting issues that are thought to be caused by third-parties: users playing in their browser's developer console, browser extensions, etc. But some of these can be particularly difficult to detect, e.g. if a browser extension changes the DOM which results in an error caused by your application's code, there's not much we can do.

---

Again, we want to hear from you. Is email notification noise a problem for your organization/team? Do any of the proposed solutions feel like they'd be helpful? Do you have other ideas we should consider? Please chime in on this issue.

[joshdholtz]

I think Bring notifications into the UI would be the best universal solution. There are probably people, like myself, that have disgustingly messy inboxes or don't like cluttered inboxes. Bringing notifications into the UI would allow me to see these when I want to see them and when I'm able to work on them. I could also then opt-in to receive up to one email a day (at whatever time I want) that would tell me "You have X amount of new issues to go look at today" instead of emails for every issue.

What would be super cool, but also probably a nightmare to develop, would be integration into task systems (Pivotal, Trello, Sprintly, etc.) so that those new issues can get taken look at right from my workflow. But I know this would be a pretty big undertaking and could get really messy really quickly :wink:

That is my opinion. You know where to reach me with questions or other thoughts :innocent:

[paulbjensen]

Hi,

We are encountering a bit of email noise when the team is working on an application and they haven't adjusted the notification settings for that project. Each environment (dev,staging, production) has it's own project, and as you can imagine the one with "development" in the name gets a few emails.

There is already the ability to hush the emails for the project in "Notification Settings" link - I'm wondering if a button link saying "adjust emails for this project" that takes you to the notification settings page with the project highlighted might help.

[jessepollak]

Woo! I'm so glad this is being discussed by y'all. It's something we've talked a ton about internally at Clef, and we've considered switching away from Sentry because of it, so it's exciting to see that y'all are taking the issue head on.

I recently emailed support asking for three things:

1. A daily/weekly summary of "trending" errors
2. The ability to only be notified when an error affects a certain number of users
3. A way to hide and/or not send email notifications for events with certain tags (i.e. IE8).

1,2 are definitely covered in what y'all describe, so I just want to add that those would be awesome. I don't think 3 is really in there, but I imagine the improved client whitelisting/grouping would help here.

As you mention, this almost entirely impacts us in JS errors — we have somewhere between 10-20x the number of JS errors as application errors, so they quickly drown out the most important notifications.

Thanks for digging into this y'all!

[benvinegar]

@jessepollak – thanks for the feedback.

Re: item 3 – I think this is possible today. Here's a rule that only sends a notification email if an event's "browser" tag does not equal "IE 8.0":

[mallyvai]

Hey peeps,

Sentry is great, and I appreciate that you're soliciting feedback on this!

Ultimately what I care about is action. This generally falls into two categories:

1) Immediate (Some failure mode is blocking revenue and we need to act immediately)
2) Longer-term patterns with how "things" are going wrong (e.g, old JS triggering failures on long-tail pages that we need to prioritize)

I never want to miss or miscategorize any kinds of errors of (1), and especially not in the short-term.

Almost all cases of type (1) are those that are already marked as 'red' in the UI somehow. And for the sake of not being overwhelmed in some outsize case, but still seeing everything important, something like a summary email sent out for every, say, 5-10 minutes worth of traffic would be great and IMO strike a decent compromise.

The longer-term stuff would be suitable as weekly summary email.

[ longer pie-in-sky thoughts - One thing I have been thinking about is the viability of algorithmic feeds or dashboards for monitoring. What the features would be, and how the classification would work, or training sets determined, I'm not entirely sure. But given both the breadth (install-base) and depth (volume of errors) Sentry manages, it's certainly interesting to think about... ]

[dcramer]

I agree with @mallyvai and my biggest fear is always not seeing something critical (like a billing error, which may only happen once a week).

Here's some additional proposal. We take level and shift it into an impact/priority concept. We make debug/info logs set the priority to low (by default), and others set to medium. We then support a way to dynamically prioritize items (maybe rules, part of the SDK, based on rates/etc). Finally, we could then make it so notifications (more specifically, actual alerts that you'd get in your inbox/slack) can be triggered based on priority. "Email me when a new high priority item is created". "Email me when an item becomes high priority". "Email me when a high priority item is unresolved after 1 hour".

[bradleybernard]

I'm not sure if this is the right discussion, but I would like to propose changing the email format. I used BugSnag before moving to Sentry and the first thing I noticed was that the emails were not as well formatted. Here is an example:

BugSnag:
https://www.dropbox.com/s/bhrc1rqpsfe48tv/Screenshot%202016-02-12%2015.44.52.png?dl=0

So the first thing I notice is that it tells me the exception in the first line and tells me what file and line it occured at. In addition, it tells me the error description along with stage, severity, and a mini stack trace that at the bottom. Very simple and elegant. A big thing is that all this info fits into the preview pane in the Mail app!

Here is Sentry's: https://www.dropbox.com/s/n80hqzi0ogacwl4/Screenshot%202016-02-12%2015.45.54.png?dl=0

Instead of telling me the exception in bold at the top, I have to look at the exception box (very long stack trace). It says BadMethodCallException but I had to scan for that. When I open this email in my Mail client on Mac, the preview window does not show enough information at a glance. I don't know where this occured (URL), the severity, and the stage it occurred, in the preview pane. I'm only presented with the Application it occurred on and the stack trace that is hard to decipher exactly what is wrong.

I think you guys should adopt BugSnag's format and include this info at the top:

1. Exception name
2. Exception description
3. Location (file and line num)
4. Stage
5. Severity
6. Limited Stack Trace

I get a lot of data from Sentry at the bottom of the email like the Tags (browser, device, environment) but it is just a lot to look at. Basically everything on the website it thrown into the email. I would much rather have the important data only and then a link to view the error online. I can't tell the severity from Sentry's emails but I can with BugSnag. The information between the two services online is nearly identical but the emails are very different.

[dcramer]

@bradbernard one complexity with comparing us to other tools is we accept a large variety of data. Exceptions are absolutely the most important type of issue, but we currently support other things as well (like CSP reports). In addition to these we put a strong emphasis on freedom of data (e.g. tags, arbitrary context), which can make some things more challenging as well.

That's not to say we couldn't change emails when exception data is present, but there are considerations:

• Other services likely don't support chained exceptions -- its important to consider how those function and may change what's relevant in a message.
• Our emails are very generic, in that we allow bits of structured data to reflect in the email as entries. We could customize them based on event type (exception, csp report, generic), but its worth noting.
• We support stage (environment) as well as many other things via less structured tags, and while nice for some people, its worth considering that its mostly relevant in web products.

We have talked about greatly customizing issue details pages based on the core event type, so it might be worth exploring that in more detail as part of this.

[bradleybernard]

@dcramer I understand it's a generic email, but I would strongly suggest customizing per events and possibly per language or something similar. I'm not sure what is on your guys' roadmap but if it's not too cluttered I would definitely put it up there.

I have no problems with the core service itself, just the reporting. I also think it is very important to be able to articulate which data is most important for exceptions rather than just a dump of data in the email. Maybe let us customize our emails to create templates for certain exceptions on specific applications. I don't want to see the browser, device, logger, IP address and other extraneous information in my email notifications, I just want critical information like the list I posted above.

I understand this could be thought as a feature over a necessity, but I think it would allow a lot of flexibility on reporting, and could introduce some new logic. If this exception name is new, send me the long version (generic), however if this has been seen or already reported then send me the short and sweet version saying hey it is happening again.

[carmandrew]

Noisy alerts are definitely the biggest problem we have with Sentry (still love it though!), and within that, the feature that would help the most is better grouping. We get a lot of alerts that are identical to each other except for a uuid that's different in the path. We're using the golang client. For example, we get something like this about once a day, but each time the part covered in red has a different uuid in it, so they don't get grouped together:

It would be awesome if there was a way to create our own custom grouping rules. For example, if we could specify a regex on the title and anything matching gets grouped together. This would help tremendously, and let us easily group and mute events we know to be irrelevant. The alternate, afaik, is to silence those warnings in the code - which has a much higher overhead.

[dcramer]

@carmandrew you can use fingerprinting on the client side to improve grouping -- unfortunately languages like Go down't provide very powerful abstractions for error reporting yet.

[aaron-zipnosis]

We'd be interested in a Rule that would hush away some of the duplicates. We would like to see larger time intervals. Essentially, I want to get 1 email if event happened 1 or more times in 24 hours. This way each day we see the error. Currently to quiet it down, we're only reporting the first time it happens. It's easy to forget about them and to know if it's a 1 off or a real problem.

[seanknox]

Hi Sentry folks, I'm curious where you ultimately landed on this issue. Like others on this thread, I'm most interested in capturing errors in a tracking system (we use Pivotal Tracker) in addition to email.

[dcramer]

@seanknox we have a number of active projects, but first and foremost our goal is to make Sentry more useful. Those priorities will be around improving the notifications we generate, less so than deferring all of our features to third parties. We'll be publishing some stuff in the next few weeks that moves in this direction.

[seanknox]

first and foremost our goal is to make Sentry more useful. Those priorities will be around improving the notifications we generate, less so than deferring all of our features to third parties.

@dcramer I hear you on that. The problem, as I read it, is that notifications aren't as actionable as they could/need to be; whether notifications are sent via email, the number of times they're sent, support for notifying through other mediums (Slack, tickets, etc.) are feature implementation details.

[dcramer]

@seanknox I agree. We're focusing on breaking up notifications into two systems: alerts and workflow. The short term improvements will be greatly expanding the workflow support (and splitting them), but we'll then refocus on alerts and how they function.

[benvinegar]

Hey folks,

I want to bring your attention to some changes we've made recently that are related to this thread:

1) Inbound data filters – we now have some quick toggles for automatically throwing out low-signal errors, e.g. those from legacy browsers, known browser extensions, and search engine crawlers.

2) User frequency in rules – you can now create notification rules based on # affected users over an interval.

3) Weekly email summaries are currently in beta (you may have already gotten them), and highlight new and re-opened issues you may have missed.

As always, your feedback on these improvements (particularly as they affect notification noise) is greatly appreciated.

[elynnaie]

I echo what @joshdholtz said:

What would be super cool, but also probably a nightmare to develop, would be integration into task systems (Pivotal, Trello, Sprintly, etc.) so that those new issues can get taken look at right from my workflow.

The business folks insist that we use Pivotal Tracker for our workflow, so being able to more easily create stories (automatically would be nice) and maybe adding some default labels to new Pivotal stories would go a long way toward integrating Sentry into our workflow.

[dcramer]

@denaje we could definitely add to the pivotal integration -- we've been moving things into sentry-plugins and migrating them to a new framework. We did that with pivotal recently[1], but we didn't expand the functionality. We did that with GitHub and some others. It wouldn't be overly hard to add to Pivotal, though I'll be honest and say its not a priority for us right now.

[1] https://github.com/getsentry/sentry-plugins/blob/master/src/sentry_plugins/pivotal/plugin.py

[wuservices]

@benvinegar I just started using Sentry and was comparing to a bunch of other services. I ultimately started testing with Sentry, TrackJS and Bugsnag and felt Sentry was the best!

The biggest thing I wanted was a better way to get alerted about JavaScript errors, figure out what was important, and sort through the noise. This discussion is spot on. Grouping is definitely the most important to getting rid of noise, and coming from errorception, Sentry is great (but can still get smarter a bit on the JS side).

Is there a way to get notified the _once_ somethings happens to more than a few users? I see options where you can perform actions _once every n minutes_, and _if an event is seem more than n users in interval_, but the what if I want to get notified once about an issue ever, but only after it's seen by more than n users? Or at the very least, give me a 1 week interval.

The use case here is to get alerted about new issues, only if they're affecting at least a few people. However, I don't want to get bugged again later and our site has big days and slow days so even with existing interval options, 1 day isn't enough.

Overall, Sentry is the winner, but TrackJS had some interesting features in terms of daily email reports (especially top messages trending messages) and alerting rules. See http://docs.trackjs.com/ui/alerts.html which talks about trending errors and error rate increases. I could see that these 2 things require a bit more analysis and may not fit into existing models, but they seem valuable across all languages.

Trending Errors
Sometimes being notified every time there is a new error can be noisy. This is particularly true on high traffic sites when there are hundreds or thousands of unique error messages. To combat that, we offer trending error notifications. We will only notify you if we see an uptick in a particular error. That is, the error was previously seen, but it is now occurring more frequently.

We will only notify if the error has moved more than 10 places. We will also notify if there’s a new error that is in the top 10 most common errors on your site. In order to keep noise to a minimum, we require an error to be seen a minimum of 10 times before it’s eligible for alerting.

You can also notify only in the case of more than one user affected.

Error Rate Increase
We will send you a notification if we see your error rate for an application increase 20% above the previous high-water mark. We look back several days to ensure it’s not just a cyclical traffic increase. Please note that if your incoming error rate is high enough to cause throttling, the error rate increase alerts may also be affected.

[dcramer]

@wuservices a few things:

1. Did you actually see anyone else doing a better job at aggregation? I actually think we're about as good as it can get these days (barring various edge cases that we dont yet know about).

2. Agreed on the alerts. It's a hard problem and ends up being fairly application specific. We plan to provide some better defaults going forward. There's some complexity as well in tracking unique users over a longer period of time. @tkaemming do you know if we could reasonably increase the unique user set to 45 days (so we could offer a one month condition)?

3. We're doing some really cool stuff which should help you hone in better on new issues, which are what we think of as often the most important.

[tkaemming]

@tkaemming do you know if we could reasonably increase the unique user set to 45 days (so we could offer a one month condition)?

We already track these at daily resolution over 90 days. (They have the same retention policy as the other TSDB metrics.) It should only be a matter of adding it to the event frequency condition implementation. For some reason, I thought there may have been as reason that we don't provide the ability to select a larger interval, but I can't think of why that'd be?

[wuservices]

@dcramer thanks for checking out the feedback!

1. I'd agree, the aggregation is pretty good overall. However, it just so happens that I was debugging an issue that kept showing up with little aggregation in Sentry. While this code has a couple paths, almost all these errors in the screenshot are identical.

2. I can definitely see how alerting rules will be application specific. However extending the unique user rule as you've proposed and has now been merged will be great! Thanks for getting that in.

3. Can't wait! We're now subscribed and paying :)

[mac2000]

Hi guys, I'm new to sentry, just connect it to our site and now I'm receiving 100500 mails per day, will be nice if there will be big button to temporary turn off mail notifications untils things are configured/fixed

[dcramer]

@mac2000 sounds like you're probably not sending events that are able to group together (or you have some non-default rule which is generating way too many alerts). I dont disagree that maybe a "mute all" could be useful, but its usually better to address the root cause.

[kimmobrunfeldt]

Our idea was to setup three levels of email lists:

List | Purpose
--------- | -----------------
[email protected] | Alerts where revenue is blocked until fixed (Sentry level === fatal)
[email protected] | Alerts which should be fixed ASAP. E.g. nightly integration stopped working (Sentry level === error)
[email protected] | Alerts for new uncaught exceptions etc. (Sentry level <= warning)

These email lists would allow people to decide what level of information they want.

We would've liked to configure so that Sentry will send email based on the event's level. However this wasn't supported by Sentry Alert actions. It was only possible to send alerts to all configured email recepients. I feel like this could be a good way to reduce the noise.

[vincentwoo]

Hi @dcramer, has there been any progress on this? Email noise is my #1 sentry problem. I'm using the browser JS client with source maps and I just get absolutely blasted.

[dcramer]

We’ve done a bunch of things around this but there will be some significant things coming out around January to stop gap fix this. The larger problem is a lot harder but in general the signal to noise problem is what we’re focused on with most product efforts right now.

[suleimanmahmoud]

Did anyone get any luck setting up the alert rules to exclude a specific logger?

I was trying to test disabling emailing for javascript notifications.
Once I add the rule:
An event's tags match logger does not contain javascript
or
An event's tags match logger does not equal javascript

I am not able to receive emails from any other logger.
Throwing exceptions javascript, I can see the new issues with the different logger, but no emails arrive.
Any ideas @benvinegar?

[krishami]

We had a simple use case

1. Turn off all email notifications
2. Keep all the notifications flowing to our company Slack channel

I tried searching for this in GitHub and finally ended up in this discussion.

Couldn't figure out if it is available or not available. Some of the discussion seemed to indicate that this would be available earlier this year?

[benvinegar]

@krishami – should be possible. Add the Slack integration, then edit your Alert Rules for each Project and change the destination to Slack/#channel.

[fmiqbal]

We currently use sentry for staging and production, but the summary I want to see on my email is only for production, is it possible? filter by environment/filter by project, how about the UI we talking about here, do we have that?

[BYK]

@fikrimi23 that sounds like an interesting idea. Would you mind submitting a new issue for that as this discussion is quite dated and I'm closing it for clarity.