Sentry-ruby: event data is lost when failing to send to sentry

[gingerlime]

I'm continuing a discussion with getsentry.com support. We bumped into this recently and frankly I'm a bit baffled by the way this is handled. Please correct me if I misunderstood how the code works, or what the implications are.

If you look at the failed_send method, you can see quite a bit being logged, but what's NOT being logged is the actual event itself. Only the message (if it exists). If the message doesn't exist on the event (happens to us from time to time), all you end up with is Failed to submit event: <no message value>. And maybe the exception for why we couldn't send the event to sentry in the first place. So lots of stuff, besides the actual event itself.

To me the event is the most important thing. That's the reason we use sentry. We don't want to lose important errors, and we want to get notified. If we can't reach sentry for some reason, we still want to know what happened.

I'm aware that we can use the transport_failure_callback and log the event. We now do. But shouldn't logging the event fully be the default behaviour instead of essentially silencing it if it fails to reach sentry? (happy to submit a PR to fix this, but want to check first)

[nateberkopec]

Events can be rather large JSON blobs (100kb->1MB). In the worst case, when many exceptions are failing at once (quota overrun plus a bad deploy, perhaps), we could be writing gigabytes to peoples logs. So, we should either have a really good reason to write such a huge amount of data to the log, or have a way to write as little data as would be useful, or leave it up to the user.

Events can fail to send for many reasons, including quotas and inbound filters, network conditions, and other reasons which have nothing to do with the event itself. So, logging the event wouldn't be helpful for debugging failed sends in most cases. The exception (raised during the send, not the original one that created the event) is actually far more useful. Transport failure callback does not yet provide the exception, which will change in 3.0.

So that leaves logging the event because you don't want to lose it. Again, we can end up with a failed send for many different reasons, many of which you probably don't care about what the event is - quotas and filters, again. IMO there is no "sensible default" here because the use cases are so varied (do you want a log? do you want an email? what if its a filtered event? do you use a log service which charges you by the GB/month?) which is why we provide the transport callback so you can decide what to do with it.

[gingerlime]

I think your perspective is (obviously) from sentry's viewpoint, but not from the end-user's perspective.

From my perspective as a user of sentry, I couldn't care less why it failed to send to sentry (well, I'm exaggerating a bit). What I care about is the failure itself. If I didn't care about the failure, I wouldn't use sentry in the first place. Secondary to that, I would be curious to know why it failed to send to sentry, but that's really not my primary concern.

I therefore disagree that there's no sensible default here. I think the sensible default is to do whatever we can to log the primary failure and make sure the error that failed to get sent to sentry is still available.

Tracking application failures is the primary objective here. Tracking sentry failures is not the primary objective of the customer.

[nateberkopec]

My perspective is from working on this client for almost five years and seeing all the different ways people use Sentry, which differ from yours. I've already said events may "fail" to send to Sentry because the user has explicitly changed a setting causing them to fail (filtering and quotas), logging 100% of the time would be a waste when we're talking about logging hundreds of KB per failure.

We already provide transport_failure_callback. I'm open to logging a little more of each event, of course, though I'm not sure what information would be most useful. Logging the stack trace is out, because that's the largest part of every event. message is the most useful attribute, but already logged. culprit and/or fingerprint might work.

[gingerlime]

I'm not trying to underestimate your experience and knowledge, nor your understanding of what people who use sentry expect (although I'm curious to learn in what ways other customer use of sentry differs from ours). I've not used sentry extensively so far, but I do have experience with logging, monitoring, error capturing etc.

If I'm not mistaken, even the raven python client logs more than its ruby counterpart in this case?

I totally get it that when you have a flood of errors, the last thing you want is filling out your disk space and making things worse (even though logging diskspace is rarely a concern I've come across personally). At the same time, this flood-protection is causing us to potentially lose very significant single-instance events. A temporary network timeout can cause sentry to stop sending events for a few minutes, and then an important event we send to sentry, expecting to get notified about, goes completely missing...

I still therefore feel that a better default would be to log more significant data about the original failure than only the message (if present). I would personally prefer something similar to the python version, or more like the way you capture the send failure itself

[nateberkopec]

Link #742

[cxspxr]

would like to see it as well