Sentry-php: v3 Performance issues

[codepuncher]

Since upgrading to v3 of the sentry package, our sites have become almost unusable.
We are having to remove Sentry from our sites to prevent 504s.

What change in v3 has caused this huge impact and what solutions are there to resolve this?

Here's the code that we are running on our WordPress sites:

// Sentry.  
if ('development' !== WP_ENV && ! empty(env('SENTRY_DSN'))) {   
    $sentryInitArgs = [ 
        'dsn' => env('SENTRY_DSN'), 
        'environment' => WP_ENV,    
        'error_types' => E_ALL, 
    ];  

    if (! empty(env('GIT_SHA'))) {  
        $sentryInitArgs['release'] = env('GIT_SHA');    
    }   

    \Sentry\init($sentryInitArgs);  

    if (! empty(env('RELEASE_VERSION'))) {  
        \Sentry\configureScope(function (\Sentry\State\Scope $scope): void {    
            $scope->setTag('trellis_release_version', env('RELEASE_VERSION'));  
        }); 
    }   
}

Maybe is it the case that the implementation needs to change for v3 usage?

Any information is appreciated.

Thanks

CC @tangrufus @bufferoverno

[stayallive]

Did the amount of errors to Sentry also increase, this would be my very first thing to suspect.

You have set the error_types to E_ALL (you might already be doing that in v2), WordPress is known to be quite noisy in the notice department.

I would suspect this will cause the slowdowns of Sentry trying to push vastly more events to sentry.io.

If the amount of errors did not increase I am very curious if you are sending many events per request since some things were changed and async (that was never async) was removed in v3.

You don't have to remove Sentry, you can use v2 which worked great for you until we figure it out what the problem is but this might require some deep diving on your side since I suspect the culprit to be in the events or amount of events being pushed which might have changed from v2 -> v3.

(I assumed you upgraded from v2 -> v3, if you were running another version before do let me know!)

[TangRufus]

I assumed you upgraded from v2 -> v3

Correct.

We are thinking of a way to:

1. store the sentry events in the database (instead of sending them to sentry right away)
2. using a async job queue, get the sentry events from the database
3. send the sentry events to sentry

Can you help with with steps 1 and 3, i.e:

• how to stop sending events to sentry
• how to "capture" the sentry events (so that i can serialize and save them in database)
• how to send events to sentry

[stayallive]

From your comment I assume there is a sudden increase in events or this was always a problem and the removal of (not really) async stuff made this apparent in v3.

---

I want to suggest what I think would be the "best" option that might be a possibly solution for you:

It is under the performance tracing documentation but would work just as well for this "problem":

https://docs.sentry.io/platforms/php/performance/#improve-response-time

This explains how to use Relay as a local event accepter and proxy to sentry.io (or your own instance) to not have to wait on an external network request which could solve the issue and saves you from building your own queuing system.

---

If you really wanted to go for the "I'll send them later and now have em in the database" solution we don't have a ready made solution and I'll have to dig into how to actually send them but serialisation can happen in the before_send hook (retuning null from that would stop Sentry from sending the event and) it gives you the already processed event you could (with the PayloadSerializer) serialize to JSON and store in the database to send later.

[ste93cry]

serialisation can happen in the before_send hook

The best and more correct way in this case would be to implement a custom transport which saves to the database rather than sending the event via HTTP. The downside is that you cannot use init() anymore to initialize the SDK because you have to set the transport instance on the client, so it's a bit more technical and low-level solution

I assume there is a sudden increase in events or this was always a problem and the removal of (not really) async stuff made this apparent in v3

Since I find unlikely that there has been an increase in events just because Sentry got updated to the latest version (if you configured explicitly error_types then nothing would have changed, and in 3.x we now default that option to error_reporting() instead of E_ALL which would in the majority of cases report less errors than before, I point the finger against the dropping of the old behaviour of the HttpTransport transport which was sending events in batch and in parallel (but the sending operation was still blocking execution of the script, that's why we say it was not async)

[TangRufus]

Thank you both for the detailed answer!

I point the finger against the dropping of the old behaviour of the HttpTransport transport which was sending events in batch and in parallel

Before diving into the custom transport way, is there an option/setting/way to restore "sending events in batch and in parallel" in v3?

[ste93cry]

is there an option/setting/way to restore "sending events in batch and in parallel" in v3?

No, although since it's an handy feature we may want to restore this feature by creating another transport that users can choose to use