Sentry-laravel: ErrorException: Warning: PDO::prepare(): MySQL server has gone away

[sebestenyb]

There 's a few tickets in the Laravel repositories, but it looks like the reason is an upgrade of the Sentry library to 1.0

Basically we all see ErrorException Warning: PDO::prepare(): MySQL server has gone away reports in Sentry, while it supposed to be caught by Laravel, and handle it.

But since the 1.0 update of Sentry library, it somehow bubbles up :)

https://github.com/laravel/horizon/issues/583
https://github.com/laravel/framework/issues/28920

[stayallive]

Interesting. We would have to dig where the warning is emitted from since it might not be silenced and thus will be caught by Sentry which is correct.

Since 2.x of the base SDK (and 1.x of Laravel) "errors" (like warnings and notices) are captured much better which might be the reason of the uptick of the errors. We also now capture _every_ "error" by default which you can change using the error_types option.

---

To reiterate what is said in the other issues, this is a warning and Laravel recovers from this gracefully so it's mere informational that this warning is emitted, it does not crash your apps or jobs these warnings were always emitted but not always captured.

[shengslogar]

Also having this issue.

@stayallive Since this is an ErrorException, and not a warning or notice, wouldn't changing the error_types to ignore one exception ignore every exception? Obviously could use excluded_exceptions, but I don't like the thought of this coming back to bite me later.

[stayallive]

@shengslogar, we convert it to an ErrorException (https://github.com/getsentry/sentry-php/blob/master/src/ErrorHandler.php#L346) because Sentry has no notion of a “error” and it has no real representation, so we use the (Silenced)ErrorException to encapsulate PHP errors. So changing the error_types will 100% work in this specific case 😃

But yes, this will silence _all_ errors of the type “warning”.

[Mult1Hunter]

I ended up adding this method to Exceptions/Handler.php:
PHP public function shouldReport(Exception $e) { // check if sentry is trying to falsely report 'Warning: Error while sending STMT_PREPARE packet. PID=XX' if (strpos($e->getMessage(), 'STMT_PREPARE packet') !== 'false') { return parent::shouldReport($e); } }

[shengslogar]

Without going too far down the rabbit hole, I have an idea of where/why this is happening, but not necessarily the best way to solve this.

Whenever a query fails because of a lost connection (determined by the DetectsLostConnections trait), Laravel catches the exception and retries it -- as others have stated. However, it always logs the query in the form of a QueryExecuted event. See here and here.

Sentry appropriately picks up on this (see here, ref-ing 1.1.0 since this is the version I'm on) as it always has, but I think the reason this is only being reported now is the addition of this block in recent versions (specifically the flushEvents method):

https://github.com/getsentry/sentry-laravel/blob/1e5f644b62ee73424ad8316e37b9a2dcf7290de8/src/Sentry/Laravel/EventHandler.php#L113-L119

This change was part of a1b66de5a629fcfd089d789de75244910912b6d7, explicitly added to improve queue reporting.

So technically Sentry is now reporting errors it always should've, but because of the way Laravel is logging failed queries, I don't see any good way to distinguish between an actually failed query and a failed-but-successfully-retried query without making a change to Laravel's logging (e.g. moving the logQuery method into the try{} block or something similar).

String matching through Sentry is a good "fix" for now, but this will obviously cause actual failed queries to be ignored.

I don't know enough about the internals of either codebases to recommend a good, accepted solution, but hopefully this will give a jumpstart to any efforts to resolve this.

[LasseRafn]

Wish I had any insight into how to solve this.

Would it be worthwhile looking into how competitors (Bugsnag for example) handles this? I've never seen this before going to Sentry

[Mult1Hunter]

To be frank, above code didn't solve this warning. It did skip it however when I manually threw this warning. I kind of gave up on this all tho this is very annoying. I have service that makes 300k db entries every day and I get at least 50-100 warnings every day. I did what I could optimising mariadb, but couldn't get rid of it.

[shengslogar]

@Mult1Hunter I had the same issue; my guess is the App\Exceptions\Handler is not run on the queue. Here's some experimental code I put in my config/sentry.php file successfully (see docs on before_send):

<?php

use Illuminate\Support\Str;
use Sentry\Event;

return [
    'before_send' => function (Event $event): ?Event {
        $exceptions = $event->getExceptions();

        // don't mess with large stacks
        if (\count($exceptions) === 1) {
            $exceptionMessage = $exceptions[0]['value'] ?? null;
            if (Str::contains($exceptionMessage, [
                'Error while sending STMT_PREPARE packet',
                'server has gone away'
            ])) {
                // skips exception
                return null;
            }
        }

        return $event;
    }
];

Obligatory disclaimer to any future copy-pasters that this will suppress even legitimate database issues, so keep whatever other notifications you have available to you on, or just deal with getting false positives.

CC @stayallive since you're the only Sentry member so far. Is this being actively looked into?

[Mult1Hunter]

@shengslogar Uff, nice implementation! Can you elaborate why this would silence legitimate database issues? To me it looks like this would only silence exceptions that contain 'Error while sending STMT_PREPARE packet', 'server has gone away'. Thanks for sharing, will give this a try!

[stayallive]

As far as we can tell short of error supressing database calls (putting the @ character in front of these calls: https://d.bouma.dev/3XfflIwWpJn0) the warning cannot be prevented even if the Exception that is also generated is caught by a framework like Laravel (which handles this warning gracefully and retries).

As far as the context below people seem to agree (although older answer) the only way to prevent this warning is to not capture warnings (which Sentry does do).

Context: https://stackoverflow.com/questions/34098828/php-pdo-exception-warning-on-mysql-has-gone-away?lq=1

I am not sure this is anything we can do apart from hardcode the warning supressing it from the SDK but that seems like a horrible band-aid fix that possibly will supress those warnings where it does matter.

Open to ideas!

[LasseRafn]

While the above before_send solution might work, it will prevent config caching.

I suggest creating a service provider, and putting it in the boot method as follows:

\Sentry::getClient()->getOptions()->setBeforeSendCallback( function ( Event $event ): ?Event {
        $exceptions = $event->getExceptions();

        // don't mess with large stacks
        if (\count($exceptions) === 1) {
            $exceptionMessage = $exceptions[0]['value'] ?? null;
            if (Str::contains($exceptionMessage, [
                'Error while sending STMT_PREPARE packet',
                'server has gone away'
            ])) {
                // skips exception
                return null;
            }
        }

        return $event;
});

[shengslogar]

@Mult1Hunter The premise is that the same errors get thrown when there is a lost connection AND an actual database issue. Laravel is wired to retry a request once if it thinks it could be a lost connection, and then fail if it doesn't work that time. Since we're writing a blanket statement to suppress specific errors, we have no way of determining if it's because there was a lost connection (and successfully reconnected) or if there's actually a legitimate issue with your database. This is the same issue Sentry has internally which is creating this whole problem to begin with.

This is why if you don't have separate database monitoring alerts available to you and you're concerned about your database actually failing, I would recommend against implementing this solution and just deal with the false alerts.

@stayallive Thanks for your reply! So it sounds like at this point we should close out this issue and move towards making a PR to Laravel itself to not log "false" errors in this context.

@LasseRafn Nice! Right on. Totally forgot about the serializing closures thing.

[stayallive]

@shengslogar I don't think there is anything Laravel or Sentry can or should do.

I have discussed this with a few people and we think this is more a "bug" in PDO.

I'll tell you why we think that.

When this error occurs "MySQL server has gone away" and others like it PDO throws an exception and generates an error with the E_WARNING level. There is _no_ way in PHP to prevent a warning short of prefixing a function call with @ which is generally accepted to be a bad idea™. In my opinion if PDO throws the Exception the warning should not be generated. Laravel captures the Exception thrown and handles it gracefully but Sentry captures the error.

I want to emphasize that although Sentry says ErrorException it actually is an error (there is a difference). That is because in the SDK we convert the PHP error to an exception so we can log it to Sentry (hence the ErrorException, see: https://github.com/getsentry/sentry-php/blob/master/src/ErrorHandler.php#L346).

---

As for the "fix" of not sending it to Sentry, you can also use a class like this:

<?php

namespace App\Exceptions;

use Sentry\Event;
use Sentry\Severity;

class Sentry
{
    public static function before(Event $event): ?Event
    {
        if ($event->getLevel()->isEqualTo(Severity::warning())
            && !empty($exception = array_first($event->getExceptions()))
            && Str::contains($exception['value'] ?? '', [
                'Error while sending STMT_PREPARE packet',
                'server has gone away'
            ])) {
            return null;
        }

        return $event;
    }
}

And you can use this in your sentry.php config file like:

<?php

return [
    'before_send' => [App\Exceptions\Sentry::class, 'before'],
];

[Jean85]

It is possible that the PHP bug is this one: https://bugs.php.net/bug.php?id=63812

[shengslogar]

@stayalive Thanks for that write-up! This makes sense. Whew, PHP does have its nuances.

Now just for the sake of understanding, if Laravel internally raised all warnings to ErrorExceptions (as Sentry does) and then caught all Exceptions in the query block, that would work, right?

[stayallive]

@shengslogar no what you add to the report method as per the installation instructions has nothing to do with the way errors are handled, only how Exceptions are handled.

We listen to errors separately from Laravel and do this automatically (and you can configure which errors we capture with the error_types option, but defaults to E_ALL which means all errors, even notices). That MySQL (warning) error we are discussing here also does not flow through the report function of your apps ExceptionHandler since it's an error not an Exception.

So depending on how your PHP error reporting is configured those warnings will also be logged in your PHP error log on your servers.

---

Also remember Laravel does capture all query exceptions and that is how it handles and retries the "MySQL has gone away" exception by automatically reconnecting.

The "problem" is that next to the Exception (which can be caught) PDO also generates an error (which cannot be caught only silenced).

[shengslogar]

@stayallive Okay, I think I understand now.

So regarding the error_types option:

• captureException method doesn't apply the error_types filter -- only the Sentry native error handler does
• Laravel is using a similar error handler and is piping those all through the service provider which (assuming setup per Sentry docs) should be calling captureException (which would not include the PDO warning it catches)

...would I then be correct to assume most of the errors Sentry is natively receiving are redundant? And as a result (and related to your original point), I would be safe with even the most aggressive error_types filter as long as I was only concerned about Laravel-reported issues?

My last firing brain cell appreciates all your help.

[stayallive]

This issue was closed however I never mentioned we think it's fixed in 1.4.0, please do open it back up or open a new issue if the problem is not solved.

[shengslogar]

@stayallive Thanks for your work on this!!

[driesvints]

People are reporting that this is still a problem in the 1.5.0 version of sentry: https://github.com/laravel/framework/issues/28228#issuecomment-573294825

[stayallive]

I'll re-open this then and try to find a reproducible cause that might still be causing this.

@Rishats are you able to tell us a bit more about how you setup the Sentry package, are you using log channels and/or the snippet added to the App/Exceptions/Handler.php?

[pithyone]

I had the same issue with latest version

are you able to tell us a bit more about how you setup the Sentry package, are you using log channels and/or the snippet added to the App/Exceptions/Handler.php?

I use log channels alone

composer.lock

"name": "laravel/framework",
"version": "v6.10.1",
...
"name": "sentry/sentry-laravel",
"version": "1.5.0",

config/logging.php

'channels' => [
    'stack' => [
        'driver' => 'stack',
        'channels' => ['daily', 'sentry'],
        'ignore_exceptions' => false,
    ],

    'daily' => [
        'driver' => 'daily',
        'path' => storage_path('logs/laravel.log'),
        'level' => 'debug',
        'days' => 14,
    ],

    'sentry' => [
        'driver' => 'sentry',
    ],
],

[andypa]

@stayallive I upgraded to sentry-laravel 1.5.0 and the error still comes up (For some more information: https://github.com/laravel/framework/issues/28228#issuecomment-574068251).

I'm using the exact same snippet as described here: https://docs.sentry.io/platforms/php/laravel/

Running Laravel 5.8.36.

[stayallive]

Thanks for the info, maybe something with the log channel integration causing issues still. I know we fixed it when not using log channels. We're looking!

[stayallive]

@pithyone & @andypa is your sentry/sentry version higher or equal to 2.3.0?

You can check this by running:

composer info | grep 'sentry'

I think we might have found the bug (#327) that re-introduced this problem since sentry/sentry:^2.3 and should be resolved when #327 is released.

[andypa]

@stayallive yes, here's my output:

sentry/sdk                                  2.1.0              This is a metapackage shipping sentry/sentry with a recommended http client.
sentry/sentry                               2.3.1              A PHP SDK for Sentry (http://sentry.io)
sentry/sentry-laravel                       1.6.1              Laravel SDK for Sentry (https://sentry.io)

Looking forward to the fix 🤞

[stayallive]

Ahhh gotcha, this makes sense. Then I think the culprit is found. Will be fixed soon-ish (in the next few days).

[pithyone]

@stayallive here's my output:

sentry/sdk                              2.1.0    This is a metapackage sh...
sentry/sentry                           2.3.0    A PHP SDK for Sentry (ht...
sentry/sentry-laravel                   1.5.0    Laravel SDK for Sentry (...

[stayallive]

sentry/sentry-laravel 1.6.2 was just released which fixes the regression causing duplicate exceptions, thanks all for the information and patience!

[andypa]

@stayallive I just deployed the newest version and will report back if the problem still exists. Thanks for your work 👍

[tmishutin]

@stayallive Seems like problem is still there... Laravel 5.8.38

`
composer info | grep 'sentry'
sentry/sdk 2.1.0 This is a metapackage shipping sentry/sentry with a recommended http client.

sentry/sentry 2.3.2 A PHP SDK for Sentry (http://sentry.io)

sentry/sentry-laravel 1.7.1 Laravel SDK for Sentry (https://sentry.io)
`

A lot of SQLSTATE[HY000]: General error: 2006 MySQL server has gone away errors coming in Sentry, while not in Laravel log file. All this errors are from queue jobs.

[stayallive]

@tmishutin how have you registered Sentry. Using log channels or the changes in the exception handler or both?

[tmishutin]

@stayallive only exception handler, as described here https://docs.sentry.io/platforms/php/laravel/#laravel-5x--6x