Sentry-php: Could not encode value into JSON format. Error was: \"Malformed UTF-8 characters, possibly incorrectly encoded

[4n70w4]

{
"message": "Could not encode value into JSON format. Error was: \"Malformed UTF-8 characters, possibly incorrectly encoded\".",
"exception": "Sentry\\Exception\\JsonException",
"file": "/var/www/app/vendor/sentry/sentry/src/Util/JSON.php",
"line": 30,
"trace": [
{
"file": "/var/www/app/vendor/sentry/sentry/src/Transport/HttpTransport.php",
"line": 78,
"function": "encode",
"class": "Sentry\\Util\\JSON",
"type": "::"
},
{
"file": "/var/www/app/vendor/sentry/sentry/src/Client.php",
"line": 105,
"function": "send",
"class": "Sentry\\Transport\\HttpTransport",
"type": "->"
},
{
"file": "/var/www/app/vendor/sentry/sentry/src/Client.php",
"line": 96,
"function": "captureEvent",
"class": "Sentry\\Client",
"type": "->"
},

[Jean85]

Can you provide us the payload that generated this error? Do you have any other indication to where this issue may have originated?

[4n70w4]

@Jean85 not now, but I set a trap in production to collect detailed information if such an error occurs again.

[kocoten1992]

might related this one, https://github.com/getsentry/sentry-laravel/issues/236

[Jean85]

If we have binary data here too, yeah...

[4cc355-d3n13d]

+1, very annoying

[4cc355-d3n13d]

@Jean85 catching payload for this error is not the easy thing. Possible multibyte chars (emojies and arabic chars) causing this error

[4n70w4]

Example exception message:

Command "жопж78yuihjkТ is not defined.

May be symbols filtered by github.

Example exception message as base64:

Q29tbWFuZCAi0LbQvtC/0DY3OHl1aWhqa7AiIGlzIG5vdCBkZWZpbmVkLg

Example exception message as hex:

436F6D6D616E642022D0B6D0BED0BFD0
363738797569686A6BB022206973206E
6F7420646566696E65642E

@Jean85

[ejunker]

I am using the laravel-sentry integration and I ran into this issue because I have a database table with a column of type VARBINARY and the sql.query breadcrumbs ocassionaly contained binary strings that caused the json_encode() to fail. I was able to update my queries so that instead of inserting binary data directly, I ran in through MySQL's HEX() function to convert it to a hex string which then prevented the UTF-8 error.

I think this is an example of a simple way to reproduce this error:

        addBreadcrumb(new Breadcrumb(
            Breadcrumb::LEVEL_ERROR,
            Breadcrumb::TYPE_ERROR,
            'error_reporting',
            pack('H*', md5('binary string'))
        ));

        throw new \Exception('testing sentry');

I think the only possible options are to ignore the JSON UTF-8 errors or else to convert all the strings to UTF-8.

If you want to ignore the errors, with PHP >= 7.2 you could add JSON_INVALID_UTF8_IGNORE to Sentry\Util\JSON so that it looks like:

$encodedData = json_encode($data, JSON_UNESCAPED_UNICODE|JSON_INVALID_UTF8_IGNORE);

I did not have these errors until I upgraded from version 1.9 to 2.0 of sentry/sentry-php. It looks like in the old Raven client that it had code like mb_convert_encoding($value, 'UTF-8'); Raven Serializer handles converting everything to UTF-8 and didn't have this issue with binary data. I don't think Sentry can predict what type of data will be sent to it. Maybe the new version needs to convert all strings to UTF-8 to prevent this issue.

[pdbreen]

As others have indicated, the primary issue is failure to encode json if accumulated event data contains binary (just surfaced for us). There's a secondary issue here as well -- the Json exception being thrown is escaping the Sentry API boundary (captureEvent). Failure to capture an event should not create a new unexpected (and undocumented) exception condition.

[Jean85]

If you want to ignore the errors, with PHP >= 7.2 you could add JSON_INVALID_UTF8_IGNORE to Sentry\Util\JSON ...

We could do that and leverage the Symfony PHP 7.2 polyfill. @ste93cry WDYT?

[ste93cry]

I would like to use JSON_INVALID_UTF8_SUBSTITUTE which seems to better fit this since it would replace the invalid UTF-8 character with the standard replacement char, however the polyfill cannot replace existing core functions so it won't work. We should solve the issue on our side, probably not everything is running through the serializer and converted to UTF-8

[Jean85]

Probably since binary content is a pure string in PHP, it goes untouched through the serializer.

As others have indicated, the primary issue is failure to encode json if accumulated event data contains binary (just surfaced for us). There's a secondary issue here as well -- the Json exception being thrown is escaping the Sentry API boundary (captureEvent). Failure to capture an event should not create a new unexpected (and undocumented) exception condition.

I completely agree with this

[alexchuin]

We have the same problem here since yesterday...

Uncaught Exception: Could not encode value into JSON format. Error was: "Malformed UTF-8 characters, possibly incorrectly encoded". {"exception":"[object] (Sentry\\Exception\\JsonException(code: 0): Could not encode value into JSON format. Error was: "Malformed UTF-8 characters, possibly incorrectly encoded".

It's caused by this log :
Uncaught PHP Exception Doctrine\DBAL\Exception\DriverException: "An exception occurred while executing 'INSERT INTO xxx (...) VALUES (...)' with params ["2019-07-22 17:15:15", 3, "", null, "...", "FR", "\x42\x65\x61\x75\x6d\x6f\x6e\x74\x2d\x65\x6e\x2d\x76\xe9\x72\x6f\x6e", null, 7]

[kocoten1992]

Any chance we could push this ?

I (have to) temporary remove sentry package, since it doesn't report Exception due to this error

[ferranfg]

Our temporary "fix" is ignoring the errors regarding this issue, so we have something like:

/**
 * Report or log an exception.
 *
 * @param  \Exception  $exception
 * @return void
 */
public function report(Exception $exception)
{
    if ( ! app()->isLocal() && app()->bound('sentry') && $this->shouldReport($exception))
    {
        try
        {
            app('sentry')->captureException($exception);
        }
        catch (Exception $e)
        {
            // https://github.com/getsentry/sentry-php/issues/828
        }
    }

    parent::report($exception);
}

I know how ironic it is and we are missing some errors with Sentry, but still it's the only solution we have found.

[jun3453]

I had the same problem.
When I examined it in detail, it was an error because I used MsgPack.
https://github.com/getsentry/sentry-php/issues/828#issuecomment-512802064
I was able to send it by doing the following.

Sentry\init([
        'dsn' => $dsn,
        'before_send' => function (Sentry\Event $event): ?\Sentry\Event {
            $requestData = $event->getRequest();
            $data = $requestData['data'];
            $unpacker = new \MessagePack\BufferUnpacker();
            $unpacker->reset($data);
            $data = $unpacker->unpack();
            $requestData['data'] = $data;
            $event->setRequest($requestData);
            return $event;
        },
    ]);

I think you can do the same with other binary formats.

[Jean85]

Unfortunately we cannot use JSON_INVALID_UTF8_IGNORE with json_encode since it requires PHP 7.1. Maybe we should recursively pass all strings through iconv('UTF-8', 'UTF-8//IGNORE', $s), but I'm not sure if we have a centralized place in the code to cover all cases.

[4cc355-d3n13d]

Maybe adding custom JSON encoder (possibility of giving already encoded Sentry Event) as a simple workaround (solution)? 🙏

[Jean85]

That's already possible, you can set the serializer in the ClientBuilder

[rafaeltovar]

+1

In my case is when:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 68000032 bytes) {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\FatalErrorException(code: 1): Allowed memory size of 134217728 bytes exhausted (tried to allocate 68000032 bytes) at /home/pandora/www/pandora-20191004-1319/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php:308)
[stacktrace]
#0 {main}
"}

If I leave Sentry Exception capture, then:

Could not encode value into JSON format. Error was: \"Malformed UTF-8 characters, possibly incorrectly encoded

[jamesgraham]

What can be done to resolve this? I am in favour of the 7.2 workaround, for those of us using modern PHP it at least removes an awful hack...

[rafaeltovar]

What can be done to resolve this? I am in favour of the 7.2 workaround, for those of us using modern PHP it at least removes an awful hack...

Maybe something like?

// Sentry\Util\JSON.php
...
public static function encode($data): string
{

  if(version_compare (PHP_VERSION , "7.2" , ">=")
    $encodedData = json_encode($data, JSON_UNESCAPED_UNICODE|JSON_INVALID_UTF8_IGNORE);
  else {
    // another $data parser or no
    $encodedData = json_encode($data, JSON_UNESCAPED_UNICODE);
  }

  if (JSON_ERROR_NONE !== json_last_error() || false === $encodedData) {
    throw new JsonException(sprintf('Could not encode value into JSON format. Error was: "%s".', json_last_error_msg()));
  }

  return $encodedData;
}

[voodoo-dn]

My solution while not fixed in repository:

composer.json
"autoload": { "exclude-from-classmap": ["vendor/sentry/sentry/src/Util/JSON.php"], "psr-4": { "App\\": "src/", "Sentry\\Util\\": "src/Infrastructure/Sentry/Util/" } }

Copied Sentry\Util\JSON to rc/Infrastructure/Sentry/Util/ folder, added JSON_INVALID_UTF8_IGNORE in json_encode function.

[jaylinski]

Unfortunately we cannot use JSON_INVALID_UTF8_IGNORE with json_encode since it requires PHP 7.1. Maybe we should recursively pass all strings through iconv('UTF-8', 'UTF-8//IGNORE', $s), but I'm not sure if we have a centralized place in the code to cover all cases.

The option JSON_INVALID_UTF8_IGNORE requires PHP 7.2 (if anyone wondered why it wasn't used)

https://www.php.net/manual/en/function.json-encode.php

[jamesgraham]

Unfortunately we cannot use JSON_INVALID_UTF8_IGNORE with json_encode since it requires PHP 7.1. Maybe we should recursively pass all strings through iconv('UTF-8', 'UTF-8//IGNORE', $s), but I'm not sure if we have a centralized place in the code to cover all cases.

The option JSON_INVALID_UTF8_IGNORE requires PHP 7.2 (if anyone wondered why it wasn't used)

https://www.php.net/manual/en/function.json-encode.php

That's true but PHP 7.2 has been stable for a year. For teams using this version can the SDK not provide some relief? What is the rationale behind not supporting recent versions?

[Jean85]

The issue is not in "supporting recent versions", whether to require it, cutting out users still using 7.1.

I'll try to dive a bit onto this issue now, to see if I can draft a possible fix.

[ste93cry]

I would add my two cents on the discussion: I'm more than happy to use JSON_INVALID_UTF8_IGNORE wrapped around a version_compare check, however this is not a fix for all users (what about people using older PHP versions that we still support?) so I don't think we should go this way. Instead, we should patch the serializer (and honestly I would rewrite it from scratch if I had the time to do it as its code is one of the worst I've ever seen)

[Jean85]

I've attempted a fix with #920. @4n70w4 @4cc355-d3n13d @ejunker @pdbreen @alexchuin @kocoten1992 @ferranfg @jun3453 @rafaeltovar @jamesgraham @voodoo-dn @jaylinski can you please test it out to see if that resolves your issue?

The fix is not perfect due to not wanting to break BC; in the next major version I would prefer to require at least PHP 7.2 and use JSON_INVALID_UTF8_SUBSTITUTE, and move the serializer completely in the flow, making it the last step before passing data to the transport, so we can be sure that we would catch anything.

[TheLevti]

Php version 7.1 runs out of support on the 1st of Dec 2019 anyway.

Got the same issue on my projects.

[Jean85]

The fix doesn't seem to work, due to the fact that our serializer isn't able to serialize whole objects :disappointed:

[ste93cry]

Php version 7.1 runs out of support on the 1st of Dec 2019 anyway.

Indeed, and I would really like to drop support for it but it's not feasible because of a lot of people still using it. Anyway, I opened a new PR that attempts to fix the issue for all PHP versions that we support. Since I'm touching how we serialize events the code is pretty critical and I hope to not have broke something, so any person willing to test it and report back if it works of it there are problems is more than welcome!

[KaviiChathuranga]

I am using the laravel-sentry integration and I ran into this issue because I have a database table with a column of type VARBINARY and the sql.query breadcrumbs ocassionaly contained binary strings that caused the json_encode() to fail. I was able to update my queries so that instead of inserting binary data directly, I ran in through MySQL's HEX() function to convert it to a hex string which then prevented the UTF-8 error.

I think this is an example of a simple way to reproduce this error:

        addBreadcrumb(new Breadcrumb(
            Breadcrumb::LEVEL_ERROR,
            Breadcrumb::TYPE_ERROR,
            'error_reporting',
            pack('H*', md5('binary string'))
        ));

        throw new \Exception('testing sentry');

I think the only possible options are to ignore the JSON UTF-8 errors or else to convert all the strings to UTF-8.

If you want to ignore the errors, with PHP >= 7.2 you could add JSON_INVALID_UTF8_IGNORE to Sentry\Util\JSON so that it looks like:

$encodedData = json_encode($data, JSON_UNESCAPED_UNICODE|JSON_INVALID_UTF8_IGNORE);

I did not have these errors until I upgraded from version 1.9 to 2.0 of sentry/sentry-php. It looks like in the old Raven client that it had code like mb_convert_encoding($value, 'UTF-8'); Raven Serializer handles converting everything to UTF-8 and didn't have this issue with binary data. I don't think Sentry can predict what type of data will be sent to it. Maybe the new version needs to convert all strings to UTF-8 to prevent this issue.

wow. this can fix that Error
if ($data instanceof Jsonable) {
$this - > data = $data - > toJson($this - > encodingOptions);
}
elseif($data instanceof JsonSerializable) {
$this - > data = json_encode($data - > jsonSerialize(), $this - > encodingOptions);
}
elseif($data instanceof Arrayable) {
$this - > data = json_encode($data - > toArray(), $this - > encodingOptions);
} else {
// $this->data = json_encode($data, $this->encodingOptions);
$ᴛʜɪꜱ - > ᴅᴀᴛᴀ = ᴊꜱᴏɴ_ᴇɴᴄᴏᴅᴇ($ᴅᴀᴛᴀ, ᴊꜱᴏɴ_ᴜɴᴇꜱᴄᴀᴘᴇᴅ_ᴜɴɪᴄᴏᴅᴇ | ᴊꜱᴏɴ_ɪɴᴠᴀʟɪᴅ_ᴜᴛꜰ8_ɪɢɴᴏʀᴇ);
}

if (!$this - > hasValidJson(json_last_error())) {
throw new InvalidArgumentException(json_last_error_msg());
}