Sentry-ruby: Encoding::UndefinedConversionError while tracking something that contains Unicode characters

[avel]

While tracking, either with capture_exception() or capture_message(), works fine generally, there were times where it would fail to report an error.

Here are the errors and the relevant backtrace:

Encoding::UndefinedConversionError in (method name...)

"\xE2" from ASCII-8BIT to UTF-8

Application Trace
app/controllers/application_controller.rb:64:in `respond_with_server_error'
app/controllers/application_controller.rb:31:in `block in <class:ApplicationController>'


Framework Trace
json (1.7.7) lib/json/common.rb:254:in `encode'
json (1.7.7) lib/json/common.rb:254:in `generate'
json (1.7.7) lib/json/common.rb:254:in `fast_generate'
sentry-raven (2.5.1) lib/raven/client.rb:67:in `encode'
sentry-raven (2.5.1) lib/raven/client.rb:37:in `send_event'
sentry-raven (2.5.1) lib/raven/instance.rb:81:in `send_event'
sentry-raven (2.5.1) lib/raven/instance.rb:126:in `capture_type'
activesupport (4.2.4) lib/active_support/rescuable.rb:114:in `instance_exec'
activesupport (4.2.4) lib/active_support/rescuable.rb:114:in `block in handler_for_rescue'
activesupport (4.2.4) lib/active_support/rescuable.rb:80:in `call'
activesupport (4.2.4) lib/active_support/rescuable.rb:80:in `rescue_with_handler'
....

Now, this seemed to happen only when the payload contained some funky characters, perhaps 4-byte unicode characters (emojis).

As you can see above it was triggered with json (1.7.7) gem, but I upgraded to latest json gem and it still happened.

You can probably reproduce this by including some emojis such as ▷▷ ⬧ ✈ in the messgae.

Also, FYI I reverted to the earlier version raven-ruby 2.4.0 and this fixed the problem. It did happen with gem versions 2.5.0 and 2.5.1.

If you cannot reproduce this, it might be that my ruby version is to blame ― granted it's a bit old: ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux] In that case let me know so that I can do more tests with newer ruby versions.

[nateberkopec]

No, this is our fault, not yours. What I'm trying to figure out is how a character which is invalid in ASCII slipped through the processor. Can you give me some more details on reproduction, like where exactly these characters are? In the request body? In the querystring?

[avel]

@nateberkopec Great to know we haven't done anything wrong! :) I'll provide you with a complete example soon.

[avel]

Looked a bit deeper into this.

Turns out it's something in the request data that's coming along malformed:

[8] pry(#<Raven::Client>)> hash[:request][:data]
=> "{\"title\":\"Sentry Debugging\",\"description\":null,\"subject_lines\":[{\"body\":\"\xE2\x9C\x89 Hello ...."} ]

(redacted some of my JSON data)

Triggering the error:

[10] pry(#<Raven::Client>)> JSON.generate(hash[:request][:data])
Encoding::UndefinedConversionError: "\xE2" from ASCII-8BIT to UTF-8
from /home/avel/.rvm/gems/ruby-2.1.1@abplus/gems/json-1.7.7/lib/json/common.rb:223:in `encode'

Fixing the problem at this stage:

[13] pry(#<Raven::Client>)> hash[:request][:data].encoding
=> #<Encoding:ASCII-8BIT>
[14] pry(#<Raven::Client>)> hash[:request][:data].force_encoding 'utf-8'
=> "{\"title\":\"Sentry Debugging\",\"description\":null,\"subject_lines\":[{\"body\":\"✉ Hello ...}]
[15] pry(#<Raven::Client>)> hash[:request][:data].encoding
=> #<Encoding:UTF-8>

And after that, this

[19] pry(#<Raven::Client>)> JSON.generate(hash)

works fine!

What I'm currently researching now is, why does that request hash end up there malformed like this in my application.

In fact I've tried to manually POST data from my browser, in various ways (encoding as json, with UTF-8 content-type headers, without, encoding as URL-encoded form), and could not reproduce this. I'll look for the cause - something in the middle is causing this - and will let you know. In the meantime I hope this is useful information.

[nateberkopec]

Thanks. 3.5.0 basically removed forcefully trying to re-encode every string as UTF8, which is what caused this. My assumption was that we could let any string through the processor pipeline as long as it a) had an encoding and b) that string was valid with that encoding. So I'm trying to figure out how that could be not true here.

[dentarg]

@nateberkopec https://github.com/getsentry/raven-ruby/blob/master/changelog.md says KNOWN ISSUE: Character encoding errors [#689], was this fixed in 2.5.2 or was that line just to give users a heads-up about this issue?

[nateberkopec]

No, "known issue" indicates we know it's a problem but haven't fixed it.

[nateberkopec]

Yay, I finally have a replicating test! 🎉

After @sgrif pointed out on Twitter that ASCII-8BIT is Ruby's "I don't know what this is, it's just raw bytes" encoding. It contains all possible codepoints, so "valid_encoding?" will always return true. It's clear that most requests bodies may use that encoding.

So we'll need a special case in the UTF8 converter processor for ASCII_8BIT.

[nateberkopec]

@avel just curious, on production, where you see this problem, what is the output of ruby -e 'p Encoding.default_external'

[avel]

@nateberkopec It's #<Encoding:UTF-8>

Today I investigated this again... Really quickly I'd just like to mention that the whole thing seems to originate at integrations/rack.rb, where you read from this stream:

      elsif request.body # JSON requests, etc
        data = request.body.read(2048) # Sentry server limit

This data will always have the unwanted ASCII-8BIT encoding for me.

I also noticed the same with request object's RAW_POST_DATA, it's ASCII-8BIT encoding with invalid characters.

Fixing my front-end app to send the payload with content-type 'application/json; charset=utf-8' did not do anything to change this.

The above happens with rails 4.2.4. Took a look through actionpack's changelog in case there's something in newer rails but didn't find anything conclusive.

I have no idea if this is any helpful at all :blush:

---

Edit: Just saw your comment above about replicating! Great news.

[avel]

@nateberkopec Great fix! :+1: :+1: :+1:

[doitian]

async still raises encoding error when request data contains unicode.

[nateberkopec]

@doitian please open another issue with more information. What encoding exactly, a stacktrace, etc.