Akka-http: Problem with emoji?

[troypayne]

I'm using "com.typesafe.akka" %% "akka-http-spray-json-experimental" % "2.4.7", and doing something like:

case class User(username: String)
val userFormat = jsonFormat1(User)

and whenever username: String has an emoji in it, it does not properly handle it. I get the question mark in a bounding box.

This is something new as I just recently upgraded to scala 2.11.8, sbt 0.13.13, and replaced spray.io with akka-http, where things were being handled properly.

Has anyone seen this and know what to do?

I've confirmed that the marshaller is the problem, as I wrote a custom marshaller and the problem goes away.

I've also written a reproducer that can be found here:
https://www.dropbox.com/s/46fy46uojcnjs1u/echo.zip?dl=0

Just download the folder, cd into echo, type: sbt run and then open another terminal on the same machine echo is running and type:

curl -XPOST 'localhost:8080/echo' -d '{"msg":"😎"}' --header "Content-Type: application/json"

You will get something back like:

{"
"msg": "�"
}

Thanks!

[ktoso]

Thanks for investigating over at spray-json @troypayne!
Let's see what we find here hm.

[sorenbs]

Just want to add that we are seeing the same issue. This worked for us previously, but I haven't yet tried to pinpoint exactly when it changed.

[troypayne]

@ktoso @sirthias
Comparing the code of spray-json's handling of unicode encodings appears that it supports surrogate pairs, but akka-http-spray-json doesn't. The code is different and in my opinion should be the same. I'm not sure if I can fix the issue and submit a PR because I'm unable to test my local copy of akka-http as when I publish locally, it only publishes jackson and the parsing package.

If you guys can help me figure out how to get my fork'd akka-http running alongside my project, I can attempt to add support for handling unicode surrogate pairs by simply doing exactly what spray-json is doing. See this diff:

https://github.com/spray/spray-json/commit/0c9b2feed5d5d253181a49beb1e806ba9df3bd1b#diff-3cad8170296c5b48752df35b7a89db77

vs.

https://github.com/akka/akka-http/blob/master/akka-http-marshallers-scala/akka-http-spray-json/src/main/scala/akka/http/scaladsl/marshallers/sprayjson/SprayJsonByteStringParserInput.scala

[ktoso]

Great find! I've prepped a PR and will add tests to akka-http-spray-json itself as well, thanks for helping find this 👍

[troypayne]

@ktoso any guess on when you think this fix will be something I can start using? Or perhaps you can share how you run a local version or a pull-requested version?

[ktoso]

In 10.0.1 ;)
In a few weeks

On Nov 23, 2016 04:54, "Troy Payne" [email protected] wrote:

@ktoso https://github.com/ktoso any guess on when you think this fix
will be something I can start using? Or perhaps you can share how you run a
local version or a pull-requested version?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/akka/akka-http/issues/564#issuecomment-262430000, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAHYk5f1SCCfTNC0nANLNLGu_-y-xv3zks5rA7jbgaJpZM4K4tEN
.

[troypayne]

@ktoso ouch. Are there instructions on how to run a local or forked checkout? I'm having trouble with this for some reason.

What I did was forked akka-http, then did sbt publish local and checked my ~/.ivy2/local folder to see all that I could reference, and the only things listed were the jackson and the parsing package. Please help?

[schickling]

Any update on this @ktoso? This is causing quite a lot of problems for us. Any workaround for now?

[troypayne]

@schickling +1. I'm patiently waiting for 10.0.1 to be released, which I believe is just 1 week away. I haven't been able to figure out how to publish akka-http locally to run the fix myself yet, but if you figure it out please share. Here's what I've been tracking: https://github.com/akka/akka-http/pull/568

[ktoso]

It will be released, please do not rush. We were having a very intense week and are now catching up with things.

Publishing is as simple as sbt publishLocal

[timothyklim]

@troypayne You can publish your own build into free bintray account and use it with resolvers += Resolver.bintrayRepo("account", "maven) in build.sbt if you build project through CI.

[ktoso]

Merged, will be in 10.0.1

[troypayne]

So 10.0.1 was released today, and the fix works, but only for 10.0.1 for scala 2.12, not 10.0.1 for scala 2.11... @ktoso do you know why this is? I was able to update to 2.12 but hit a roadblock because one of my modules (akka-stream-apns) doesn't have a version built for 2.12 and it's causing conflicts while compiling.

Could this be another bug?

[jrudolph]

Hmm, seems unlikely that there's a difference in behavior for those two versions. The tests should have caught it, otherwise.

Could it be a dependency problem? Can you share the output of sbt "show dependencyClasspath"?

[ktoso]

Yeah it sounds as if you did not really run with 10.0.1 there, please
double check dependencies.

On Dec 23, 2016 10:50, "Johannes Rudolph" [e-mail] wrote:

Hmm, seems unlikely that there's a difference in behavior for those two
versions. The tests should have caught it, otherwise.

Could it be a dependency problem? Can you share the output of sbt "show
dependencyClasspath"?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/akka/akka-http/issues/564#issuecomment-268965159, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAHYkzgsfbgLfkYs4PCyjbWuZ9wRH7Npks5rK5lUgaJpZM4K4tEN
.

[troypayne]

@jrudolph @ktoso Here are the results of show dependencyClasspath
https://gist.github.com/troypayne/13a3e1a6e8d6450f10355333ecc0d672

Here's what my build.sbt looks like:
https://gist.github.com/troypayne/7da72ae6bd43363f13ea277d2927f098

Thanks for your help!

[troypayne]

@jrudolph @ktoso any ideas?