Akka-http: Akka Websocket 4x slower when updating Scala 2.12 to 2.13

[galbarm]

Hello,
Akka version is 2.5.26 and Play is 2.7.3.
Following is a sample server that demonstrates the issue:

object Test extends App{

  val app = GuiceApplicationBuilder(configuration = Configuration.apply("play.server.websocket.frame.maxLength" -> 11 * 1024 * 1024)).build

  val routes: PartialFunction[RequestHeader, Handler] = {
    case GET(p"/test_pressure") => WebSocket.accept[play.api.http.websocket.Message, play.api.http.websocket.Message] { request => {
      Flow.fromSinkAndSource(Sink.foreach(in => {
        println("got message")
      }), Source.maybe)
    }
    }
  }

  val config = ServerConfig(
    rootDir = new File(""),
    port = Some(0),
    sslPort = None,
    address = "127.0.0.1",
    mode = Mode.Test,
    properties = System.getProperties,
    configuration = app.configuration
  )

  val server = AkkaHttpServer.fromRouterWithComponents(config)(_ => routes)

  println("port = " + server.httpPort.get)
}

A client connects to this server and sends 100 websocket messages, 10MB in size each (1GB total) as fast as possible.

I’m starting the server in 2 configurations:

scalaVersion := “2.12.10” in build.sbt
scalaVersion := “2.13.1” in build.sbt
This is the only difference between the two tests.

The issue is that in Scala 2.12.10 the test takes 6 seconds to complete, whereas in Scala 2.13.1 the same test takes 22 seconds.

[galbarm]

Likely to be related or caused by:
https://github.com/akka/akka/issues/28114

[galbarm]

As I mentioned in https://github.com/akka/akka/issues/28114
The issue only reproduces when using AkkaHttpServer via it's Play wrapper (play-akka-http-server).

When using the stand alone version Http().bindAndHandle... , or when using Netty as backend NettyServer.fromRouterWithComponents... the issue does not reproduce.

[jrudolph]

Thanks for the information, @galbarm. Play reimplements part of the WS stack so it might interact differently. If you have an example play project to share that shows the issue that would be great.

We identified (and are fixing) at least one issue in akka/akka#28116. I also improved WS masking performance in #2801. Once we have backported the fix to 2.5.x, would you be able to try with a nightly snapshot?

I've filed https://github.com/playframework/playframework/issues/9799.

[galbarm]

@jrudolph Thanks for the attention. The code above https://github.com/akka/akka-http/issues/2788#issue-513177681 is a complete App demonstrating the issue.
The only references I have in build.sbt are

  "com.typesafe.play" %% "play" % "2.7.3",
  "org.scalatestplus.play" %% "scalatestplus-play" % "4.0.3"

and scalaVersion := "2.12.10" / "2.13.1"

I will definitely try the nightly snapshot once the changes are pulled.

[jrudolph]

The code above #2788 (comment) is a complete App demonstrating the issue.

Ah, I didn't realize. That's great.

[galbarm]

As for the client, I've reproduced it with several of them.
Here's a sample one using https://github.com/andyglow/websocket-scala-client

import com.github.andyglow.websocket.WebsocketClient

object Main extends App {

  val port = 9666
  val ws = WebsocketClient[Array[Byte]](s"ws://localhost:$port/test_pressure"){
    case _ =>
  }

  val socket = ws.open()
  val b = new Array[Byte](10 * 1024 * 1024)

  for (w <- 0 to 100){
    socket ! b
    println(s"sent message #$w")
  }

}

[jrudolph]

I reproduced it also using your example. It's indeed the same problem there. I guess play collects frames slightly differently so ByteString.toArray will work on a composite ByteString compared to the simple ByteString in akka-http. That will make the performance even worse.

[galbarm]

@jrudolph Your assumption seems to the right. See here:
https://github.com/akka/akka/issues/28114#issuecomment-567108603

I've also noticed while debugging that indeed AkkaHttpServer via it's Play wrapper creates a large ByteString composed of many smaller ones and then calls ByteString.toArray in FrameEventParser.
But in direct use of AkkaHttpServer ByteString.toArray is being called on each piece separately.

[galbarm]

Attaching the result of testing how the message size effects the performance.

[jrudolph]

Fixes have been released with the latest akka versions. Maybe you could try and see if it has been fixed with Akka 2.5.29?

[galbarm]

@jrudolph Tested with Akka 2.6.3, I can confirm the issue has been fixed.
Thanks a lot.

[jrudolph]

Great, thanks for testing again!

[jrudolph]

(Marking as duplicate as it was fixed in akka/akka)

[raboof]

Thanks :tada: