Akka-http: TCP RST was sent with large response body

[keimoon]

Akka HTTP version: 3.0.0-RC1

How to reproduce:

• Use small MTU for network interface (for example 1500). You can set MTU for loopback interface with sudo ifconfig lo0 mtu 1500
• Response with large body content (should be larger than 128KiB) using complete and ToResponseMarshallable
• Make sure client send header Connection: close.

Expected result:

Client should receive normally

Actual result:

Client got Recv failure: Connection reset by peer

This causes problems with some proxy like nginx or kong because they send Connection: close to upstream by default.

[ktoso]

Thanks for reporting. I'll mark as bug though I'm not sure about it.

[jrudolph]

It could be like this: when the server sees Connection: close on a request it fully closes the connection directly after having sent out the last bit of the response which might close the socket too soon?

@keimoon which client did you use? Could you try to capture a network trace using tcpdump and attach it here? Thanks ;)

[keimoon]

I used curl. Here is network trace that you can load into wireshark

akka.pcap.tar.gz

[keimoon]

Also I can only reproduce the bug with HttpEntity.Strict

[jrudolph]

Great, thanks @keimoon. That will help a lot.

[jrudolph]

I had a quick try, this issue can be reproduced using your instructions.

The problem is that when the HTTP server decides to close the connection, it does so by completely killing the HTTP stack which will complete the write-side of the TCP connection stream and cancel the read-side. If cancel arrives first (which seems to be the case), the Tcp stream implementation will abort the connection (= socket.close with soLinger(0)). The result is that the OS just doesn't know about the connection any more and if any data was not sent yet this data is lost and any further packets will be responded with RST.

We've seen issues like that for a long time (IIRC also with spray). I guess the solution could be just to keep cancellation from reaching the TCP connection. (I don't think that this creates leaks, hopefully leaks would be prevented by the idle-timeout on the connection but it might make sense to think a bit about it.)

[keimoon]

Great. Currently we are implementing an work around that will use HttpEntity.Chunked if the response body is large.

[mariszin]

We migrated our app to akka-http 10.0.0, and received the same error - when using anything, that sends header "Connection: close" (nginx, curl), the request will fail on Strict entities, that are larger than 128K.
Error in curl:
curl: (56) Recv failure: Connection reset by peer

[guntiso]

Chunked does not help - changed response entity to HttpEntity.Chunked(mediaType, Source(Seq(ByteString(prettyJsonString)))), and now it works with curl, but still fails with nginx. Is there any reliable workaround?

[daandi]

Same here :( Would really appreciate a way to solve this.

[ktoso]

So if I understand correctly, we'd need a "CancellationBarrier" that can be "opened" by external signal, which we could use to resolve the race (making it always "cancel" after the completion went "all the way"). Reasonable @jrudolph ?

[jrudolph]

We might not need to send cancellation to the tcp stream at all but I haven't checked if that would work.

[jrudolph]

I reproduced it with this code https://github.com/jrudolph/akka-http/commit/a929a03f9d68177f9f04153dac6409acddbcc9dd and curl:

curl -vvv -H "connection:close" http://localhost:9001/big > /dev/null

[gisql]

Workaround with nginx here: https://github.com/akka/akka/issues/19542

[jonas]

Tested a locally published version of Akka HTTP and I no longer see connection resets.

[ktoso]

Cool, thanks for testing!

[antonkatz]

I am perhaps seeing this again with nginx.
I'm serving an html page with Akka-Http 10.0.9 with getFromFile().
The file contains an tag with src containing base64 encoded image.

Nginx has this in the error logs.
upstream prematurely closed connection while reading upstream

The error is resolved by using the workaround in akka/akka#19542

[jonas]

@antonkatz Is the traffic served over HTTPS?

[jrudolph]

@antonkatz we fixed issue #1219 in 10.0.10 which was similar to this bug when running on HTTPS. Can you check if 10.0.10 still has the issue?