Reactor-netty: Got Connection prematurely closed BEFORE response

[thsnoopy]

Expected behavior

When using the WebClient to POST something in parallel, I should be getting successful results.

Actual behavior

Intermittently, I got an error with reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response. In my experience, it arises with 20 percent probability.

Steps to reproduce

I googled and searched related issue and they say it may be resolved in latest reactor-netty version. But unfortunately, even if I use the latest spring-boot & reactor-netty version I got a similar issue.

I also read recently registered issue #774 , and it requires more reproducible information like logging and wiretap.

So, I've made a reproducible demonstration: https://github.com/thsnoopy/reactor-netty-report .
(It is configured with logging.level.reactor.netty=debug and httpServer.wiretap)

The demonstration is composed of three apps :

• server-actor : Provides POST API endpoint
• client-actor : A client that calls POST API using WebClient
• test-runner : Just a trigger app that client-actor communicates with server-actor

test-runner --(GET)--> client-actor --(POST)--> server-actor

I hope it will help you for debugging :)

You can check out and run ./runTest.sh and see the log file in client-actor/log/server-actor-***.log. In my experience, the test failed with 20 percent probability.
And one more, if you run the test after restarting the server(server-actor and client-actor), you will get an error with a higher probability(runTest.sh script already involve this).

Reactor Netty version

• reactor-netty : 0.8.9.RELEASE
• spring-boot : 2.1.6.RELEASE

JVM version (e.g. java -version)

java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

OS version (e.g. uname -a)

Darwin name 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64 (macOS Mojave 10.14.2)

[thsnoopy]

Added full error log with Connection prematurely closed BEFORE response
Download : https://github.com/thsnoopy/reactor-netty-report/blob/master/out.log

[harish0000]

this will help
https://github.com/spring-projects/spring-framework/issues/22464

[violetagg]

@harish0000 As I wrote here https://github.com/spring-projects/spring-framework/issues/22464#issuecomment-499474686
The cause for this particular issue is reactor/reactor-netty#632 and it is fixed (a header Connection with more than one value)

[violetagg]

@thsnoopy Thanks for the reproducible scenario. We are working on a fix.

[thsnoopy]

@violetagg Thank you for investigating.

[smaldini]

@thsnoopy just a quick heads up - we root caused the issue to two things :

• the exception is not very self descriptive, but its a valid one
• the root cause of the exception : a connect timeout due to too many concurrent connection open/acquired.

We are going to change the default connection pool for our clients in 0.9, and document how to set those up. What happens right now is there is no limit, so you are opening 80000 connections to a single destination, thats more than ephemeral ports and thats overall a lot of pending events that could be delayed and cause a connect timeout. Usually, connection pools are capped by default, e.g. Golang has 100 (including only "2" by destination), other libraries run 64, 100 or again 2-6 (the RFC recommendation for browsers). We should limit the number of open connections too by default and we benchmarked with ConnectionProvider.fixed("test", 500) without any troubles.

So i'm leaving the issue open until we change that default and document it overall but you should try as a workaround to set your client this way:

HttpClient.create(ConnectionProvider.fixed("test", 500))

[smaldini]

For reference https://github.com/reactor/reactor-netty/issues/578

[thsnoopy]

@smaldini
Thank you for detailed explanation!

I have one more question. Is this symptom also happen with GET request? Actually, I only got this error with POST request in my production service not GET request.
But theoretically, if the GET response with more slow response time, then can a similar error occur?

[violetagg]

@thsnoopy with your example I'm reproducing it with GET also.
I'm closing this - in 0.9.x I switched the connection pool by default to be fixed with max connections 500 - #578

[softwareklinic]

I tried out the code sample posted by @thsnoopy --- using the latest reactor netty and the spring reactor netty started 2.2.4 -- but the issue still persists... it is using netty 4.1.45.Final and still seeing the premature connection issue.

[YuryYaroshevich]

I have the same error: Connection prematurely closed BEFORE response. I have the following config for http client:

HttpClient.create(ConnectionProvider.fixed(baseUrl, 500, 60_000));

spring version: id 'org.springframework.boot' version '2.2.4.RELEASE'
reactor-netty: 0.9.4.RELEASE
Could someone please clarify what are possible reasons for this error? In what direction we need to investigate?

[violetagg]

@YuryYaroshevich If you have a reproducible example, please create a new issue with it.

[YuryYaroshevich]

@violetagg I have an application which sends dozens of requests here there and I see this error without any correlation with request type or 3rd party service. What example can I create? Could you please give me some clue on what can cause an error with such message?
Connection prematurely closed BEFORE response

[violetagg]

@YuryYaroshevich this issue here was related to the ConnectionProvider.elastic usage.
You see Connection prematurely closed BEFORE response with ConnectionProvider.fixed, so let's not spam this issue.
I would suggest to move this discussion to Gitter. And the first thing that you should do is to take a tcp dump to see which party closes the connection and why.

[sanjosva]

I get same error when using the WebClient to POST something in parallel(I am using spring-boot-starter-webflux:jar:2.2.4(which comes with the reactor-netty:jar:0.9.4.RELEASE). the webclient is in an async executer.
I get this error when the http request is very large. for small ones, no issues. is it because the other/main thread that was working completed?

[violetagg]

@sanjosva Please create a new issue with a reproducible example against the latest released version.

[saralpa]

Hello All,

I also got similar issue today. I am using latest version of azure-storage-blob :
compile group: 'com.azure', name: 'azure-storage-blob', version: '12.7.0'
which internally uses :
compile group: 'io.projectreactor', name: 'reactor-core', version: '3.3.7.RELEASE'

I am sending approx 7000 request in parallel to azure destination.
Please find attached logs

Please let me know how can I resolve this issue.

[violetagg]

@saralpa This issue's been closed already, create a new issue and provide tcp dump so that we can see which peer closes the connection.

[nitinkalenk]

@violetagg any other issue opened for this?.

[violetagg]

@nitinkalenk please open new issue