Reactor-netty: PrematureCloseException: Connection prematurely closed BEFORE response in load testing

[GaelGirodon]

Expected Behavior

Reactor Netty (in Spring Cloud Gateway) should be able to handle a high request rate with many concurrent connections.

Actual Behavior

During a load testing using vegeta, when I try to send 500 requests/sec with up to 50 concurrent workers, Spring Cloud Gateway fails handling some of the requests (less than 1%, but this ratio becomes higher with more req/s and workers) and logs the following error message:

2020-03-24 20:28:29.625 ERROR 118133 --- [or-http-epoll-4] a.w.r.e.AbstractErrorWebExceptionHandler : [f650df5a-20596]  500 Server Error for HTTP GET "/test/api"

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        |_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ HTTP GET "/test/api" [ExceptionHandlingWebHandler]

Everything looks good with a lower rate and less concurrent workers.

Steps to Reproduce

• Create an empty Spring Cloud Gateway project
• Add a route like this one in the main configuration file:

spring:
  cloud:
    gateway:
      routes:
        - id: test-api
          uri: http://localhost:3000 # A super-fast stub server
          predicates:
            - Path=/test/api/**
            - Method=GET
          filters:
            - RewritePath=/test/api/?(.*), /api/v1/$\1

• Run the gateway behind a reverse proxy (Nginx in my case)
• Load test the gateway with a high number of requests/sec and many concurrent workers

Possible Solution

Several issues are already open in Spring Cloud Gateway and Reactor Netty repositories but none of them seems to provide a working solution for this error. I tried the following things, without success:

• Set a value for the maxIdleTime (null -> 100 ms)
• Set the pool type to FIXED instead of ELASTIC to avoid getting too much opened connections
  
  
  
  • Allow more connections (maxConnections)
  
  
  • Set a higher value for acquireTimeout
  
  

In the end, I'm unable to find the right configuration to avoid this bug. I have another older project (CXF/Camel) running on an Apache Karaf server, the response time may be higher but it is able to handle many requests and connections without failing. I didn't expect such an error with a non-blocking web stack ;)

Your Environment

• Reactor version(s) used: 0.9.4
• Other relevant libraries versions (eg. netty, ...): Spring Cloud Gateway 2.2.1
• JVM version (java --version): OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
• OS and version (eg uname -a): CentOS Linux release 7.7.1908

[violetagg]

@GaelGirodon We have fixes in 0.9.6.RELEASE related to the connection pool, please test with that version and share the results.

[GaelGirodon]

Thank you for your prompt response! I upgraded to 0.9.6.RELEASE and it seems much better! I don't get this exception anymore but I still have some performance issues with a high rate and many concurrent workers.

Some results of my load testing (mean latency in ms), with a basic route and a route with multiple Gateway filters:

| Req/s / Workers | 1 / 1 | 10 / 1 | 50 / 10 | 100 / 20 | 500 / 50 |
| --- | --- | --- | --- | --- | --- |
| Basic route | 2 | 2 | 1 | 1 | 50 |
| Multiple filters | 10 | 11 | 8 | 8 | 206 |

As you can see, there is a clear performance drop with 500 req/s and 50 concurrent workers. With a server like Apache Karaf and a Camel/CXF stack, the mean latency is globally higher (x2 to x20), except for the last test where the mean latency is lower (23ms and 106ms).

Do you have some advices to improve the performance with this kind of scenario?

[violetagg]

@GaelGirodon Please provide some reproducible scenario.

[GaelGirodon]

@violetagg The load testing results are associated with the scenario from the issue description (see Steps to Reproduce) but it may not be very quick to reproduce as it is necessary to setup a load testing environment.

Maybe I should open an issue in the Spring Cloud Gateway repository as they should be able to reproduce this scenario easily.

Furthermore, with the last basic route test (500 req/s and 50 concurrent workers), I have some connection timeouts: max latency is equal to 60s and leads to a timeout and a 504 Gateway Time-out as if the Reactor Netty WebClient wasn't able to handle as many concurrent connections from the same host.

[violetagg]

@GaelGirodon

The load testing results are associated with the scenario from the issue description (see Steps to Reproduce) but it may not be very quick to reproduce as it is necessary to setup a load testing environment.

predicates:
            - Path=/test/api/**
            - Method=GET
          filters:
            - RewritePath=/test/api/?(.*), /api/v1/$\1

so the issue is existing if the target server endpoint does nothing?

Maybe I should open an issue in the Spring Cloud Gateway repository as they should be able to reproduce this scenario easily.

that would be better

Furthermore, with the last basic route test (500 req/s and 50 concurrent workers), I have some connection timeouts: max latency is equal to 60s and leads to a timeout and a 504 Gateway Time-out as if the Reactor Netty WebClient wasn't able to handle as many concurrent connections from the same host.

this is with elastic or fixed connection pool?

[GaelGirodon]

@violetagg

so the issue is existing if the target server endpoint does nothing?

The issue seems to be caused by the gateway not being able to respond fast enough on this endpoint with a high rate and many concurrent workers (too much opened connections?). I'm using the same stub server (a simple "echo" HTTP server written in Go) on each test machine (Camel/CXF/Karaf server and Spring Cloud Gateway server) and I don't have these performance drops with the Camel/CXF/Karaf stack (or at least, the performance is not that worse).

that would be better

Ok! I'll open it.

this is with elastic or fixed connection pool?

I use the default Spring Cloud Gateway configuration so elastic.

[violetagg]

I use the default Spring Cloud Gateway configuration so elastic.

Is it possible that you exhaust the system resources?
What's the behaviour if you use fixed connection pool?

[GaelGirodon]

Is it possible that you exhaust the system resources?

It's a possibility yes. The VM has 2 vCPU (Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz), 8 GB of RAM and 130 GB of HDD. But the VM with the Camel/CXF/Karaf stack has approximatively the same hardware configuration (a bit older and no Nginx reverse proxy) and I don't have such performance drops on this one.

What's the behaviour if you use fixed connection pool?

spring:
  cloud:
    gateway:
      httpclient:
        pool:
          type: fixed

Test results (with 500 req/s and 50 workers):

• Mean latency: 52ms (max: 60s, min: <1ms)
• Results: 4957 200 OK, 39 500 Internal Server Error, 4 504 Gateway Time-out

Logs associated to 500 Internal Server Error:

2020-03-25 12:05:20.311 ERROR 19269 --- [or-http-epoll-3] a.w.r.e.AbstractErrorWebExceptionHandler : [552ca111-5154]  500 Server Error for HTTP GET "/test/api"

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        |_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
        |_ checkpoint ⇢ HTTP GET "/test/api" [ExceptionHandlingWebHandler]

EDIT: a second run gives a much better latency (13ms, still 10x slower than tests with a lower rate and less workers) and one 504 Gateway Time-out.

_Interesting note_: when I test the other route (that has multiple filters with many request body modifications), the mean latency increases from 10ms to 265ms with a 500/50 test but I haven't gotten any timeout so far. I would prefer that: a worse latency but no error!

EDIT 2: with more req/s and workers (e.g. 1000 req/s and 100 workers), I get more 50x errors.

[violetagg]

@GaelGirodon I'm trying to minimise the scenario.
If you remove the rewrite filter do you receive the same numbers?
If you remove Nginx do you receive the same numbers?

[GaelGirodon]

You're right! I should have tried to bypass Nginx before.

I didn't expect Nginx to cause this problem (I mean... it's not Nginx's fault, but having a reverse proxy in front of the gateway, on the same machine, seems to cause this problem). I cannot explain this behavior precisely (too many opened connections within the same machine, I/O limitations?).

| Req/s / Workers | 1 / 1 | 10 / 1 | 50 / 10 | 100 / 20 | 500 / 50 | 1000 / 100 |
| --- | --- | --- | --- | --- | --- | --- |
| Mean latency (ms) | 1.7 | 1.5 | 1.3 | 1 | 0.9 | 2.1 |

Is it safe to expose a Netty server directly on the network without reverse proxy? (we're used to configure Apache HTTPD or Nginx in front of Tomcat/Jetty/etc.)

[violetagg]

@GaelGirodon Great! May be it is better to install Nginx on a separate host to avoid sharing the system resource like CPU/ulimit.