Reactor-netty: syscall:read(..) failed: Connection reset by peer

[deepkimchi]

Actual behavior

    @Bean
    public WebClient webClient(ReutersSetting reutersSetting, ExchangeStrategies exchangeStrategies) {
            return WebClient.builder()
                        .baseUrl(ReutersEndPoints.HOST)
                        .defaultHeader(HEADER_APP_ID, reutersSetting.getApplicationId())
                        .exchangeStrategies(exchangeStrategies)
                        .build();
    }

    @Bean
    public ExchangeStrategies exchangeStrategies() {
        ObjectMapper mapper = objectMapper();
        return ExchangeStrategies
                .builder()
                .codecs(clientDefaultCodecsConfigurer -> {
                    clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonEncoder(new Jackson2JsonEncoder(mapper, MediaType.APPLICATION_JSON));
                    clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder(mapper, MediaType.APPLICATION_JSON));
                }).build();
    }

    public ObjectMapper objectMapper() {
        return  Jackson2ObjectMapperBuilder
                .json()
                .failOnUnknownProperties(false)
                .featuresToEnable(SerializationFeature.WRAP_ROOT_VALUE)
                .featuresToEnable(DeserializationFeature.UNWRAP_ROOT_VALUE)
                .build();
    }

WebClient initiated via above configuration, throws Exception occasionally such as below.

2019-01-07 11:33:22.188 ERROR [-,,,] 92270 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x6f488001, L:/xx.xx.xx.xx:53500 - R:api.trkd.thomsonreuters.com/xx.xx.xx.xx:443] Pooled connection observed an error

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

WebClient is called regularly under Scheduled task.

Steps to reproduce

Happen randomly

Reactor Netty version

reactor-netty:0.8.3.RELEASE

JVM version (e.g. java -version)

openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

OS version (e.g. uname -a)

Linux xxxx 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[darekurlik]

I've set ChannelOption.SO_KEEPALIVE to false to make it go away as follows:

@Bean
public WebClient webClient(final ClientHttpConnector clientHttpConnector) {
    return WebClient.builder()
        .clientConnector(clientHttpConnector)
        .build();
}

@Bean
public ClientHttpConnector clientHttpConnector(@Value("${webclient.enable-keep-alive}") final boolean keepAlive,
                                               @Value("${webclient.read-timeout-in-seconds}") final int readTimeout,
                                               @Value("${webclient.write-timeout-in-seconds}")
                                               final int writeTimeout) {
    return new ReactorClientHttpConnector(HttpClient.from(TcpClient.create()
        .option(ChannelOption.SO_KEEPALIVE, keepAlive)
        .doOnConnected(connection -> connection
            .addHandlerLast(new ReadTimeoutHandler(readTimeout))
            .addHandlerLast(new WriteTimeoutHandler(writeTimeout)))));
}

Please be aware that you are disabling keep alive connections and it may have and impact on latency.
Also check if there is nothing between your client and server that you are trying to hit. I found that any proxy or load balancer in between can complicate your life.

[qingmu-io]

I use haproxy as a pre-entry and continue to throw the same exception when the backend is reactor-netty.

[darekurlik]

We still have a lot of problems due to invalid haproxy configuration... This post helped us a lot to clear most of the issues. Unfortunately we still need to deal with problems on the health cheak. I hope it helps.

[violetagg]

@SeungJinCho @247687009 Can you provide tcpdump so that we can analyse why the connection is reset?

[qingmu-io]

@darekurlik Thanks!
I used the following configuration in haproxy, which is a perfect solution.

defaults
mode http
option forwardfor except 10.42.0.0/24
option prefer-last-server
balance leastconn
timeout server 600s

[deepkimchi]

@violetagg Sorry, I've been away from a while from this issue.
As it keep happening, let me prepare the dump data you asked in a near future.
Thanks

[deepkimchi]

@violetagg I'm attaching the dump and you can find [RST, ACK] in No.86
dump.zip

[violetagg]

@SeungJinCho Do you have some error log messages on the server and on the client for the corresponding ports?

[deepkimchi]

As I'm client side, this is only one i got..

2019-01-18 18:34:31.770 ERROR [-,,,] 4772 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider   : [id: 0x6378b7b6, L:/10.127.152.122:49600 - R:api.trkd.thomsonreuters.com/159.220.24.218:80] Pooled connection observed an error

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

[violetagg]

@SeungJinCho For this specific port 49600 there is nothing in the tcpdump. The only thing that I can say now is that the server closed the connection.

[antongrinenko]

I have the same issue in my spring boot project:

ERROR [reactor-http-epoll-1] [reactor.core.publisher.Operators] Operator called default onErrorDropped io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

If I can help you with some additional debug info fill free to ask.
Waiting for fix

[igorkatz]

I have the same issue:

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source) ~[netty-transport-native-unix-common-4.1.33.Final.jar!/:4.1.33.Final]

I use spring boot 2.1.3.RELEASE that brings netty 4.1.33.Final
The same version is used on the server and client side
The applications are dockerized
The problem occurs after an idle time.
If I disable the connection pool the problem disappears, but this has performance penalties

[violetagg]

@antongrinenko What's the Reactor Netty (Spring Boot) version?

@igorkatz Do you see this as an ERROR log or as a DEBUG log? Do you see this on the server or on the client?

[igorkatz]

It was ERROR on the client side.
I understood the problem. I use Swarm to orchestrate my dockerized applications. Swarm has a load balancing (LB) to implement its routing mesh. LB closes inactive connections. Enabling of keep alive for channels was not enough because it started after connection closing.
Implementation of keep alive on application level for idle connections solved the problem.

[tonyskulk]

@igorkatz Glad that you got this to work. Can you specify what exactly you meant by

Implementation of keep alive on application level for idle connections solved the problem

.

[igorkatz]

See below link for more information:
https://netty.io/4.1/api/io/netty/handler/timeout/IdleStateHandler.html

[jarpz]

@igorkatz
Hi I would like to implement that solution (https://netty.io/4.1/api/io/netty/handler/timeout/IdleStateHandler.html) but I don't know which PingMessage class comming from. Could you help me?

[igorkatz]

This is my implementation:

    @Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception
    {
        if (evt instanceof IdleStateEvent) {
            IdleStateEvent e = (IdleStateEvent) evt;
            if(e.state() == IdleState.ALL_IDLE) {
                HttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, healthCheckRawPath);
                ctx.writeAndFlush(request).addListener(future -> {
                    if(!future.isSuccess()) {
                        logger.error("Health check error");
                        ctx.close(); 
                    }
                });
            }
        }
    }

[SudBisht]

I am also getting error while accessing Remote Redis. any help would be appreciated

org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to x.x.x.x:6379 at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1106) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1085) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:876) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:377) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:91) at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:56) at reactor.core.publisher.MonoLiftFuseable.subscribe(MonoLiftFuseable.java:55) at reactor.core.publisher.Mono.subscribe(Mono.java:3852) at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:123) at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to x.x.x.x:6379 at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:235) at io.lettuce.core.RedisClient.connect(RedisClient.java:204) at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.lambda$getConnection$1(StandaloneConnectionProvider.java:113) at java.base/java.util.Optional.orElseGet(Optional.java:369) at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.getConnection(StandaloneConnectionProvider.java:113) at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1104) ... 15 common frames omitted Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

[hanscrg]

@igorkatz,

In Springboot application, we need to solve the same issue which Netty HTTP Client used by Spring Security got connection disconnected in its pool by Load balancing Server. How could we implement your suggestion so that the setting will take effective in Netty HTTP client used by Spring Security?

In main function, we did like below, but our channel handler which doing ping does not take effective.

public static void main(String[] args) {        
           ServerBootstrap bootstrap = new ServerBootstrap();

        bootstrap = new ServerBootstrap ();
        bootstrap.group ( new NioEventLoopGroup (), new NioEventLoopGroup () );
        bootstrap.channel ( NioServerSocketChannel.class );
        bootstrap.option ( ChannelOption.SO_BACKLOG, 5 );
        bootstrap.option ( ChannelOption.SO_REUSEADDR, true );           
        bootstrap.childHandler(new MyChannelInitializer());

           SpringApplication.run(Mypplication.class, args); 

[violetagg]

@hanscrg ServerBootstrap is for the server not for the client.

[violetagg]

@hanscrg you might find this helpful https://github.com/spring-projects/spring-security/issues/4921

[hanscrg]

@violetagg,
Thanks you so much for the info. The sample project does have WebClientConfig, I guess, I can put @darekurlik's clientHttpConnector there to set ChannelOption.SO_KEEPALIVE to false to solve the Netty connection reset issue. I will try that shortly and report if that works.

Again thanks a lot for your input @violetagg.

[hanscrg]

@violetagg,

I created a new class in Spring Cloud Gateway application as below, try to disable Netty connection pool, but all seems to be not effective. Spring security OAuth Filter still use Netty connection pool. I still see logs showing r.n.resources.PooledConnectionProvider. What I am doing wrong? Is this WebClientConfig supposed to be effective globally, I mean application wise?

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.reactive.ClientHttpConnector;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;
import reactor.netty.tcp.TcpClient;

@Configuration
public class WebClientConfig {
@Bean
public WebClient webClient(final ClientHttpConnector clientHttpConnector) {
return WebClient.builder().clientConnector(clientHttpConnector).build();
}
@Bean
public ClientHttpConnector clientHttpConnector() {
return new ReactorClientHttpConnector(HttpClient.from(TcpClient.newConnection()));
}
}

[darekurlik]

@hanscrg
Hi! This is what reduced most of our problems:

import reactor.netty.resources.ConnectionProvider;

@Bean
public ClientHttpConnector clientHttpConnector() {
    return new ReactorClientHttpConnector(HttpClient.create(ConnectionProvider.newConnection()));
}

but I strongly encourage you too look into you infrastructure to see if there is any proxy/load balancer/gateway or anything else between your client and server you are trying to reach. Its important as any component between can terminate your connection due to its own timeout.

[hanscrg]

@darekurlik,

Thank you so much for the input. I tried you way to put the bean below in our Spring Cloud Gateway application. The Spring Security OAuth2 filter still use r.n.resources.PooledConnectionProvider.

@Bean
public ClientHttpConnector clientHttpConnector() {
return new ReactorClientHttpConnector(HttpClient.create(ConnectionProvider.newConnection()));
}

I checked the logs when application starts up, it shows below "Did not match ... onBeanCondition". is that the reason why the clientHttpConnector not used by Spring? But why?

ClientHttpConnectorConfiguration.ReactorNetty:
Did not match:
- @ConditionalOnBean (types: org.springframework.http.client.reactive.ClientHttpConnector; SearchStrategy: all) found beans of type 'org.springframework.http.client.reactive.ClientHttpConnector' clientHttpConnector (OnBeanCondition)
Matched:
- @ConditionalOnClass found required class 'reactor.netty.http.client.HttpClient' (OnClassCondition)

[hanscrg]

BTW, @darekurlik,, we do have load balancer between our Spring Cloud Gateway and Identify Provider. The logs do shows connection between our application and Identify Provider get reset by peer. But I guess there is not much load balancer can do? Because their side complain Netty Connection Pool does not handle load balancing situation well. That leaves only option for us to disable the Netty Pool.

[hanscrg]

This is spring application startup logs which is related to the clientHttpConnector,

2019-12-19 15:35:00.707 DEBUG 7 --- [main] o.s.b.f.s.DefaultListableBeanFactory: Creating shared instance of singleton bean 'clientHttpConnector'

2019-12-19 15:35:03.055 DEBUG 7 --- [main] o.s.b.f.s.DefaultListableBeanFactory: Autowiring by type from bean name 'clientConnectorCustomizer' via factory method to bean named 'clientHttpConnector'

ClientHttpConnectorAutoConfiguration#clientConnectorCustomizer matched: - @ConditionalOnBean (types: org.springframework.http.client.reactive.ClientHttpConnector; SearchStrategy: all) found bean 'clientHttpConnector' (OnBeanCondition)

ClientHttpConnectorConfiguration.ReactorNetty:
Did not match:
- @ConditionalOnBean (types: org.springframework.http.client.reactive.ClientHttpConnector; SearchStrategy: all) found beans of type 'org.springframework.http.client.reactive.ClientHttpConnector' clientHttpConnector (OnBeanCondition)
Matched:
- @ConditionalOnClass found required class 'reactor.netty.http.client.HttpClient' (OnClassCondition)

[hanscrg]

This the logs shows the Netty connection pool issue, even after I put the clientHttpConnector bean (listed above) to disable the pool,

2019-12-19 16:12:05.902 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider : [...] onStateChange(POST{uri=/oauth/token, connection=PooledConnection{channel=[id: 0x19541261, L:/xxx.xxx.xxx.xxx:27918 - R: xxx.xxx.xxx.xxx:443]}}, [request_sent])

2019-12-19 16:12:05.904 DEBUG 7 --- [reactor-http-epoll-4] r.netty.http.client.HttpClientConnect : [id: 0x19541261, L:/xxx.xxx.xxx.xxx:27918 - R: xxx.xxx.xxx.xxx:443] The connection observed an error, the request will be retried

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

2019-12-19 16:12:05.905 DEBUG 7 --- [reactor-http-epoll-4] r.n.resources.PooledConnectionProvider : [id: 0x19541261, L:/xxx.xxx.xxx.xxx:27918 - R: xxx.xxx.xxx.xxx:443] Channel acquired, now 2 active connections and 0 inactive connections

[darekurlik]

Sorry a got confused. You are own both server and the client or both?

• You are facing problems on the client or server side? You keep mentioning spring security so I guess its server side problem - correct? If so I think that maybe enabling IdleStateHandler for netty might do the trick. I once did something like that:

import io.netty.handler.timeout.IdleStateHandler;
import org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactory;
import org.springframework.boot.web.embedded.netty.NettyServerCustomizer;
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.stereotype.Component;
import reactor.netty.http.server.HttpServer;

@Component
public class MyNettyServerCustomizer implements WebServerFactoryCustomizer {

@Override
public void customize(final NettyReactiveWebServerFactory factory) {
    factory.addServerCustomizers(new IdleConnectionsCleanerCustomizer());
}

public static class IdleConnectionsCleanerCustomizer implements NettyServerCustomizer {
    @Override
    public HttpServer apply(final HttpServer httpServer) {
        return httpServer.tcpConfiguration(tcpServer -> tcpServer.bootstrap(
            serverBootstrap -> {
                // https://netty.io/4.0/api/io/netty/handler/timeout/IdleStateHandler.html
                serverBootstrap.childHandler(new IdleStateHandler())
                return serverBootstrap;
            }));
    }
}

}

• on the client side:
  @Bean
  public WebClient webClient(final ClientHttpConnector clientHttpConnector) {
  return WebClient.builder()
  .clientConnector(clientHttpConnector)
  .build();
  }

@Bean
public ClientHttpConnector clientHttpConnector() {
return new ReactorClientHttpConnector(HttpClient.create(ConnectionProvider.newConnection()));
}

So on the client side make sure that you are injecting correct client, as it might happen you have more then one in you context.

[hanscrg]

@darekurlik, Thank you so much for taking time to help. Yes, you are right, Our application is Spring Cloud Gateway which is running as server, it takes request and then forward to backend API. When request reach the server, the Spring Security OAuth2 filter, which is one of the filter of Gateway will kick in to do authentication/authorization which posts Auth code to IDP to get Auth token. So to do this http post, Spring Security uses reactor.netty.http.client.HttpClientConnect. That is where the problem happens. Connection in Netty Pool reset by peer after a while of idle when OAuth2 pick up the connection in the pool and trying to do HTTP post.

So the problem happens at Spring Cloud Gateway (Server) but it also use HTTP Client to send out HTTP request to IDP for Authentication/Authorization. Either HTTP Server or Client is part of Spring lib and we do not explicitly create it. So we can only customize it application wise.

I will try your solution now to see if it works. No matter what, thank you so much!

[hanscrg]

@darekurlik, here is what I try, for server, I have class below but I do no see any MyHandler logs come even I put idle time as (10, 10, 10)

@Component
public class MyNettyServerCustomizer implements WebServerFactoryCustomizer {
@Override
public void customize(final NettyReactiveWebServerFactory factory) {
factory.addServerCustomizers(new IdleConnectionsCleanerCustomizer());
}

public static class IdleConnectionsCleanerCustomizer implements NettyServerCustomizer {
    @Override
    public HttpServer apply(final HttpServer httpServer) {
        return httpServer.tcpConfiguration(tcpServer -> tcpServer.bootstrap(
                serverBootstrap -> {
                    // https://netty.io/4.0/api/io/netty/handler/timeout/IdleStateHandler.html
                    serverBootstrap
                    .childHandler(new IdleStateHandler(10, 10, 10))
                    .childHandler(new MyHandler());
                    return serverBootstrap;
                }));        
    }
}

}

@ChannelDuplexHandler.Sharable
public class MyHandler extends ChannelDuplexHandler {
private static final Logger LOG = LoggerFactory.getLogger(MyHandler.class);

     @Override
     public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
         LOG.debug("Idle event comes!");
         if (evt instanceof IdleStateEvent) {
             LOG.debug("Channle closed!");
            ctx.close();                
         }
     }
 }

For client, I have class below, but Netty clientHttpConnector still use its Pool Provider.
@Configuration
public class WebClientConfig {
private static final Logger LOG = LoggerFactory.getLogger(WebClientConfig.class);

@Bean
public WebClient webClient(final ClientHttpConnector clientHttpConnector) {
    return WebClient.builder()
            .clientConnector(clientHttpConnector)
            .build();
}
   @Bean
    public ClientHttpConnector clientHttpConnector() {
         return new ReactorClientHttpConnector(HttpClient.create(ConnectionProvider.newConnection()));
    }

}

I uploaded a simple GitHub project to demo those Netty customization does not work at the link below,
https://github.com/hanscrg/Sample-SpringCloudGateway-UAA

[hanscrg]

@darekurlik,
I uploaded the simple demo project to Github at the link below,
https://github.com/hanscrg/Sample-SpringCloudGateway-UAA

It is very simple, UAA is just Identity Provider. Build and run command are listed on the home page. The Gateway project only has 3 files. One file is for Gateway, other two files are trying to customize WebClient or ClientHTTPConnector used by Spring Security filter.

When you hit URL, http://localhost:8080/, the gateway will go through OAuth2 flow and finally show Test OK page. See the logs in Gateway project sample.log, you will find line

[reactor-http-nio-2] r.n.resources.PooledConnectionProvider : Creating new client pool [http] for localhost:8090

That means whatever http client or server customization does not take effective on the reactor.netty.http.client.HttpClientConnect as it still use Pool.

So anyway to disable that Netty Pool and let it create new connection every time?

[violetagg]

@hanscrg the discussion about WebClient with/without connection pool enabled will be covered in this issue https://github.com/spring-cloud/spring-cloud-gateway/issues/1493

[violetagg]

Most of the issues here were caused by a proxy/server that closes the connections on some timeout.
Because of this we are going to expose a configuration property for switching pool's lease strategy from FIFO to LIFO (FIFO is by default) #962

Using LIFO leasing strategy + max idle timeout will give you the behaviour below

• When the connection is acquired it will be the most recently used
  
  
  
  • If max idle timeout is reached this means that this connection will be closed and as this connection was the most recently used this means that all the rest (those that are not active) in the pool will also be closed because of the max idle timeout. A new connection will be created and used for the request.
  
  
  • If the connection is closed by the remote peer between acquire and the actual usage - Connection reset by peer will be received and we will retry the request. As this connection was the most recently used and it was closed by the remote peer this mean all the rest (those that are not active) in the pool also will be closed thus again a new connection will be used for the second attempt.
  
  

[violetagg]

The configuration will be available in 0.9.5 -> #962