Reactor-netty: java.io.IOException: Connection reset by peer

Created on 11 Jul 2018 · 63Comments · Source: reactor/reactor-netty

Actual behavior

@Bean
fun webClient(): WebClient {
    return WebClient
        .builder()
        .baseUrl(someUrl)
        .filter(logResponseStatus())
        .build()
}

```kotlin
@RestController
@RequestMapping("v1")
class OrderController(private val orderService: OrderService) {

@GetMapping("/orders/{storeId}")
fun orders(@RequestHeader("Authorization") auth: String,
    @PathVariable("storeId") storeId: String) = mono(Unconfined) {
    orderService.orders(storeId, auth).unpackResponse()
}

```kotlin
 //Repostiory class
 suspend fun deliveries(storeId: String, auth: String): List<ActiveOrdersRequestResponse>? {
        return webclient
            .get()
            .uri("v1/stores/$storeId/deliveries/$queryString")
            .header("Authorization", auth)
            .retrieve()
            .onStatus({ it == HttpStatus.FORBIDDEN || it == HttpStatus.UNAUTHORIZED }, { Mono.error(ValidationException(ErrorCode.AUTHENTICATION_ERROR, it.statusCode())) })
            .onStatus({ it == HttpStatus.NOT_FOUND }, { Mono.error(ValidationException(ErrorCode.STORE_NOT_FOUND, it.statusCode())) })
            .onStatus({ it.is5xxServerError }, { Mono.error(ValidationException(ErrorCode.AUTHENTICATION_EXCEPTION, it.statusCode())) })
            .bodyToFlux<ActiveOrdersRequestResponse>(ActiveOrdersRequestResponse::class.java)
            .collectList()
            .awaitSingle() //to prevent blocking the main thread
    }

For some reason executing this code throws randomly a

java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(FileDispatcherImpl.java)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1108)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
    at java.lang.Thread.run(Thread.java:748)

And I get a 500. This seems to happen mostly on my docker container than my main machine. I have tried numerous times to replicate it but no dice. There is no pattern whatsoever.

Load test result bombading my local server:

================================================================================
---- Global Information --------------------------------------------------------
> request count                                       1000 (OK=1000   KO=0     )
> min response time                                    400 (OK=400    KO=-     )
> max response time                                   1707 (OK=1707   KO=-     )
> mean response time                                   824 (OK=824    KO=-     )
> std deviation                                        221 (OK=221    KO=-     )
> response time 50th percentile                        802 (OK=802    KO=-     )
> response time 75th percentile                        975 (OK=975    KO=-     )
> response time 95th percentile                       1219 (OK=1219   KO=-     )
> response time 99th percentile                       1384 (OK=1384   KO=-     )
> mean requests/sec                                333.333 (OK=333.333 KO=-     )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                           496 ( 50%)
> 800 ms < t < 1200 ms                                 446 ( 45%)
> t > 1200 ms                                           58 (  6%)
> failed                                                 0 (  0%)
================================================================================

Result pointing to a remote server on 5 machines with 4gb and 2.5 VCPU

================================================================================
---- Global Information --------------------------------------------------------
> request count                                      10000 (OK=9965   KO=35    )
> min response time                                    105 (OK=1041   KO=105   )
> max response time                                  31431 (OK=31431  KO=28589 )
> mean response time                                  5610 (OK=5618   KO=3425  )
> std deviation                                       3475 (OK=3438   KO=8984  )
> response time 50th percentile                       5452 (OK=5463   KO=198   )
> response time 75th percentile                       8189 (OK=8194   KO=241   )
> response time 95th percentile                       9772 (OK=9769   KO=28406 )
> response time 99th percentile                      10771 (OK=10740  KO=28527 )
> mean requests/sec                                    250 (OK=249.125 KO=0.875 )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms                                             0 (  0%)
> 800 ms < t < 1200 ms                                  83 (  1%)
> t > 1200 ms                                         9882 ( 99%)
> failed                                                35 (  0%)
---- Errors --------------------------------------------------------------------
> status.find.in(200,304,201,202,203,204,205,206,207,208,209), b     35 (100.0%)
ut actually found 500
================================================================================

Steps to reproduce

Random

Reactor Netty version

io.projectreactor.ipc:reactor-netty:0.7.5.RELEASE

JVM version (e.g. `java -version`)

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (IcedTea 3.8.0) (Alpine 8.171.11-r0)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS version (e.g. `uname -a`)

Linux machine-name #1 SMP Sun Mar 11 19:39:47 UTC 2018 x86_64 Linux

typbug

Source

picaso

👍2

Most helpful comment

The issue seems to be still present in current versions. Using Spring Boot 2.1.0 -> reactor 3.2.2 -> reactor-netty 0.8.2 -> netty 4.1.29 we intermittently still get:

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
    at io.netty.channel.unix.FileDescriptor.readAddress(..)

Unfortunately I couldn't try the workarounds (disabling the pool and switching to nio) as suggested in one of the previous comments by @violetagg since the API changed for the 0.8.2 version and preferNative(boolean preferNative) seems not to exist anymore. Thus any hints on how to switch to nio and disable pooling using the 0.8.2 API would be appreciated.

rreimann on 29 Nov 2018

👍7

All 63 comments

@picaso Please try with the latest Reactor Netty version. We have a lot of fixes since 0.7.5.
Also Connection reset by peer means that the other party closed the connection.

violetagg on 11 Jul 2018

I Updated to reactor-netty version to the latest version in spring boot 2.0.3-RELEASE. The error message is just different now but the same Connection reset by peer error.

I was going to investigate further to see what was happening, but the team decided to use RestTemplate and though it was slower under load, no connection was reset. I am assuming something is wrong with reactor Webclient and handling slow connections. I will investigate further and check with Spring Boot.

picaso on 14 Jul 2018

@picaso Do you have any updated on this one? Thanks.

violetagg on 1 Aug 2018

Even I find similar issues. And I'm on the latest version of spring boot and reactor-netty - 0.7.8. Any updates on this?
What is the root cause of these issues?

balajeetm on 2 Aug 2018

I experienced the reset connections about 50 % of times when on Spring Boot 2.0.1 / Reactor-Netty 0.7.6. I upgraded to the latest Spring Boot 2.0.4 / Reactor-Netty 0.7.8 and haven't seen a single connection reset yet. I did not perform any load tests, though, just manual tests.

adammichalik on 3 Aug 2018

@adammichalik /Team,
Please keep the issue open. I'm using Spring boot version 2.0.3 with reactor-netty 0.7.8 but I still face this issue in production/live
I'll check again by upgrading and performing some load tests and revert soon.

Can I please get a little more details on why this issue was happening in the older versions? Since I do not know the root cause, I experience the issue intermittently and my tests may not capture the scenario. If the team can help me with the root cause, I can reproduce the scenario, upgrade and confirm everything is working.

PS : I'm running the application inside a docker.

balajeetm on 3 Aug 2018

Hi Team,

I did a performance test and still intermittently get the connection reset on reactor-netty 0.7.8. Would moving to spring boot 2.0.4 help? Can I know what the root cause is? I can try to contribute some fix if I'm able to reproduce the same.

balajeetm on 5 Aug 2018

Hi Team,

I still face this issue.
Any pointers to the real problem would be helpful.
Can someone help me with this?

balajeetm on 6 Aug 2018

Hi,

Are you able to provide some reproducible scenario that we can use?

Regards,
Violeta

violetagg on 6 Aug 2018

Hi Violeta,

I'm running a Spring Boot 2.0.3 web application with Servlet Container (spring web starter with tomcat). On receiving a request, the application makes a "POST" call to another service with a simple JSON body. The call is made using Reactor web client as velow

public <T extends Object> Mono<ResponseEntity<String>> apiCall(URI uri, HttpHeaders headers, T body) {
        RequestBodySpec request =
            webClient.method(HttpMethod.POST).uri(uri).headers(h -> updateHeaders(headers, h));
        if (body != null) {
          request.body(BodyInserters.fromObject(body));
        }
        return request
            .exchange()
            .flatMap(r -> r.toEntity(String.class))
            .publishOn(Schedulers.parallel());
      }

  private void updateHeaders(HttpHeaders headers, HttpHeaders h) {
    if (MapUtils.isNotEmpty(headers)) {
      h.addAll(headers);
    }
  }

Intermittently, this request keeps throwing a connection reset by client error as below:

Suppressed: io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
    at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

I'm unable to recreate the error when I ran the application natively on my VM (but I'm not sure this issue will never happen on a VM). I'm basically unable to zero in on the root cause

Regards,
Balajee

balajeetm on 6 Aug 2018

Is it related to this bug?
(https://github.com/netty/netty/issues/3539)
It seems to be closed
I'm on 4.1.25 version of netty

balajeetm on 6 Aug 2018

Can you try to reproduce with Reactor Netty 0.7.9.BUILD-SNAPSHOT - it uses Netty 4.1.28.Final, or just change Netty version to 4.1.28.Final?

violetagg on 6 Aug 2018

Will do it and revert shortly!

balajeetm on 6 Aug 2018

I've updated the spring boot version 2.0.4.RELEASE and the Netty version to 4.1.28.Final. Keeping an eye out for this issue. will keep you posted. Can I however know what is the root cause? This is a production impact, so if anyone can point me to the commit that fixed this, it would help. Without the RCA, I wouldn't be able to take this change to production.

Thanks,
Balajee

balajeetm on 7 Aug 2018

Further to updating the versions, the frequency of the resets have reduced. Will revert after more checks

balajeetm on 7 Aug 2018

@violetagg ,

Since you suggested to switch to nio transport.
I'm using the below to create a webclient. I'm forcing the epoll mechanism to level trigger (which is the default epoll mechanism in java nio). Is this what you meant by change to nio?

ExchangeStrategies strategies =
        ExchangeStrategies.builder()
            .codecs(
                c -> {
                  c.customCodecs().decoder(new Jackson2JsonDecoder(objectMapper));
                  c.customCodecs().encoder(new Jackson2JsonEncoder(objectMapper));
                })
            .build();
    ReactorClientHttpConnector connector =
        new ReactorClientHttpConnector(
            options ->
                options
                    .option(ChannelOption.SO_TIMEOUT, 3000)
                    .option(EpollChannelOption.EPOLL_MODE, EpollMode.LEVEL_TRIGGERED)
                    .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 3000));
    Builder builder = WebClient.builder().exchangeStrategies(strategies).clientConnector(connector);
    Webclient client = builder.build();

balajeetm on 7 Aug 2018

In options part call preferNative(false)

              /**
         * Set the preferred native option. Determine if epoll/kqueue should be used if available.
         *
         * @param preferNative Should the connector prefer native (epoll/kqueue) if available
         * @return {@code this}
         */
        public final BUILDER preferNative(boolean preferNative) {
            this.preferNative = preferNative;
            return get();
        }

Then when execute the scenario instead of reactor-http-client-epoll as thread name you should see reactor-http-nio

violetagg on 7 Aug 2018

@violetagg

I can now confirm that even after using the latest versions of spring boot and netty (4.1.28.RELEASE), the issue still exists and i still face connection reset issues.

Suppressed: io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

I'm now changing the calls to nio transport as suggested and trying the same. However, we still want to work in 'epoll' mode. I'll revert with the soak test and confirm the behavior.

Were there any changes made to the io.netty.channel.epoll package?
I see a similar issue raised 3 years ago and the issue we are facing is eerily similar
https://github.com/netty/netty/issues/3539

balajeetm on 7 Aug 2018

@violetagg

I've enabled a lot of debug logs to understand the rootcause. Connection reset by peer happens quite frequently with netty but the error is rarely bubbled up. However, everytime I see the error bubble up, I see the exception is always the below

USER_EVENT: SslCloseCompletionEvent(java.nio.channels.ClosedChannelException)

USER_EVENT: io.netty.channel.socket.ChannelInputShutdownReadComplete@7940e46a

Does that help, I'm trying to get more details from the prod environment

balajeetm on 8 Aug 2018

@violetagg

I can see, this is a sequence for failures.

When it tries to make an http call, it tries to acquire an existing channel from pool (DefaultPromise)

Acquiring existing channel from pool: DefaultPromise@2db14843(success: [id: 0x3cbc9cc3, L:/1xx.1xx.0.10:37230 - R:api-xyz.com/3x.2xx.1xx.41:443]) SimpleChannelPool{activeConnections=0}

The active Connections is always 0
It performs a ChannelInputShutdownReadComplete

[id: 0x8abf8526, L:/1xx.1xx.0.10:39918 ! R:api-xyz.com/3x.2xx.1xx.41:443] USER_EVENT: io.netty.channel.socket.ChannelInputShutdownReadComplete@18a456f8

It fails publishing the below event

[id: 0x8abf8526, L:/1xx.1xx.0.10:39918 ! R:api-xyz.com/3x.2xx.1xx.41:443] USER_EVENT: SslCloseCompletionEvent(java.nio.channels.ClosedChannelException

Logs a connection reset by peer

Is it possible that unhealthy connections are kept in the pool
I see some similar issues
https://github.com/reactor/reactor-netty/issues/177
https://github.com/netty/netty/issues/7262

I've currently disabled the pool in the options part.
Would there be any repercussions?

balajeetm on 8 Aug 2018

@violetagg , @smaldini , @simonbasle ,

I can confirm that by disabling the pool, the errors completely disappeared. I'm running some soak tests to confirm the same.
But the latency increases drastically when the pool is disabled. Does that give some pointers?
Just connecting the dots, I feel it possible that unhealthy connections are kept in the pool.

Can someone help me with this?
We just moved away from Vertx and got into the reactor stack. This issue would impede us from going ahead with reactor stack in production.
Thanks

balajeetm on 9 Aug 2018

Is disabling the pool the solution to this? Is anyone looking at it?

Regards,
Balajee

balajeetm on 13 Aug 2018

@balajeetm Yes I'm looking at it.

violetagg on 16 Aug 2018

I'm facing the same issue
io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

in springboot-2.0.3-release or updates on this or I have to switch to NIO?

jicui on 11 Sep 2018

@jicui ,
You can try switching to nio and check if that works.
That did not work me for me however. So I disabled the pool & the errors disappeared.

Cheers,
Balajee

balajeetm on 11 Sep 2018

While waiting a fix for this issue, I solved it this way :

For microservice with spring cloud gateway I used NIO instead of EPoll (by using builder.preferNative(false)) and I used reactor-netty 0.7.9 RELEASE
For microservice with spring webflux I used Undertow instead of Netty by adding dependency "org.springframework.boot:spring-boot-starter-undertow" just after "org.springframework.boot:spring-boot-starter-webflux" dependency in my build.gradle

bcoste on 14 Sep 2018

Can you try the latest Reactor Netty SNAPSHOT (0.7.10) and the latest Spring Framework SNAPSHOT?

violetagg on 8 Oct 2018

Fixed with #483. If you still see the issue reopen this ticket.

violetagg on 27 Oct 2018

@violetagg Does this mean, We don't need to disablePool or use "preferNative" anymore?

balajeetm on 27 Oct 2018

@balajeetm We think we were able to fix that issue. If you are able to test that version and provide a feedback it will be great. Thanks.

violetagg on 27 Oct 2018

@violetagg Will do so and revert soon. Thanks a ton

balajeetm on 27 Oct 2018

I'm having this issue with io.projectreactor.netty:reactor-netty:0.8.2.RELEASE.

java.io.IOException: Connection reset by peer
    at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:283)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:250)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:226)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:382)
    at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:234)
    at org.eclipse.jetty.io.NetworkTrafficSelectChannelEndPoint.fill(NetworkTrafficSelectChannelEndPoint.java:47)
    at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:331)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:243)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.base/java.lang.Thread.run(Thread.java:844)

I set up WireMock and do a bunch of requests with reactor.netty.http.client.HttpClient to that WireMock instance and if number of tests higher than let's say 10 then two or three tests fail and logs contain the error above and:

java.nio.channels.ClosedChannelException: null
    at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:502)
    at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353)
    at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:216)
    at org.eclipse.jetty.io.NetworkTrafficSelectChannelEndPoint.onClose(NetworkTrafficSelectChannelEndPoint.java:98)
    at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225)
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192)
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175)
    at org.eclipse.jetty.io.AbstractConnection.close(AbstractConnection.java:248)
    at org.eclipse.jetty.server.HttpChannelOverHttp.earlyEOF(HttpChannelOverHttp.java:234)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1551)
    at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:360)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:250)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.base/java.lang.Thread.run(Thread.java:844)

YuryYaroshevich on 30 Oct 2018

@YuryYaroshevich Please report this as a separate issue and provide more information (an example will be very helpful). I cannot see here any Reactor Netty/Netty stack.

violetagg on 30 Oct 2018

Issue disappeared after I added keepAlive(false) to HttpClient config:

HttpClient.create()
        .baseUrl("http://localhost:" + port)
        .keepAlive(false)     <---------------
        .headers(headers -> headers.add(HEADER_CONTENT_TYPE, CONTENT_TYPE_APPLICATION_JSON))

Inspired by https://groups.google.com/forum/#!topic/vertx/3o_DEwIK9dY

YuryYaroshevich on 31 Oct 2018

👍2

@YuryYaroshevich Even though it would be better to find the cause instead of disabling keepAlive.

violetagg on 31 Oct 2018

@violetagg what is the difference between

<dependency>
    <groupId>io.projectreactor.netty</groupId>
    <artifactId>reactor-netty</artifactId>
</dependency>

and

<dependency>
    <groupId>io.projectreactor.ipc</groupId>
    <artifactId>reactor-netty</artifactId>
</dependency>

Is IPC deprecated?

balajeetm on 9 Nov 2018

@balajeetm Reactor Netty 0.7.x has group Id io.projectreactor.ipc, for Reactor Netty 0.8.x we changed the group Id to io.projectreactor.netty

violetagg on 9 Nov 2018

@violetagg Thanks. I saw there are some changes to the ReactorClientHttpConnector constructor signatures in 0.8.x.
Will move everything forward.

balajeetm on 9 Nov 2018

The issue seems to be still present in current versions. Using Spring Boot 2.1.0 -> reactor 3.2.2 -> reactor-netty 0.8.2 -> netty 4.1.29 we intermittently still get:

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
    at io.netty.channel.unix.FileDescriptor.readAddress(..)

rreimann on 29 Nov 2018

👍7

@rreimann Please create a new issue with more details about your use case as Connection reset by peer can be caused by different reasons.

violetagg on 29 Nov 2018

@violetagg I see the same issue with 2.1.1.RELEASE when running as backend server from haproxy
2018-12-10 00:40:45,708 [ERROR] {reactor-http-epoll-1} reactor.netty.tcp.TcpServer - [id: 0xc97be89d, L:/xx.xxx.x.xxx:8080 - R:/96.117.7.140:37420] onUncaughtException(SimpleConnection{channel=[id: 0xc97be89d, L:/xx.xxx.x.xxx:8080 - R:/xx.xxx.x.xxx:37420]}) io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

Please let me know if you need more information?

binishchandy on 10 Dec 2018

@binishchandy Please create a new issue with a reproducible scenario

violetagg on 10 Dec 2018

@jicui ,
You can try switching to nio and check if that works.
That did not work me for me however. So I disabled the pool & the errors disappeared.

Cheers,
Balajee

hello，how to disable the pool, can you show some code please? thanks