Reactor-netty: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Operation timed out

Created on 15 Mar 2019 · 25Comments · Source: reactor/reactor-netty

Following link https://pivotal.io/security should be used to report security related issues

Gitter should be used for questions and discussions

Expected behavior

Actual behavior

Steps to reproduce

Reactor Netty version

JVM version (e.g. `java -version`)

OS version (e.g. `uname -a`)

Spring boot 2.1.3-RELEASE / Reactor-Netty 0.8.5-RELEASE

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Operation timed out

Used NIO 100% [reactor-http-nio-7]

Happens one or two times per hour (Docker). Almost impossible to reproduce locally (MacBook Pro).

Is that platform dependent error?

statuneed-investigation

Source

konstantinshevchuk

👍7

Most helpful comment

@meiyuisgithub @bearRunning 请求你们的这个问题最后解决了吗？

iblogc on 20 Mar 2021

👍2

All 25 comments

Do you have a reproduction use case or a scenario that looks like your problem?

smaldini on 21 Mar 2019

@smaldini
I can show only the WARN line in my log file

[id: 0xdea62af2, L:/172.19.0.5:53938 - R:comoany-dev.oktapreview.com/15.182.2.60:443] The connection observed an error

We use okta as authentication provider. Sometimes it tries to access okta host and this error appears.
Only reproducible in Docker. On local Mac OS never see it.
Because of this error sometimes we can't get an authorisation token.

This happens couple of time per hours. No more errors in the log file.

konstantinshevchuk on 21 Mar 2019

Me too. I'm afraid I don't have better reproducible steps.
It happens every time after an idle period.

Java 12
Spring Boot 2.1.4
Reactor-netty: 0.8.6
Ubuntu/Docker/Kubernetes/Azure Container Service

Warn message is with salesforce oauth2:

WARN 18 --- [or-http-epoll-1] r.netty.http.client.HttpClientConnect    : [id: 0x1bf5a0c4, L:/xxx - R:login.salesforce.com/85.222.153.195:443] The connection observed an error
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
        at io.netty.channel.unix.Errors.newIOException(Errors.java:122) ~[netty-transport-native-unix-common-4.1.34.Final.jar!/:4.1.34.Final]
        at io.netty.channel.unix.Errors.ioResult(Errors.java:146) ~[netty-transport-native-unix-common-4.1.34.Final.jar!/:4.1.34.Final]
        at io.netty.channel.unix.FileDescriptor.readAddress(FileDescriptor.java:177) ~[netty-transport-native-unix-common-4.1.34.Final.jar!/:4.1.34.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doReadBytes(AbstractEpollChannel.java:343) ~[netty-transport-native-epoll-4.1.34.Final-linux-x86_64.jar!/:4.1.34.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:782) ~[netty-transport-native-epoll-4.1.34.Final-linux-x86_64.jar!/:4.1.34.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) ~[netty-transport-native-epoll-4.1.34.Final-linux-x86_64.jar!/:4.1.34.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) ~[netty-transport-native-epoll-4.1.34.Final-linux-x86_64.jar!/:4.1.34.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.34.Final.jar!/:4.1.34.Final]
        at java.base/java.lang.Thread.run(Thread.java:835) ~[na:na]

thomasduchatelle on 10 May 2019

same here :(

Log

2019-07-18 17:16:36.273  INFO 14157 --- [or-http-epoll-3] c.o.o.m.web.controllers.ListController   : Request to create a LIST object
2019-07-18 17:16:36.311  WARN 14157 --- [or-http-epoll-2] reactor.netty.channel.FluxReceive        : [id: 0xce1b374d, L:/172.30.8.55:60716 - R:api.integration.***.com/***:443] An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
2019-07-18 17:16:36.319  INFO 14157 --- [or-http-epoll-3] reactor.netty.http.server.AccessLog      : 127.0.0.1 - - [18/Jul/2019:17:16:36] "POST /api/lists HTTP/1.0" 200 0 8090 50 ms

It results into my service replies with HTTP 200 + empty response body

Configuration

SpringBoot: v2.1.6.RELEASE
Reactor-Netty: 0.8.9.RELEASE (reactor-netty-0.8.9.RELEASE.jar)
Java:

$ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+7-post-Debian-5)
OpenJDK 64-Bit Server VM (build 11.0.3+7-post-Debian-5, mixed mode, sharing)

System:

$ uname -a
Linux integration 4.19.0-5-cloud-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

darklynx on 18 Jul 2019

@dkulig this issues is for Connection timed out not Connection reset by peer. May be you need this one #564

violetagg on 28 Jan 2020

My apologizes @violetagg - wrong issue

dkulig on 28 Jan 2020

Hi, @violetagg. The keepAlive parameter is currently enabled on my Netty server. Will the problem be triggered after receiving a short TCP connection?

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

asdf2014 on 18 Mar 2020

@asdf2014 This issue is for operation timed out. Please open a new issue and describe your scenario as well as the Reactor Netty version.
Thanks

violetagg on 18 Mar 2020

@violetagg Sure.

asdf2014 on 18 Mar 2020

@konstantinshevchuk did you get any solution?

SudBisht on 3 May 2020

👍1

请教下大佬，gateway代理前端工程，页面暂停操作一段时间就会出现io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer报错：

2020-06-09 09:42:05.828 [reactor-http-epoll-1] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [23b53156-170] 500 Server Error for HTTP GET "/gbd/index.html"
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/gbd/index.html" [ExceptionHandlingWebHandler]
Stack trace:
2020-06-09 09:42:06.750 [reactor-http-epoll-2] WARN r.n.http.client.HttpClientConnect - [id: 0x18f3cd56, L:/172.17.0.104:37314 - R:192.168.42.50/192.168.42.50:80] The connection observed an error
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
2020-06-09 09:42:06.752 [reactor-http-epoll-2] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [23b53156-171] 500 Server Error for HTTP GET "/gbd/index.html"
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/gbd/index.html" [ExceptionHandlingWebHandler]

meiyuisgithub on 9 Jun 2020

请教下大佬，网关代理前端工程，页面暂停操作将会出现io.netty.channel.unix.Errors $ NativeIoException：readAddress（..）失败：对等连接重置

2020-06-09 09：42：05.828 [reactor-http-epoll-1]错误oswsaHttpWebHandlerAdapter-[23b53156-170] 500 HTTP GET服务器错误“ /gbd/index.html”
io.netty.channel.unix.Errors $ NativeIoException：readAddress（..）失败：对等方重置连接
抑制：reactor.core.publisher.FluxOnAssembly $ OnAssemblyException：
在以下站点发现错误：
| _检查点⇢org.springframework.web.cors。
react.CorsWebFilter [DefaultWebFilterChain] | _检查点⇢org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
| _检查点⇢HTTP GET“ /gbd/index.html” [ExceptionHandlingWebHandler]
堆栈跟踪：
2020-06-09 09：42：06.750 [reactor-http-epoll-2] WARN rnhttp.client.HttpClientConnect-[id：0x18f3cd56，L：/172.17.0.104：37314-R：192.168.42.50/192.168.42.50： 80]连接观察到错误
io.netty.channel.unix.Errors $ NativeIoException：readAddress（..）失败：对等节点
2020-06-09 09：42：06.752 重置连接[reactor-http-epoll-2]错误oswsaHttpWebHandlerAdapter-[23b53156-171] HTTP GET“ /gbd/index.html”
io.netty.channel.unix.Errors $ NativeIoException的500服务器错误：readAddress（..）失败：对等方重置连接被
禁止：reactor.core。 Publisher.FluxOnAssembly $ OnAssemblyException：
在以下站点发现错误：
| _检查点⇢org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
| _检查点⇢org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
| _检查点⇢HTTP GET“ /gbd/index.html” [ExceptionHandlingWebHandler]

同问你的问题解决了么

bearRunning on 17 Jun 2020

@konstantinshevchuk did you get any solution? Thanks

sivapolisetty on 20 Aug 2020

@sivapolisetty Do you experience this with the latest release?

violetagg on 20 Aug 2020

Yes @violetagg .
Spring boot 2.3.2-RELEASE / Reactor-Netty 0.9.10-RELEASE.
Docker/Openshift environment. Getting "logger_name":"reactor.netty.http.client.HttpClientConnect","thread_name":"reactor-http-epoll-4","level":"WARN","level_value":30000,"stack_trace":"io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Operation timed out\n"

Not able to reproduce on Mac

sivapolisetty on 20 Aug 2020

@sivapolisetty Is it possible to take a tcpdump?

violetagg on 21 Aug 2020

@violetagg experiencing same java.io.IOException: Operation timed out.
Attached the tcpdump log. Please let me know if more details are needed.
Also trying to understand why this happens. Does this mean that underlying socket connection is closed but channel is trying to retransmit on the same socket ?
Screen Shot 2020-09-16 at 10 05 28 AM

Reactor Netty Log

r.netty.http.client.HttpClientConnect    : [id: 0x467e6fd4, L:/<host_ip>:57609 - R:<dest_host>/<dest_ip>:443] The connection observed an error

reactor-netty : 0.9.11.RELEASE

kaladhar-mummadi on 16 Sep 2020

any update on this issue ??

darshan4545 on 30 Oct 2020

I am facing the same issue, we are using spring boot 2.2.0 and below java version-

openjdk 11.0.6 2020-01-14 LTS
OpenJDK Runtime Environment Zulu11.37+17-CA (build 11.0.6+10-LTS)
OpenJDK 64-Bit Server VM Zulu11.37+17-CA (build 11.0.6+10-LTS, mixed mode)

reactor.core.Exceptions$ReactiveException: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
at reactor.core.Exceptions.propagate(Exceptions.java:336) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:91) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
at reactor.core.publisher.Mono.block(Mono.java:1663) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]

chhaviLife360 on 1 Nov 2020

@chhaviLife360 Your exception is Connection reset by peer and the issue is for Operation timed out.
Please update to the latest Spring 2.2.x as 2.2.0 is more than a year old.

violetagg on 3 Nov 2020

@violetagg using latest version and still getting this error

darshan4545 on 4 Nov 2020

@darshan4545 Any chance for a reproducible example?

violetagg on 4 Nov 2020

https://github.com/spring-cloud/spring-cloud-gateway/tree/master/spring-cloud-gateway-sample

using same above code

darshan4545 on 30 Nov 2020

Similar problem here. We have two servers calling a load balancer VIP and they both get a "connection timeout" error about once a day. We have noticed that this happens at about the same time. Not sure it's the same problem.
We are on springboot 2.4.2.

elibus on 26 Jan 2021

@meiyuisgithub @bearRunning 请求你们的这个问题最后解决了吗？

iblogc on 20 Mar 2021

👍2

Was this page helpful?

0 / 5 - 0 ratings

Related issues

How to disable Expect: 100-continue default processing by underlying netty

hmble2 · 3Comments

`Broken pipe` is not throwed with netty-transport-native-epoll dependency

flisky · 6Comments

Sometimes deadlock is occurred due to blocking DNS resolver

be-hase · 6Comments

Add option for changing the max URI tags

violetagg · 8Comments

Connection not released in ConnectionPool when PrematureCloseException occurs

rreynaud · 3Comments