Spring-cloud-gateway: RetryGatewayFilterFactory causes OutOfDirectMemoryError

Created on 27 Jul 2020 · 25Comments · Source: spring-cloud/spring-cloud-gateway

Describe the bug

Version

spring cloud : Hoxton.RELEASE
spring cloud starter gateway : 2.2.2.RELEASE
spring boot starter : 2.2.5.RELEASE
netty : 4.1.45.Final
reactor-netty: 0.9.5.RELEASE

Usage

spring.cloud.gateway.default-filters[0].name=Retry
spring.cloud.gateway.default-filters[0].args.retries=1
spring.cloud.gateway.default-filters[0].args.series=SERVER_ERROR
spring.cloud.gateway.default-filters[0].args.statuses=NOT_FOUND
spring.cloud.gateway.default-filters[0].args.methods = GET,PUT

JVM

-server -Xms64m -Xmx64m -Xmn64m -XX:MaxMetaspaceSize=256m -Xss256k -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly -XX:AutoBoxCacheMax=20000 -XX:-OmitStackTraceInFastThrow -Dio.netty.leakDetectionLevel=advanced

Describe

When the back-end service processes the related business and returns 404 or 5xx, the gateway will retry, and the OutOfDirectMemoryError will appear after a certain number of requests. But I did not find where the RetryGatewayFilterFactory did not release the DirectMemory.

Exception

io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2013265927, max: 2021654528)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:762)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1089)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2013265927, max: 2021654528)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:726)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:681)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:215)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:93)
at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:762)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1089)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)

bug

Source

Burt-L

🎉1

All 25 comments

Can you please try Hoxton.SR6?

ryanjbaxter on 27 Jul 2020

Hoxton.SR6 version also has this problem.

Vsersion

spring cloud : Hoxton.SR6
spring cloud starter gateway : 2.2.3.RELEASE
spring boot starter : 2.2.5.RELEASE

Burt-L on 28 Jul 2020

020-08-05 10:33:38.294 [reactor-http-nio-1] [ERROR] i.n.util.ResourceLeakDetector.reportTracedLeak:320 [] - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:

1:

io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

2:

io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:850)
io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:592)
io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218)
io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

3:

io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:557)
io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:193)
io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

4:

Hint: 'reactor.left.httpCodec' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

5:

Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

6:

io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
: 1 leak records were discarded because they were duplicates
: 11 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

Burt-L on 5 Aug 2020

I think I found the problem.
io.netty.handler.codec.http.HttpObjectDecoder#decode L336

HttpContent chunk = new DefaultHttpContent(buffer.readRetainedSlice(toRead));

If retries = 1, this line of code will be called twice, so it will create two new bytebuf objects, but only release the last bytebuf object created.

Burt-L on 5 Aug 2020

The problem is reactor.netty.channel .FluxReceive#drainReceiver.

Burt-L on 5 Aug 2020

The problem is reactor.netty.channel .FluxReceive#drainReceiver.

Do you fix it？I meet same question。

BigArtist on 16 Sep 2020

hello,who fix this question?

ListenToTheVoiceOfTheWillow on 18 Sep 2020

We don't maintain reactor-netty here. @violetagg is this familiar to you?

spencergibb on 21 Sep 2020

The problem is reactor.netty.channel .FluxReceive#drainReceiver.

@Burt-L Is that reported as Reactor Netty issue?

violetagg on 21 Sep 2020

Vsersion
spring cloud : Hoxton.SR8
gateway : 2.2.5.RELEASE
spring boot starter : 2.3.4.RELEASE

we do not configure Netty version ,use gateway default Netty.
And our product environment program work 20 hours, the issue will appear.
We try some version ,but not effective. we used spring boot 2.3.3 spring cloud Hoxton.SR6 spring-gateway 2.2.2. and change some version . but all not effective.

ListenToTheVoiceOfTheWillow on 22 Sep 2020

09:10:51.198 car-trace-logging [reactor-http-epoll-4] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [2dd40a73] Error [reactor.netty.ReactorNetty$InternalNettyException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023410183, max: 1029177344)] for HTTP POST "/example", but ServerHttpResponse already committed (200 OK)
09:10:51.199 car-trace-logging [http-nio-9000-exec-9] ERROR o.a.c.c.C.[.[.[.[httpHandlerServlet] - Servlet.service() for servlet [httpHandlerServlet] threw exception
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023410183, max: 1029177344)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:754)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:709)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:755)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:731)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:247)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:356)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:777)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
09:10:51.200 car-trace-logging [http-nio-9000-exec-9] ERROR o.a.c.c.C.[.[.[.[httpHandlerServlet] - Servlet.service() for servlet [httpHandlerServlet] in context with path [] threw exception [Failed to create response content] with root cause
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023410183, max: 1029177344)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:754)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:709)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:755)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:731)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:247)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:356)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:777)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

ListenToTheVoiceOfTheWillow on 23 Sep 2020

I'm looking at this one

violetagg on 23 Sep 2020

Ok,thanks ,we need solve this question quickly.

ListenToTheVoiceOfTheWillow on 23 Sep 2020

we restart our program everyday to avoid this question.

ListenToTheVoiceOfTheWillow on 23 Sep 2020

@spencergibb I think the following happens (I made one very simple test with RetryGatewayFilterFactory):
A response from target peer was received and it was 404 (in my case), but there is no subscription for the body and it stays in the Reactor Netty buffer

2020-09-23 16:23:34.724 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x411c95ef, L:/XXX - R:YYY]}}, [response_received])
2020-09-23 16:23:34.732 DEBUG 98353 --- [ctor-http-nio-3] r.n.http.client.HttpClientOperations     : [id: 0x411c95ef, L:/XXX - R:YYY] Received last HTTP packet
2020-09-23 16:23:34.732 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x411c95ef, L:/XXX - R:YYY]}}, [response_completed])
2020-09-23 16:23:34.732 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x411c95ef, L:/XXX - R:YYY]}}, [disconnecting])
2020-09-23 16:23:34.733 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] Releasing channel
2020-09-23 16:23:34.734 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] Channel cleaned, now 1 active connections and 1 inactive connections

When the request is retried you can see that the response body is consumed the second time:

2020-09-23 16:23:34.977 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x3e6609dd, L:/XXX - R:YYY]}}, [response_received])
2020-09-23 16:23:34.980 DEBUG 98353 --- [ctor-http-nio-3] reactor.netty.channel.FluxReceive        : [id: 0x3e6609dd, L:/XXX - R:YYY] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2020-09-23 16:23:34.980 DEBUG 98353 --- [ctor-http-nio-3] r.n.http.client.HttpClientOperations     : [id: 0x3e6609dd, L:/XXX - R:YYY] Received last HTTP packet
2020-09-23 16:23:34.985 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x3e6609dd, L:/XXX - R:YYY]}}, [response_completed])
2020-09-23 16:23:34.986 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x3e6609dd, L:/XXX - R:YYY]}}, [disconnecting])
2020-09-23 16:23:34.986 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] Releasing channel
2020-09-23 16:23:34.986 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] Channel cleaned, now 0 active connections and 2 inactive connections

So there are two solutions

Spring Gateway closes the connection when a retry is needed thus Reactor Netty buffer will be cleaned
Spring Gateway drains the response body from the target peer and thus keeps and reuses the connection

violetagg on 23 Sep 2020

so,when solve this problem? Our project have to restart program everyday.

peichunting on 24 Sep 2020

I'll get with @violetagg and work on a fix.

spencergibb on 24 Sep 2020

👍1

@peichunting @Burt-L @ListenToTheVoiceOfTheWillow @BigArtist
Are any of you willing to try Hoxton.BUILD-SNAPSHOT with an attempted fix I just submitted?
5e0baf5

spencergibb on 25 Sep 2020

👍1

@peichunting @Burt-L @ListenToTheVoiceOfTheWillow @BigArtist
Are any of you willing to try Hoxton.BUILD-SNAPSHOT with an attempted fix I just submitted?
5e0baf5

We use maven cannot download this version .
Could not find artifact org.springframework.cloud:spring-cloud-dependencies:pom:Hoxton.BUILD-SNAPSHOT in spring-milestones (https://repo.spring.io/libs-milestone)

peichunting on 26 Sep 2020

It is a snapshot located in https://repo.spring.io/snapshot

spencergibb on 26 Sep 2020

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

spring-projects-issues on 5 Oct 2020

@peichunting @Burt-L @ListenToTheVoiceOfTheWillow @BigArtist
Are any of you willing to try Hoxton.BUILD-SNAPSHOT with an attempted fix I just submitted?
5e0baf5

I've tested it and it's probably released correctly. Thank you.

Burt-L on 10 Oct 2020

2020-09-23 16:23:34.724 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x411c95ef, L:/XXX - R:YYY]}}, [response_received])
2020-09-23 16:23:34.732 DEBUG 98353 --- [ctor-http-nio-3] r.n.http.client.HttpClientOperations     : [id: 0x411c95ef, L:/XXX - R:YYY] Received last HTTP packet
2020-09-23 16:23:34.732 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x411c95ef, L:/XXX - R:YYY]}}, [response_completed])
2020-09-23 16:23:34.732 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x411c95ef, L:/XXX - R:YYY]}}, [disconnecting])
2020-09-23 16:23:34.733 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] Releasing channel
2020-09-23 16:23:34.734 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x411c95ef, L:/XXX - R:YYY] Channel cleaned, now 1 active connections and 1 inactive connections

When the request is retried you can see that the response body is consumed the second time:

2020-09-23 16:23:34.977 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x3e6609dd, L:/XXX - R:YYY]}}, [response_received])
2020-09-23 16:23:34.980 DEBUG 98353 --- [ctor-http-nio-3] reactor.netty.channel.FluxReceive        : [id: 0x3e6609dd, L:/XXX - R:YYY] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2020-09-23 16:23:34.980 DEBUG 98353 --- [ctor-http-nio-3] r.n.http.client.HttpClientOperations     : [id: 0x3e6609dd, L:/XXX - R:YYY] Received last HTTP packet
2020-09-23 16:23:34.985 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x3e6609dd, L:/XXX - R:YYY]}}, [response_completed])
2020-09-23 16:23:34.986 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] onStateChange(GET{uri=/test, connection=PooledConnection{channel=[id: 0x3e6609dd, L:/XXX - R:YYY]}}, [disconnecting])
2020-09-23 16:23:34.986 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] Releasing channel
2020-09-23 16:23:34.986 DEBUG 98353 --- [ctor-http-nio-3] r.n.resources.PooledConnectionProvider   : [id: 0x3e6609dd, L:/XXX - R:YYY] Channel cleaned, now 0 active connections and 2 inactive connections

So there are two solutions

Spring Gateway closes the connection when a retry is needed thus Reactor Netty buffer will be cleaned
Spring Gateway drains the response body from the target peer and thus keeps and reuses the connection

@violetagg Can you please tell us how we can disable netty retry. Or how we can actually implement this solution what you gave here.
I am using :
Spring boot: 2.3.4
Hoxton.SR8