Spring-cloud-gateway: Reproducible Leak under load when request is timed out.

Created on 31 Jan 2019  路  32Comments  路  Source: spring-cloud/spring-cloud-gateway

This is being tested under Greenwhich snapshot and Californium Snapshot.

I am testing this on PCF using SSL. Under load the gateway leaks memory when the response-timeout setting is used and a request surpasses the timeout.

The setup is as follows.

  1. Multiple get and posts are being made to the gateway via Gatling.
  2. The gateway is setup to terminate requests that take longer than 7500 milliseconds (setting response-timeout). One request takes 7500 milliseconds so it will be terminated by the gateway.

After some amount of time the gateway will eventually leak memory.

Everything you need is available in this repot

The Gatling script uses version 2.3.1. You can download it here

You will have to tweak this to enable SSL or just deploy to PCF where SSL is turned on by default. In addition you will just need to configure the url in the application.yml along with the url in the Gatling Script.

bug
Source
picture dave-fl

All 32 comments

@violetagg @smaldini This is after testing https://github.com/reactor/reactor-netty/issues/576. Started seeing the leaks only after I was doing posts and gets with a combination of long request intentionally being timed out by the gateway.

@violetagg had previously put in an alternative timeout approach into NettyRoutingFilter but it was removed because it caused some similar 504 issues on all requests (even if they were short). Given that those 504 issues are gone (reactor/reactor-netty#576), perhaps this approach should be tried again.

dave-fl picture dave-fl on 31 Jan 2019

Here is the stack trace.

2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT 2019-01-30 23:50:03.796 ERROR 14 --- [or-http-epoll-7] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT #1:
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT Created at:
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.79-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT 2019-01-30 23:50:03.806 ERROR 14 --- [or-http-epoll-3] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #1:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #2:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #3:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:601)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #4:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:581)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #5:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:850)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:208)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #6:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:566)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:202)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #7:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     Hint: 'reactor.left.httpCodec' will handle the message from this point.
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #8:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT #9:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT Created at:
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:339)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1912)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1296)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-01-30T18:50:03.80-0500 [APP/PROC/WEB/0] OUT : 3 leak records were discarded because they were duplicates
dave-fl picture dave-fl on 31 Jan 2019

Seeing this too.

2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT 2019-01-31 00:51:07.882 ERROR 14 --- [or-http-epoll-5] reactor.core.publisher.Operators         : Operator called default onErrorDropped
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT java.lang.UnsupportedOperationException: null
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at org.springframework.http.ReadOnlyHttpHeaders.remove(ReadOnlyHttpHeaders.java:107) ~[spring-web-5.1.4.RELEASE.jar!/:5.1.4.RELEASE]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at org.springframework.cloud.gateway.filter.NettyRoutingFilter.lambda$filter$3(NettyRoutingFilter.java:151) ~[spring-cloud-gateway-core-2.1.1.BUILD-SNAPSHOT.jar!/:2.1.1.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.http.client.HttpClientFinalizer.lambda$responseConnection$5(HttpClientFinalizer.java:85) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) [reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:147) [reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.http.client.HttpClientConnect$HttpObserver.onStateChange(HttpClientConnect.java:397) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:500) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:442) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:492) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436) [netty-handler-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215) [netty-handler-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249) [netty-handler-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) [netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) [netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) [netty-common-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT 2019-01-31 00:51:07.886  WARN 14 --- [or-http-epoll-5] i.n.c.AbstractChannelHandlerContext      : An exception 'reactor.core.Exceptions$BubblingException: java.lang.UnsupportedOperationException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT reactor.core.Exceptions$BubblingException: java.lang.UnsupportedOperationException
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.Exceptions.bubble(Exceptions.java:154) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.Operators.onErrorDropped(Operators.java:512) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onError(FluxTimeout.java:202) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:157) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:147) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.http.client.HttpClientConnect$HttpObserver.onStateChange(HttpClientConnect.java:397) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:500) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:442) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:492) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436) [netty-handler-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215) [netty-handler-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249) [netty-handler-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) [netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) [netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) [netty-common-4.1.33.Final.jar!/:4.1.33.Final]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT Caused by: java.lang.UnsupportedOperationException: null
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at org.springframework.http.ReadOnlyHttpHeaders.remove(ReadOnlyHttpHeaders.java:107) ~[spring-web-5.1.4.RELEASE.jar!/:5.1.4.RELEASE]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at org.springframework.cloud.gateway.filter.NettyRoutingFilter.lambda$filter$3(NettyRoutingFilter.java:151) ~[spring-cloud-gateway-core-2.1.1.BUILD-SNAPSHOT.jar!/:2.1.1.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.http.client.HttpClientFinalizer.lambda$responseConnection$5(HttpClientFinalizer.java:85) ~[reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153) ~[reactor-core-3.2.6.BUILD-SNAPSHOT.jar!/:3.2.6.BUILD-SNAPSHOT]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     ... 37 common frames omitted
dave-fl picture dave-fl on 31 Jan 2019

I should point out that this doesn't seem to be happening a lot. E.g. After 3.5 million requests there are 2 leak reports and 1 unsupported operation trace. I have a much more real world test that I will run tomorrow.

dave-fl picture dave-fl on 31 Jan 2019

About that bit:

2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT java.lang.UnsupportedOperationException: null
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at org.springframework.http.ReadOnlyHttpHeaders.remove(ReadOnlyHttpHeaders.java:107) ~[spring-web-5.1.4.RELEASE.jar!/:5.1.4.RELEASE]
2019-01-30T19:51:07.88-0500 [APP/PROC/WEB/0] OUT     at org.springframework.cloud.gateway.filter.NettyRoutingFilter.lambda$filter$3(NettyRoutingFilter.java:151) ~[spring-cloud-gateway-core-2.1.1.BUILD-SNAPSHOT.jar!/:2.1.1.BUILD-SNAPSHOT]

This means that something tried to modify HTTP headers after the HTTP message has been marked as committed - at that point, they're read-only.

It seems that Spring Cloud Gateway is trying to modify a headers map after it's been committed . I'm not sure I get which response is this (the client or the server one), but maybe this piece of code should be done before any writing to the HTTP message body happens, or if this can't be avoided, at least protect that line with a isCommitted() check.

bclozel picture bclozel on 31 Jan 2019

Based on my investigations - we have a memory leak as a result of this java.lang.UnsupportedOperationException
https://github.com/reactor/reactor-netty/issues/595

violetagg picture violetagg on 31 Jan 2019

The memory leak is fixed and the fix is available with Reactor Netty 0.8.5.BUILD-SNAPSHOT
The other issue - java.lang.UnsupportedOperationException still needs to be investigated (spring gateway, spring framework), at least the suggested check above (isCommitted()) may need to be applied.

violetagg picture violetagg on 1 Feb 2019

@violetagg Still happening.

The leak stack seems to occur without the java.lang.UnsupportedOperationException. For some reason it seems so far that it only happens once (for this test) and then doesn't happen again. I'll begin a more comprehensive test today.

2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT 2019-02-01 15:00:16.413 ERROR 20 --- [or-http-epoll-6] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #1:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #2:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #3:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:601)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #4:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:581)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #5:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:850)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:208)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #6:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:566)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:202)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #7:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     Hint: 'reactor.left.httpCodec' will handle the message from this point.
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #8:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #9:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:339)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1912)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1296)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT : 3 leak records were discarded because they were duplicates
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT 2019-02-01 15:00:16.415 ERROR 20 --- [or-http-epoll-6] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT #1:
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.41-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T10:00:16.42-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
dave-fl picture dave-fl on 1 Feb 2019

@dave-fl are there any other exceptions?

violetagg picture violetagg on 1 Feb 2019

@violetagg for this test so far no (its very basic).

dave-fl picture dave-fl on 1 Feb 2019

Started a more real world run. In the beginning of the run same leak error. Then again 30 minutes later. Will allow it to run for at least 24 hours.

2019-02-01T15:51:43.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #8:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #9:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:339)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1912)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1296)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT : 3 leak records were discarded because they were duplicates
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT 2019-02-01 21:20:12.536 ERROR 15 --- [or-http-epoll-2] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #1:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #2:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #3:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:601)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT 2019-02-01 21:21:49.347 ERROR 15 --- [or-http-epoll-8] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT #1:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:94)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:208)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:321)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:319)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:520)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT #2:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:177)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:205)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:321)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:319)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:520)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT #3:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
dave-fl picture dave-fl on 1 Feb 2019

Ok, this is interesting.

30 minutes later, we are getting a memory error again. Either a timer somewhere or that is what it takes for garbage collection to start. This stack seems to have something new drainReceiver.

2019-02-01T15:51:43.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #8:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #9:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1446)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:339)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1912)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1296)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT : 3 leak records were discarded because they were duplicates
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT 2019-02-01 21:20:12.536 ERROR 15 --- [or-http-epoll-2] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #1:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #2:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT #3:
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:801)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:601)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:20:12.54-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT 2019-02-01 21:21:49.347 ERROR 15 --- [or-http-epoll-8] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT #1:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:94)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:208)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:321)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:319)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:520)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT #2:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:177)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:205)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:321)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:319)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:520)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT #3:
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.34-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1215)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1249)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T16:21:49.35-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
dave-fl picture dave-fl on 1 Feb 2019

One thing that is definitely different.

The synthetic examples seems to leak in the beginning. And doesn't seem to happen again.

The real world test seems to leak non stop and also shows the following on each POST.

Regarding the real world test, some of the test Agents are terminating abruptly (need to fix script) and it seems like when these take place a leak is happening. An abnormal termination of a midflight connection POST or GET seems like something that could happen a lot but I am not sure how to give a test case that emulates this behavior.

ERROR 14 --- [or-http-epoll-1] o.s.w.s.adapter.HttpWebHandlerAdapter    : [51917f27] Error [reactor.netty.channel.AbortedException: Connection has been closed] for HTTP POST "/services/myservice", but ServerHttpResponse already committed (200 OK)
dave-fl picture dave-fl on 2 Feb 2019

Here's another new one. Killed the back end (did not restart the gateway). Started tests again.

2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT 2019-02-02 02:31:06.957  WARN 16 --- [or-http-epoll-5] r.netty.http.client.HttpClientConnect    : [id: 0x71bde98b, L:/IP - R:HOST/IP:443] The connection observed an error
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT java.lang.IllegalStateException: Only one connection receive subscriber allowed.
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:271) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:124) [reactor-netty-0.8.5.BUILD-SNAPSHOT.jar!/:0.8.5.BUILD-SNAPSHOT]
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) ~[netty-transport-native-epoll-4.1.33.Final-linux-x86_64.jar!/:4.1.33.Final]
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar!/:4.1.33.Final]
2019-02-01T21:31:06.95-0500 [APP/PROC/WEB/0] OUT     at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT 2019-02-02 02:31:07.931 ERROR 16 --- [or-http-epoll-4] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT Recent access records:
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT Created at:
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:322)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:166)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:152)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.MonoCollect.subscribe(MonoCollect.java:58)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.Mono.subscribe(Mono.java:3695)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:320)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.netty.http.client.HttpClientConnect$HttpObserver.onStateChange(HttpClientConnect.java:411)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:552)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-02-01T21:31:07.93-0500 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
dave-fl picture dave-fl on 2 Feb 2019

@dave-fl Test with the latest Californium-SR6

violetagg picture violetagg on 26 Mar 2019

Still seeing leaks on Netty 4.1.34, SR6, Greenwich SR1 and Boot 2.1.3. This is my main live run, which is different from the Github example. More real world.

2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT Recent access records:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #1:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT Created at:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT 2019-03-26 20:27:04.289 ERROR 14 --- [or-http-epoll-4] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT Recent access records:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #1:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #2:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #3:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.left.httpCodec' will handle the message from this point.
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #4:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #5:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT Created at:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:334)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2120)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1325)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT 2019-03-26 20:27:04.294 ERROR 14 --- [or-http-epoll-6] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT Recent access records:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT #1:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:24)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT Created at:
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.29-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT 2019-03-26 20:27:04.313 ERROR 14 --- [or-http-epoll-2] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT Recent access records:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #1:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #2:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #3:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:592)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #4:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:572)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #5:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:842)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:199)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #6:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:557)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:193)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #7:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.left.httpCodec' will handle the message from this point.
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #8:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT #9:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT Created at:
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:334)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2120)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1325)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-03-26T16:27:04.31-0400 [APP/PROC/WEB/0] OUT : 18 leak records were discarded because they were duplicates
dave-fl picture dave-fl on 26 Mar 2019

So normally when I test I do the contrived test which I believe is fixed and I do a real world test.

The real world test failed with leaks again. At first I thought maybe this was related to the timeout so I tried removing my usage of adding the read timeout handlers on the webclient and replacing them without reactive timeouts. This did not work.

There was another issue opened not too long ago, where a comment was made that if the retry filter is disabled than no leak. So I figured I would try.

This is my normal retry filter setup.

.retry(retryConfig -> retryConfig.setRetries(DEFAULT_RETRIES)
        .setExceptions(RetryableHttpClientErrorException.class, IOException.class, TimeoutException.class)
        .setStatuses(HttpStatus.UNAUTHORIZED)
        .setMethods(HttpMethod.GET, HttpMethod.POST))

Upon removing this, the leaks are gone. But the retry filter is kind of a big thing. We need this functionality.

I am pretty confident that somethign in this retry filter is triggering the leak in the reactive chain. No filter (still have timeouts). No leaks.

@spencergibb @violetagg

See comment here from @celesky.

dave-fl picture dave-fl on 27 Mar 2019

@dave-fl Can you try also the latest Spring Framework 5.1.6 snapshot as there are fixes relevant to memory leaks

cc/ @rstoyanchev

violetagg picture violetagg on 28 Mar 2019

Yes there are some recent fixes in Spring's codecs but isn't the retry applied to remote calls directly via Reactor HttpClient?

rstoyanchev picture rstoyanchev on 28 Mar 2019

That is true. But we also make use of Webclient under certain conditions. Would these fixes help with that?

dave-fl picture dave-fl on 28 Mar 2019

I think it's possible. Those were changes in writers which on the client case encode the request body.

rstoyanchev picture rstoyanchev on 28 Mar 2019

@rstoyanchev @violetagg Just ran again using 5.1.6.BUILD-SNAPSHOT. Leaked after about 20 minutes (the amount of time before the retry filter probably starts making some retries).

dave-fl picture dave-fl on 28 Mar 2019

Progress has definitely been made. But still leaking about 30-40 min into the test, probably when the system is doing a GC.

This is not easy to reproduce. System needs load + specific actions around terminating connections. It could be something else but I think this is what it is.

A possible setup.

Gateway 1 -> Gateway 2 -> Source where Source has GET and POST endpoints that return in different periods of time.

Setup Source to have different responses e.g. 200 or 401, make them random e.g. 401 30% of the time.

Setup Gateway 2 to Retry on certain auth status e.g. 401

Setup Gateway 1 to terminate the connection after time N.

Eventually Gateway 2 will leak.

I don't even know at this point if this is related to the gateway as I modified the source code and added a touch on the buffers for the Gateway within both the NettyRouting and NettyWriteResponse filters. This did not change the stack trace at all. Which leads me to believe that this error is happening before the gateway. 

2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT Recent access records:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #1:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT Created at:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT 2019-04-03 14:43:40.123 ERROR 20 --- [or-http-epoll-3] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT Recent access records:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #1:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #2:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #3:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:592)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #4:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:572)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #5:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:842)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:199)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #6:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT Created at:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:334)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2120)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1325)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT : 18 leak records were discarded because they were duplicates
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:557)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:193)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #7:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.left.httpCodec' will handle the message from this point.
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #8:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT #9:
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1486)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:218)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
2019-04-03T10:43:40.12-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
dave-fl picture dave-fl on 3 Apr 2019

This is using latest Spring Framework 5.1.7 snapshots? If you could just double check that you have this most recent change in your snapshot.

rstoyanchev picture rstoyanchev on 3 Apr 2019

@rstoyanchev It is there

dave-fl picture dave-fl on 3 Apr 2019
👍1

This is the trace after turning on touch in the FluxReceive

    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT 2019-04-04 17:18:17.786 ERROR 17 --- [or-http-epoll-4] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT Recent access records:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT #1:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     Hint: [id: 0x12055d19, L:/IP:41530 - R:HOST/IP:443] Buffered ByteBufHolder in Inbound Flux Queue
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:332)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:318)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:517)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT #2:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT Created at:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT #5:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:842)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:199)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT #6:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:557)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:193)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT #7:
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     Hint: 'reactor.left.httpCodec' will handle the message from this point.
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
    2019-04-04T13:18:17.78-0400 [APP/PROC/WEB/0] OUT     java.lang.Thread.run(Thread.java:748)
dave-fl picture dave-fl on 4 Apr 2019

https://github.com/reactor/reactor-core/issues/1649 could be related. Specifically around the usage of join, which must be done if using the retry filter.

dave-fl picture dave-fl on 5 Apr 2019

Has this issue been solved?

yechao4j picture yechao4j on 31 May 2019

I have met this bug in my production enviroment too,
Please anyone can help me??
2019-05-30 00:52:51.734 [NettyClientWorker-4-8] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more infor mation. Recent access records: Created at: io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143) io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:296) io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:103) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) java.lang.Thread.run(Thread.java:745)

yechao4j picture yechao4j on 31 May 2019

Yes it has, but hasn't been released

spencergibb picture spencergibb on 31 May 2019
👍1

Thanks.
And when and which version will be released in?

yechao4j picture yechao4j on 31 May 2019

@spencergibb - Please can you advise if the fix is released and the versions that we can use? We are using below versions in which we see the above mentioned issue - 

springBootVersion -  2.1.6.RELEASE
springCloudDependencies - Greenwich.SR2
spring-cloud-starter-gateway - 2.1.2.RELEASE
nilavalagansugumaran picture nilavalagansugumaran on 22 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

samtonyclarke picture samtonyclarke  路  68Comments
muzuro picture muzuro  路  61Comments
spencergibb picture spencergibb  路  36Comments
tony-clarke-amdocs picture tony-clarke-amdocs  路  32Comments
sincang picture sincang  路  41Comments