Spring-boot-admin: ByteBuf leak

Created on 9 Mar 2018 · 28Comments · Source: codecentric/spring-boot-admin

I have upgraded to the latest version (2.0.0-SNAPSHOT) and I am now getting an OutOfDirectMemoryError after some time of normal usage.

The server is configured to monitor the applications registered to our eureka server.

Here is the stacktrace:

io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658247, max: 257425408)
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304) ~[netty-handler-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:291) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:465) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:191) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.FutureMono.lambda$deferFuture$0(FutureMono.java:68) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:134) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:501) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658247, max: 257425408)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:113) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:92) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.22.Final.jar!/:4.1.22.Final]
        ... 37 common frames omitted

Anyone else with this problem?

bug server

Source

didiez

Most helpful comment

@joshiste what about releasing RC1 , since 90% of issues are fixed for 2.0.0?

n0mer on 19 Mar 2018

👍2

All 28 comments

How long is "some time" ?

joshiste on 9 Mar 2018

Could you give me your memory limits -Xmx?

joshiste on 9 Mar 2018

running on netty server or tomcat?

joshiste on 9 Mar 2018

How long is "some time" ?

A couple of days

Could you give me your memory limits -Xmx?

-Xmx256m -Xms256m

running on netty server or tomcat?

org.springframework.boot.web.embedded.netty.NettyWebServer

didiez on 12 Mar 2018

could you try with running on tomcat by explicitly including spring-boot-starter-web?

joshiste on 12 Mar 2018

Done, I'll let you know if I keep getting this memory error.
Thanks

didiez on 12 Mar 2018

Same problem running on tomcat.
This time the io.netty.util.internal.OutOfDirectMemoryError was thrown after 3 hours.

didiez on 15 Mar 2018

do you have the ui open all the time? I have the feeling that's somehow because of the infinite event stream....

joshiste on 15 Mar 2018

can you tell me a bit about you usage? how many instances are registered in that application? Are you frequently restarting/redeploying the clients so that the registrations changes frequently?

joshiste on 15 Mar 2018

do you have the ui open all the time? I have the feeling that's somehow because of the infinite event stream....

I'm not sure about this. Could be. I'm not the only one with access to the UI.

can you tell me a bit about you usage? how many instances are registered in that application? Are you frequently restarting/redeploying the clients so that the registrations changes frequently?

We are monitoring something around 30 applications (50 instances total). Certain applications could be deployed several times a day. If I had to say a number, I would say 30-40 registrations changes in peak days.

Nevertheless, we are deploying our applications at the same pace as we were doing with the previous stable version.

didiez on 15 Mar 2018

i'm running 2.0.0-snapshot for at least a month, no issues regarding memory leaking.

@didiez what is your exact snapshot version or date? have you tried to build from more recent snapshot?

n0mer on 19 Mar 2018

@joshiste what about releasing RC1 , since 90% of issues are fixed for 2.0.0?

n0mer on 19 Mar 2018

👍2

@nomer good to know. could you tell us your -Xmx setting and the mem usage after a long uptime?

joshiste on 19 Mar 2018

@nomer as we don't have a milestone repository there is no place to publish binaries....

joshiste on 19 Mar 2018

maybe not related with memory leak but similar case happened.
when running spring-boot-admin-server on a small spec CF (1g memory) instance, after a few days the UI can not show the health status of each instance. Currently, changed to middle spec CF (4g memory) instance and checking if the situation will be resolved.

tetsushiawano on 20 Mar 2018

happened again.
screen shot 2018-03-20 at 13 59 50

tetsushiawano on 20 Mar 2018

@tetsushiawano did you check the logs for errors?

joshiste on 20 Mar 2018

I check error log. it was same OutOfDirectMemoryError. This error does not show regularly but when I open the instance page the following error is output to log.

16:12:48.026: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.026: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.026: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.026: [APP/PROC/WEB.0]  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.026: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.026: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.027: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.027: [APP/PROC/WEB.0]  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.027: [APP/PROC/WEB.0]  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.027: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.027: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.027: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.027: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.027: [APP/PROC/WEB.0]      at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
16:12:48.027: [APP/PROC/WEB.0]      at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:870) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.028: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.028: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.028: [APP/PROC/WEB.0]      at brave.servlet.TracingFilter.doFilter(TracingFilter.java:76) [brave-instrumentation-servlet-4.18.0.jar!/:na]
16:12:48.028: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.028: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.028: [APP/PROC/WEB.0]      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.044: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.044: [APP/PROC/WEB.0]  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.044: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0] Caused by: io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 256 byte(s) of direct memory (used: 10485655, max: 10485760)
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:465) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:84) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at brave.servlet.TracingFilter.doFilter(TracingFilter.java:76) [brave-instrumentation-servlet-4.18.0.jar!/:na]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]  at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0] 2018-03-20 07:12:48.044 ERROR [spring-boot-admin,,,] 7 --- [nio-8080-exec-5] o.a.c.c.C.[.[.[.[dispatcherServlet]      : Servlet.service() for servlet [dispatcherServlet] in context with path [/spring-boot-admin] threw exception [Request processing failed; nested exception is io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 256 byte(s) of direct memory (used: 10485655, max: 10485760)] with root cause
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304) ~[netty-handler-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:501) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
16:12:48.045: [APP/PROC/WEB.0]  at org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:77) [client_certificate_mapper-1.4.0_RELEASE.jar!/:1.4.0.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304) ~[netty-handler-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:291) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:191) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.ipc.netty.FutureMono.lambda$deferFuture$0(FutureMono.java:68) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:46) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:501) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.045: [APP/PROC/WEB.0]  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.046: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.046: [APP/PROC/WEB.0]      at reactor.core.publisher.Mono.block(Mono.java:1199) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]  ... 1 common frames omitted
16:12:48.046: [APP/PROC/WEB.0]  Suppressed: java.lang.Exception: #block terminated with an error
16:12:48.046: [APP/PROC/WEB.0]      at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]
16:12:48.046: [APP/PROC/WEB.0]      at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ~[na:na]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:870) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:870) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
16:12:48.046: [APP/PROC/WEB.0] Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 256 byte(s) of direct memory (used: 10485655, max: 10485760)
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:327) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  ... 38 common frames omitted
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:465) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:191) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:134) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:48) ~[spring-cloud-sleuth-core-2.0.0.BUILD-SNAPSHOT.jar!/:2.0.0.BUILD-SNAPSHOT]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:77) [client_certificate_mapper-1.4.0_RELEASE.jar!/:1.4.0.RELEASE]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
16:12:48.046: [APP/PROC/WEB.0]      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.28.jar!/:8.5.28]
16:12:48.046: [APP/PROC/WEB.0]      ... 1 common frames omitted
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:595) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:113) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:92) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0] io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 256 byte(s) of direct memory (used: 10485655, max: 10485760)
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:595) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:327) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:113) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:92) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:291) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.ipc.netty.FutureMono.lambda$deferFuture$0(FutureMono.java:68) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:46) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
16:12:48.047: [APP/PROC/WEB.0]  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
16:12:48.047: [APP/PROC/WEB.0]  at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]

Let me also post the instance page once the issue happens.
screen shot 2018-03-20 at 16 16 25

tetsushiawano on 20 Mar 2018

I will change memory setting with -Xms -Xmx.

tetsushiawano on 20 Mar 2018

It's clearly stating that it can't allocate direct memory buffers: OutOfDirectMemoryError: failed to allocate 256 byte(s) of direct memory (used: 10485655, max: 10485760)] with root cause

IIRC the Pivotal Java Buildpacks now explicitely limit all native memory "pools" if possible. What you are looking for is -XX:MaxDirectMemorySize set in the JAVA_OPTS. (Perhaps indirectly from the Buildpack.) You'd might wan't to increase that. But you should monitor this closely (e.g. with Micrometer) in case this is allocating more and more direct memory buffers then re-using. The Hotspot JVM also has a habbit of caching once allocated buffers for re-use. Tunable with -Djdk.nio.maxCachedBufferSize if I remeber correctly.

mweirauch on 20 Mar 2018

@mweirauch
It seems java-buildpack-memory-buildpack is explicitly setting the -XX:MaxDirectMemorySize to 10MB. I will once change it to 20MB and see if it would resolve.

JAVA_OPTS: "-XX:MaxDirectMemorySize=20m"

Afterwards if there is a case of allocating more and more direct memory, I will use -Djdk.nio.maxCachedBufferSize as recommended.
Thank you so much!

tetsushiawano on 20 Mar 2018

I changed to 20MB but the same error happend.

12:20:42.623: [APP/PROC/WEB.0] Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 256 byte(s) of direct memory (used: 20971447, max: 20971520)

I will change JAVA_OPTS to below and see if it solves it.

JAVA_OPTS: "-XX:MaxDirectMemorySize=20m -Djdk.nio.MaxCachedBufferSize=20m"

tetsushiawano on 22 Mar 2018

@didiez @tetsushiawano I did some load testing and couldn't find any memory leak. I guess I need more information about your setup (project, complete logs, jvm version, platform/infrastructure). As you probably doesn't want to make this sensitive information public, would you send me some email?

joshiste on 22 Mar 2018

@joshiste in your testing, did you see how much native memory was used?

In briefly searching about the issue, I came across this blog which advises configuring Netty's leak detection. This guide from Oracle might also prove helpful in investigating this issue.

shakuzen on 22 Mar 2018

@shakuzen thanks for sharing. I had the leak detection active and there weren't any reports. I didn't observe the native memory directly but did a -Xmx128m which should also constraint the native memory or not?

joshiste on 22 Mar 2018

@joshiste No, direct memory buffer allocation is not part of the heap managed (in the sense of garbage collected) pools (Eden, Old, Metaspace, Compressed Class Space, ...). It's memory directly allocated by the JVM from the host OS. In a sense it's still managed due to being limitable by -XX:MaxDirectMemorySize and pools are reused (-Djdk.nio.MaxCachedBufferSize) for the JVM not having to re-allocate memory.

I'd have assumed this is a application specific thing and unrelated to SBA. One should monitor the metrics and see if its a steady growth or actually triggered by some specific action within the application. E.g. monitoring the direct memory buffer metrics with and without SBA polling for information on the application itself. Perhaps some queried actutor endpoints are causing steady direct buffer allocations in this netty setup.

mweirauch on 22 Mar 2018

@joshiste
We are running SBA on CloudFoundry on this buildpack. And using openjdk-1.8.0_152. I can't send entire log because we are not allowed to send entire log due to company policy.

But at the moment, SBA is working well with the following setting.

JAVA_OPTS: "-XX:MaxDirectMemorySize=20m -Djdk.nio.MaxCachedBufferSize=20m"

If this doesn't work,
I will try to monitor the direct memory buffer metrics as suggested by @mweirauch .

Thank you all for support!

tetsushiawano on 23 Mar 2018

I finally could track down the issue. -Dio.netty.leakDetectionLevel=paranoid helped. It turned out that the responses from the WebClient weren't always consumed and therefore the underlying buffers weren't released.

joshiste on 23 Mar 2018

🎉1

Was this page helpful?

0 / 5 - 0 ratings