Azure-sdk-for-java: Investigate ByteBuf resource leak Azure-KeyValut

Created on 4 Feb 2020  路  13Comments  路  Source: Azure/azure-sdk-for-java

Steps to reproduce:

Set up KeyVault Client with azure-security-keyvault-secrets: 4.1.0 and override Netty from 0.9.0 to 0.9.4 (as done by SpringBoot dependencies management)
Fetch a few entries from KeyVault

2020-01-24 15:41:04,606 [      ] [reactor-http-nio-1] ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak(320) - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2134)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1319)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
Azure.Core Client azure-spring-track2-migration tenet-reliability

Most helpful comment

@allenhumphreys, @blacelle - I can repro this using the dependencies in the maven tree. Looking into this.

All 13 comments

/cc @g2vinay as fyi

@anuchandy @alzimmermsft can you please follow up and ensure there's an external issue filed against 0.9.4?

I believe we're experiencing the same issue after updating to the latest version of the KeyVault library, specifically we upgraded
to com.microsoft.azure:azure-spring-boot:2.2.4 which utilizes com.azure:azure-security-keyvault-secrets:4.1.1

I tried several methods to work around this issue such as up/downgrading reactor netty, as well as the netty core azure library. Nothing seems to get around it.

I'm including information around our setup to hopefully expedite a fix.

Is there any update on the progress of finding a fix?

Netty Leak Detection Logs

--2020-04-21 11:05:51.046 -ERROR     -- [or-http-nio-2] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
Created at:
    io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
    io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
    io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:292)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
--2020-04-21 11:05:52.252 -ERROR     -- [or-http-nio-2] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#2:
    io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:292)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#3:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872)
    io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:572)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:214)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#4:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872)
    io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:552)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:214)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#5:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872)
    io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:928)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:195)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#6:
    Hint: 'reactor.left.httpCodec' will handle the message from this point.
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#7:
    io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
    io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1496)
    io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
    io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#8:
    io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
    io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1496)
    io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
    io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2151)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1335)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
: 15 leak records were discarded because they were duplicates
--2020-04-21 11:05:53.974 -ERROR     -- [or-http-nio-2] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    Hint: [id: 0x032065d6, L:/192.168.1.5:65042 - R:<vaultname>.vault.azure.net/<ipaddress>:443] Buffered ByteBufHolder in Inbound Flux Queue
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
    reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:348)
    reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:352)
    reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:598)
    reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:90)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
#2:
    Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
    io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
    io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)
Created at:
    io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143)
    io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67)
    io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:292)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
    io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:834)

I highly recommend running with the paranoid leak detection mode enabled via JVM arguments:

-Dio.netty.leakDetection.level=paranoid -Dio.netty.leakDetection.targetRecords=100

Our Setup:

  • OS: macOS locally, Linux
  • IDE : IntelliJ

    • JVM:

openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment (build 11.0.5+10)
OpenJDK 64-Bit Server VM (build 11.0.5+10, mixed mode)

Gradle Dependencies Report

+--- <redacted>:<redacted>-common:1.0.457
|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5
|    |    +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72
|    |    |    +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.72
|    |    |    \--- org.jetbrains:annotations:13.0
|    |    \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.70 -> 1.3.72
|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactor:1.3.5
|    |    +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*)
|    |    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*)
|    |    +--- io.projectreactor:reactor-core:3.2.5.RELEASE -> 3.3.4.RELEASE
|    |    |    \--- org.reactivestreams:reactive-streams:1.0.3
|    |    \--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.5
|    |         +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*)
|    |         +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*)
|    |         \--- org.reactivestreams:reactive-streams:1.0.2 -> 1.0.3
|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.3.5
|    |    +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*)
|    |    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*)
|    |    \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-rx2:1.3.5
|    |    +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*)
|    |    +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*)
|    |    +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.5 (*)
|    |    \--- io.reactivex.rxjava2:rxjava:2.2.8 -> 2.2.19
|    |         \--- org.reactivestreams:reactive-streams:1.0.3
|    +--- org.jetbrains.kotlinx:kotlinx-coroutines-jdk8:1.3.5
|    |    +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.70 -> 1.3.72 (*)
|    |    \--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 (*)
|    +--- com.fasterxml.jackson.module:jackson-module-kotlin:2.10.3
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3
|    |    \--- org.jetbrains.kotlin:kotlin-reflect:1.3.61 -> 1.3.72
|    |         \--- org.jetbrains.kotlin:kotlin-stdlib:1.3.72 (*)
|    +--- com.microsoft.azure:azure-servicebus-spring-boot-starter:2.2.4
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.6.RELEASE
|    |    |    +--- org.springframework.boot:spring-boot:2.2.6.RELEASE
|    |    |    |    +--- org.springframework:spring-core:5.2.5.RELEASE
|    |    |    |    |    \--- org.springframework:spring-jcl:5.2.5.RELEASE
|    |    |    |    \--- org.springframework:spring-context:5.2.5.RELEASE
|    |    |    |         +--- org.springframework:spring-aop:5.2.5.RELEASE
|    |    |    |         |    +--- org.springframework:spring-beans:5.2.5.RELEASE
|    |    |    |         |    |    \--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    |         |    \--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    |         +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|    |    |    |         +--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    |         \--- org.springframework:spring-expression:5.2.5.RELEASE
|    |    |    |              \--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.2.6.RELEASE
|    |    |    |    \--- org.springframework.boot:spring-boot:2.2.6.RELEASE (*)
|    |    |    +--- org.springframework.boot:spring-boot-starter-logging:2.2.6.RELEASE
|    |    |    |    +--- ch.qos.logback:logback-classic:1.2.3
|    |    |    |    |    +--- ch.qos.logback:logback-core:1.2.3
|    |    |    |    |    \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    |    |    +--- org.apache.logging.log4j:log4j-to-slf4j:2.12.1
|    |    |    |    |    +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    |    |    |    \--- org.apache.logging.log4j:log4j-api:2.12.1
|    |    |    |    \--- org.slf4j:jul-to-slf4j:1.7.30
|    |    |    |         \--- org.slf4j:slf4j-api:1.7.30
|    |    |    +--- jakarta.annotation:jakarta.annotation-api:1.3.5
|    |    |    +--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    \--- org.yaml:snakeyaml:1.25
|    |    +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.6.RELEASE
|    |    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    |    +--- jakarta.validation:jakarta.validation-api:2.0.2
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.33
|    |    |    \--- org.hibernate.validator:hibernate-validator:6.0.18.Final
|    |    |         +--- org.jboss.logging:jboss-logging:3.3.2.Final -> 3.4.1.Final
|    |    |         \--- com.fasterxml:classmate:1.3.4 -> 1.5.1
|    |    +--- com.microsoft.azure:azure-spring-boot:2.2.4
|    |    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.2.0.RELEASE -> 2.2.6.RELEASE (*)
|    |    |    +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30
|    |    |    +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.5.RELEASE
|    |    |    |    +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|    |    |    |    \--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.0 -> 2.10.3 (*)
|    |    |    +--- com.azure:azure-security-keyvault-secrets:4.1.1
|    |    |    |    +--- com.azure:azure-core:1.3.0
|    |    |    |    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.10.3
|    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3
|    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |    |    |    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |    |    |    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.10.1 -> 2.10.3
|    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3
|    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |    |    |    |    |    +--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.10.3
|    |    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3
|    |    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |    |    |    |    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |    |    |    |    |    |    +--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 -> 2.3.3
|    |    |    |    |    |    |    |    \--- jakarta.activation:jakarta.activation-api:1.2.2
|    |    |    |    |    |    |    \--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2
|    |    |    |    |    |    +--- org.codehaus.woodstox:stax2-api:4.2
|    |    |    |    |    |    \--- com.fasterxml.woodstox:woodstox-core:6.1.1
|    |    |    |    |    |         \--- org.codehaus.woodstox:stax2-api:4.2
|    |    |    |    |    +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30
|    |    |    |    |    +--- io.projectreactor:reactor-core:3.3.0.RELEASE -> 3.3.4.RELEASE (*)
|    |    |    |    |    \--- io.netty:netty-tcnative-boringssl-static:2.0.26.Final -> 2.0.30.Final
|    |    |    |    \--- com.azure:azure-core-http-netty:1.4.0
|    |    |    |         +--- com.azure:azure-core:1.3.0 (*)
|    |    |    |         +--- io.netty:netty-handler:4.1.44.Final -> 4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-resolver:4.1.48.Final
|    |    |    |         |    |    \--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-buffer:4.1.48.Final
|    |    |    |         |    |    \--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-transport:4.1.48.Final
|    |    |    |         |    |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    |    +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |    |    \--- io.netty:netty-resolver:4.1.48.Final (*)
|    |    |    |         |    \--- io.netty:netty-codec:4.1.48.Final
|    |    |    |         |         +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |         +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |         \--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         +--- io.netty:netty-handler-proxy:4.1.44.Final -> 4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-codec:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-codec-socks:4.1.48.Final
|    |    |    |         |    |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    |    +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |    |    +--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         |    |    \--- io.netty:netty-codec:4.1.48.Final (*)
|    |    |    |         |    \--- io.netty:netty-codec-http:4.1.48.Final
|    |    |    |         |         +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |         +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |         +--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         |         +--- io.netty:netty-codec:4.1.48.Final (*)
|    |    |    |         |         \--- io.netty:netty-handler:4.1.48.Final (*)
|    |    |    |         +--- io.netty:netty-buffer:4.1.44.Final -> 4.1.48.Final (*)
|    |    |    |         +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.48.Final (*)
|    |    |    |         +--- io.netty:netty-codec-http2:4.1.44.Final -> 4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-codec:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-handler:4.1.48.Final (*)
|    |    |    |         |    \--- io.netty:netty-codec-http:4.1.48.Final (*)
|    |    |    |         +--- io.netty:netty-transport-native-unix-common:4.1.44.Final -> 4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |    \--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         +--- io.netty:netty-transport-native-epoll:4.1.44.Final -> 4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-common:4.1.48.Final
|    |    |    |         |    +--- io.netty:netty-buffer:4.1.48.Final (*)
|    |    |    |         |    +--- io.netty:netty-transport:4.1.48.Final (*)
|    |    |    |         |    \--- io.netty:netty-transport-native-unix-common:4.1.48.Final (*)
|    |    |    |         \--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE -> 0.9.6.RELEASE
|    |    |    |              +--- io.netty:netty-codec-http:4.1.48.Final (*)
|    |    |    |              +--- io.netty:netty-codec-http2:4.1.48.Final (*)
|    |    |    |              +--- io.netty:netty-handler:4.1.48.Final (*)
|    |    |    |              +--- io.netty:netty-handler-proxy:4.1.48.Final (*)
|    |    |    |              +--- io.netty:netty-transport-native-epoll:4.1.48.Final (*)
|    |    |    |              \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*)
|    |    |    \--- com.azure:azure-identity:1.0.4
|    |    |         +--- com.azure:azure-core:1.3.0 (*)
|    |    |         +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30
|    |    |         +--- com.microsoft.azure:msal4j:0.5.0-preview
|    |    |         |    +--- com.nimbusds:oauth2-oidc-sdk:6.5 -> 6.14
|    |    |         |    |    +--- com.sun.mail:javax.mail:1.6.1
|    |    |         |    |    |    \--- javax.activation:activation:1.1
|    |    |         |    |    +--- com.github.stephenc.jcip:jcip-annotations:1.0-1
|    |    |         |    |    +--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3
|    |    |         |    |    |    \--- net.minidev:accessors-smart:1.2
|    |    |         |    |    |         \--- org.ow2.asm:asm:5.0.4
|    |    |         |    |    +--- com.nimbusds:lang-tag:[1.4.3,) -> 1.4.4
|    |    |         |    |    \--- com.nimbusds:nimbus-jose-jwt:[6.0.1,) -> 8.15
|    |    |         |    |         +--- com.github.stephenc.jcip:jcip-annotations:1.0-1
|    |    |         |    |         \--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3 (*)
|    |    |         |    +--- com.google.code.gson:gson:2.8.5 -> 2.8.6
|    |    |         |    +--- org.slf4j:slf4j-api:1.7.21 -> 1.7.30
|    |    |         |    +--- commons-codec:commons-codec:1.11 -> 1.13
|    |    |         |    +--- org.apache.commons:commons-lang3:3.9
|    |    |         |    +--- org.projectlombok:lombok:1.18.6 -> 1.18.12
|    |    |         |    \--- org.apache.httpcomponents:httpclient:4.5 -> 4.5.12
|    |    |         |         +--- org.apache.httpcomponents:httpcore:4.4.13
|    |    |         |         +--- commons-logging:commons-logging:1.2
|    |    |         |         \--- commons-codec:commons-codec:1.11 -> 1.13
|    |    |         +--- com.nimbusds:oauth2-oidc-sdk:6.14 (*)
|    |    |         +--- org.nanohttpd:nanohttpd:2.3.1
|    |    |         \--- net.java.dev.jna:jna-platform:5.4.0 -> 4.5.2
|    |    |              \--- net.java.dev.jna:jna:4.5.2
|    |    +--- com.microsoft.azure:azure-servicebus:1.2.15 -> 1.2.18
|    |    |    +--- org.apache.qpid:proton-j:0.31.0
|    |    |    +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30
|    |    |    \--- com.microsoft.azure:adal4j:1.3.0 -> 1.6.4
|    |    |         +--- com.nimbusds:oauth2-oidc-sdk:6.5 -> 6.14 (*)
|    |    |         +--- com.google.code.gson:gson:2.8.0 -> 2.8.6
|    |    |         +--- org.slf4j:slf4j-api:1.7.21 -> 1.7.30
|    |    |         +--- commons-codec:commons-codec:1.11 -> 1.13
|    |    |         \--- org.apache.commons:commons-lang3:3.5 -> 3.9
|    |    +--- com.google.code.findbugs:annotations:2.0.1
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.10.0 -> 2.10.3
|    |    \--- commons-codec:commons-codec:1.13
|    +--- com.microsoft.azure:azure-servicebus:1.2.18 (*)
|    +--- com.microsoft.azure:applicationinsights-logging-logback:2.6.0
|    |    +--- com.microsoft.azure:applicationinsights-core:2.6.0
|    |    +--- ch.qos.logback:logback-classic:1.2.3 (*)
|    |    \--- ch.qos.logback:logback-core:1.2.3
|    +--- org.springframework.boot:spring-boot-starter-security:2.2.6.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    +--- org.springframework:spring-aop:5.2.5.RELEASE (*)
|    |    +--- org.springframework.security:spring-security-config:5.2.2.RELEASE
|    |    |    +--- org.springframework.security:spring-security-core:5.2.2.RELEASE
|    |    |    |    +--- org.springframework:spring-aop:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    |    +--- org.springframework:spring-beans:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    |    +--- org.springframework:spring-context:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    |    +--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    |    \--- org.springframework:spring-expression:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-aop:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-beans:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    \--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    \--- org.springframework.security:spring-security-web:5.2.2.RELEASE
|    |         +--- org.springframework.security:spring-security-core:5.2.2.RELEASE (*)
|    |         +--- org.springframework:spring-aop:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-beans:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-context:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-expression:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |         \--- org.springframework:spring-web:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    +--- io.projectreactor:reactor-core:3.3.4.RELEASE (*)
|    +--- io.projectreactor.addons:reactor-adapter:3.3.3.RELEASE
|    |    \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*)
|    +--- io.projectreactor.addons:reactor-extra:3.3.3.RELEASE
|    |    \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*)
|    +--- io.projectreactor.kotlin:reactor-kotlin-extensions:1.0.2.RELEASE
|    |    \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.4.RELEASE (*)
|    +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.72 (*)
|    +--- org.jetbrains.kotlin:kotlin-reflect:1.3.72 (*)
|    +--- net.minidev:json-smart:2.3 (*)
|    +--- com.microsoft.azure:azure-eventgrid:1.3.0
|    |    \--- com.microsoft.azure:azure-client-runtime:1.5.0 -> 1.6.13
|    |         \--- com.microsoft.rest:client-runtime:1.6.13
|    |              +--- com.google.guava:guava:20.0 -> 28.2-jre
|    |              |    +--- com.google.guava:failureaccess:1.0.1
|    |              |    +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
|    |              |    +--- com.google.code.findbugs:jsr305:3.0.2
|    |              |    +--- org.checkerframework:checker-qual:2.10.0
|    |              |    +--- com.google.errorprone:error_prone_annotations:2.3.4
|    |              |    \--- com.google.j2objc:j2objc-annotations:1.3
|    |              +--- com.squareup.retrofit2:retrofit:2.5.0
|    |              |    \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2
|    |              |         \--- com.squareup.okio:okio:1.15.0
|    |              +--- com.squareup.okhttp3:okhttp:3.12.2 (*)
|    |              +--- com.squareup.okhttp3:logging-interceptor:3.12.2 -> 3.12.0
|    |              |    \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 (*)
|    |              +--- com.squareup.okhttp3:okhttp-urlconnection:3.12.2 -> 3.14.7
|    |              |    \--- com.squareup.okhttp3:okhttp:3.14.7 -> 3.12.2 (*)
|    |              +--- com.squareup.retrofit2:converter-jackson:2.5.0
|    |              |    +--- com.squareup.retrofit2:retrofit:2.5.0 (*)
|    |              |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.10.3 (*)
|    |              +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.8 -> 2.10.3
|    |              |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.3
|    |              |    +--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |              |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |              |    \--- joda-time:joda-time:2.9.9 -> 2.10.5
|    |              +--- org.apache.commons:commons-lang3:3.4 -> 3.9
|    |              +--- io.reactivex:rxjava:1.3.8
|    |              +--- com.squareup.retrofit2:adapter-rxjava:2.4.0
|    |              |    +--- com.squareup.retrofit2:retrofit:2.4.0 -> 2.5.0 (*)
|    |              |    \--- io.reactivex:rxjava:1.3.0 -> 1.3.8
|    |              +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30
|    |              \--- com.microsoft.azure:azure-annotations:1.7.0
|    +--- com.microsoft.azure:azure-cosmos:3.7.2
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.10.3 (*)
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.10.3 (*)
|    |    +--- com.fasterxml.uuid:java-uuid-generator:3.2.0
|    |    +--- commons-io:commons-io:2.5
|    |    +--- commons-validator:commons-validator:1.6
|    |    |    +--- commons-digester:commons-digester:1.8.1
|    |    |    +--- commons-logging:commons-logging:1.2
|    |    |    \--- commons-collections:commons-collections:3.2.2
|    |    +--- io.projectreactor:reactor-core:3.3.0.RELEASE -> 3.3.4.RELEASE (*)
|    |    +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.48.Final (*)
|    |    +--- io.netty:netty-handler:4.1.44.Final -> 4.1.48.Final (*)
|    |    +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30
|    |    +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9
|    |    +--- org.apache.commons:commons-collections4:4.2
|    |    +--- org.apache.commons:commons-text:1.6
|    |    |    \--- org.apache.commons:commons-lang3:3.8.1 -> 3.9
|    |    +--- com.google.guava:guava:25.0-jre -> 28.2-jre (*)
|    |    +--- io.dropwizard.metrics:metrics-core:4.1.0 -> 4.1.5
|    |    |    \--- org.slf4j:slf4j-api:1.7.30
|    |    +--- io.micrometer:micrometer-core:1.2.0 -> 1.3.6
|    |    |    +--- org.hdrhistogram:HdrHistogram:2.1.11
|    |    |    \--- org.latencyutils:LatencyUtils:2.0.3
|    |    +--- io.reactivex.rxjava2:rxjava:2.2.4 -> 2.2.19 (*)
|    |    \--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE -> 0.9.6.RELEASE (*)
|    +--- com.microsoft.azure:azure-cosmosdb-spring-boot-starter:2.2.4
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.6.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.6.RELEASE (*)
|    |    +--- com.microsoft.azure:azure-spring-boot:2.2.4 (*)
|    |    \--- com.microsoft.azure:spring-data-cosmosdb:2.2.3.FIX1
|    |         +--- org.springframework:spring-core:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-tx:5.2.0.RELEASE -> 5.2.5.RELEASE
|    |         |    +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|    |         |    \--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |         +--- org.springframework.data:spring-data-commons:2.2.0.RELEASE -> 2.2.6.RELEASE
|    |         |    +--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |         |    +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|    |         |    \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
|    |         +--- org.springframework:spring-expression:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- com.microsoft.azure:azure-cosmos:3.7.1 -> 3.7.2 (*)
|    |         +--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.10.0 -> 2.10.3
|    |         |    +--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |         |    \--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |         +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.0 -> 2.10.3
|    |         |    +--- com.fasterxml.jackson.core:jackson-core:2.10.3
|    |         |    \--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |         +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.0 -> 2.10.3 (*)
|    |         +--- org.json:json:20140107
|    |         \--- org.javatuples:javatuples:1.2
|    +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.6.0
|    |    \--- com.microsoft.azure:applicationinsights-web:2.6.0
|    +--- org.springframework.boot:spring-boot-starter-actuator:2.2.6.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-actuator-autoconfigure:2.2.6.RELEASE
|    |    |    +--- org.springframework.boot:spring-boot-actuator:2.2.6.RELEASE
|    |    |    |    +--- org.springframework.boot:spring-boot:2.2.6.RELEASE (*)
|    |    |    |    \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.3 (*)
|    |    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.2.6.RELEASE (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |    |    +--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-context:5.2.5.RELEASE (*)
|    |    |    \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.3 (*)
|    |    \--- io.micrometer:micrometer-core:1.3.6 (*)
|    +--- org.springframework.boot:spring-boot-starter-webflux:2.2.6.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-json:2.2.6.RELEASE
|    |    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    |    +--- org.springframework:spring-web:5.2.5.RELEASE (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.3 (*)
|    |    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.3 (*)
|    |    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.3 (*)
|    |    |    \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.10.3 (*)
|    |    +--- org.springframework.boot:spring-boot-starter-reactor-netty:2.2.6.RELEASE
|    |    |    +--- io.projectreactor.netty:reactor-netty:0.9.6.RELEASE (*)
|    |    |    \--- org.glassfish:jakarta.el:3.0.3
|    |    +--- org.springframework.boot:spring-boot-starter-validation:2.2.6.RELEASE (*)
|    |    +--- org.springframework:spring-web:5.2.5.RELEASE (*)
|    |    +--- org.springframework:spring-webflux:5.2.5.RELEASE
|    |    |    +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |    |    +--- org.springframework:spring-web:5.2.5.RELEASE (*)
|    |    |    \--- io.projectreactor:reactor-core:3.3.4.RELEASE (*)
|    |    \--- org.synchronoss.cloud:nio-multipart-parser:1.1.0
|    |         +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30
|    |         \--- org.synchronoss.cloud:nio-stream-storage:1.1.3
|    |              \--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30
|    +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT
|    |    +--- io.swagger:swagger-annotations:1.5.20
|    |    +--- io.swagger:swagger-models:1.5.20
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.5 -> 2.10.3
|    |    |    +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30
|    |    |    \--- io.swagger:swagger-annotations:1.5.20
|    |    +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT
|    |    |    \--- io.springfox:springfox-core:3.0.0-SNAPSHOT
|    |    |         +--- net.bytebuddy:byte-buddy:1.9.10 -> 1.10.8
|    |    |         +--- com.fasterxml:classmate:1.4.0 -> 1.5.1
|    |    |         +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    |         +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE
|    |    |         |    +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |    |         |    +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |    |         |    +--- org.springframework:spring-aop:5.2.0.RELEASE -> 5.2.5.RELEASE (*)
|    |    |         |    \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    |         \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE
|    |    |              +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*)
|    |    |              \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT
|    |    |    +--- io.springfox:springfox-core:3.0.0-SNAPSHOT (*)
|    |    |    \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*)
|    |    +--- io.springfox:springfox-swagger-common:3.0.0-SNAPSHOT
|    |    |    +--- io.swagger:swagger-annotations:1.5.20
|    |    |    +--- io.swagger:swagger-models:1.5.20 (*)
|    |    |    +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*)
|    |    |    +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT (*)
|    |    |    +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT
|    |    |    |    +--- io.github.classgraph:classgraph:4.1.7
|    |    |    |    +--- com.fasterxml:classmate:1.4.0 -> 1.5.1
|    |    |    |    +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    |    |    +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*)
|    |    |    |    +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*)
|    |    |    |    \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*)
|    |    |    +--- com.fasterxml:classmate:1.4.0 -> 1.5.1
|    |    |    +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    |    +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*)
|    |    |    \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*)
|    |    +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*)
|    |    +--- com.fasterxml:classmate:1.4.0 -> 1.5.1
|    |    +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*)
|    |    +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*)
|    |    \--- org.mapstruct:mapstruct:1.3.1.Final
|    +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT
|    |    +--- com.fasterxml:classmate:1.4.0 -> 1.5.1
|    |    +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
|    |    +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*)
|    |    +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*)
|    |    +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*)
|    |    \--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*)
|    +--- org.springframework.security:spring-security-oauth2-jose:5.3.1.RELEASE -> 5.2.2.RELEASE
|    |    +--- com.nimbusds:nimbus-jose-jwt:7.8.1 -> 8.15 (*)
|    |    +--- org.springframework.security:spring-security-core:5.2.2.RELEASE (*)
|    |    +--- org.springframework.security:spring-security-oauth2-core:5.2.2.RELEASE
|    |    |    +--- org.springframework.security:spring-security-core:5.2.2.RELEASE (*)
|    |    |    +--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    |    \--- org.springframework:spring-web:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    |    \--- org.springframework:spring-core:5.2.3.RELEASE -> 5.2.5.RELEASE (*)
|    +--- com.nimbusds:nimbus-jose-jwt:8.15 (*)
|    +--- com.onelogin:java-saml:2.5.0
|    |    +--- com.onelogin:java-saml-core:2.5.0
|    |    |    +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30
|    |    |    +--- joda-time:joda-time:2.10.3 -> 2.10.5
|    |    |    +--- org.apache.commons:commons-lang3:3.4 -> 3.9
|    |    |    +--- org.apache.santuario:xmlsec:2.1.4
|    |    |    |    +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
|    |    |    |    +--- com.fasterxml.woodstox:woodstox-core:5.0.3 -> 6.1.1 (*)
|    |    |    |    \--- commons-codec:commons-codec:1.12 -> 1.13
|    |    |    \--- commons-codec:commons-codec:1.12 -> 1.13
|    |    +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30
|    |    +--- joda-time:joda-time:2.10.3 -> 2.10.5
|    |    +--- org.apache.commons:commons-lang3:3.4 -> 3.9
|    |    +--- org.apache.santuario:xmlsec:2.1.4 (*)
|    |    \--- commons-codec:commons-codec:1.12 -> 1.13
|    \--- io.reactivex:rxjava-reactive-streams:1.2.1
|         +--- io.reactivex:rxjava:1.2.2 -> 1.3.8
|         \--- org.reactivestreams:reactive-streams:1.0.0 -> 1.0.3
+--- org.jetbrains.kotlin:kotlin-stdlib:1.3.72 (*)
+--- org.jetbrains.kotlin:kotlin-reflect:1.3.72 (*)
+--- org.springframework.boot:spring-boot-starter-actuator:2.2.6.RELEASE (*)
+--- org.springframework.boot:spring-boot-starter-webflux:2.2.6.RELEASE (*)
+--- org.springframework.cloud:spring-cloud-starter-config:2.2.2.RELEASE
|    +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.5.RELEASE -> 2.2.6.RELEASE (*)
|    |    +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE
|    |    |    \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE
|    |    +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE
|    |    |    \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE
|    |    \--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE
|    |         \--- org.bouncycastle:bcpkix-jdk15on:1.64
|    |              \--- org.bouncycastle:bcprov-jdk15on:1.64
|    +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE
|    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.6.RELEASE (*)
|    |    +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*)
|    |    +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE (*)
|    |    +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.5.RELEASE (*)
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 -> 2.10.3
|    |    \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.10.3 (*)
|    \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.10.3 (*)
+--- org.springframework.cloud:spring-cloud-config-server:2.2.2.RELEASE
|    +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-actuator:2.2.5.RELEASE -> 2.2.6.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-web:2.2.5.RELEASE -> 2.2.6.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-json:2.2.6.RELEASE (*)
|    |    +--- org.springframework.boot:spring-boot-starter-tomcat:2.2.6.RELEASE
|    |    |    +--- jakarta.annotation:jakarta.annotation-api:1.3.5
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-core:9.0.33
|    |    |    +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.33
|    |    |    \--- org.apache.tomcat.embed:tomcat-embed-websocket:9.0.33
|    |    |         \--- org.apache.tomcat.embed:tomcat-embed-core:9.0.33
|    |    +--- org.springframework.boot:spring-boot-starter-validation:2.2.6.RELEASE (*)
|    |    +--- org.springframework:spring-web:5.2.5.RELEASE (*)
|    |    \--- org.springframework:spring-webmvc:5.2.5.RELEASE
|    |         +--- org.springframework:spring-aop:5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-context:5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-core:5.2.5.RELEASE (*)
|    |         +--- org.springframework:spring-expression:5.2.5.RELEASE (*)
|    |         \--- org.springframework:spring-web:5.2.5.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-validation:2.2.5.RELEASE -> 2.2.6.RELEASE (*)
|    +--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE
|    +--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE (*)
|    +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r
|    |    +--- com.jcraft:jsch:0.1.54
|    |    +--- com.jcraft:jzlib:1.1.1
|    |    +--- com.googlecode.javaewah:JavaEWAH:1.1.6
|    |    \--- org.slf4j:slf4j-api:1.7.2 -> 1.7.30
|    +--- org.eclipse.jgit:org.eclipse.jgit.http.apache:5.1.3.201810200350-r
|    |    +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r (*)
|    |    +--- org.apache.httpcomponents:httpclient:4.5.5 -> 4.5.12 (*)
|    |    \--- org.apache.httpcomponents:httpcore:4.4.9 -> 4.4.13
|    +--- org.yaml:snakeyaml:1.25
|    \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.2 -> 2.10.3
|         +--- org.yaml:snakeyaml:1.24 -> 1.25
|         \--- com.fasterxml.jackson.core:jackson-core:2.10.3
+--- org.springframework.cloud:spring-cloud-starter-openfeign:2.2.2.RELEASE
|    +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE (*)
|    +--- org.springframework.cloud:spring-cloud-openfeign-core:2.2.2.RELEASE
|    |    +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.6.RELEASE (*)
|    |    +--- org.springframework.cloud:spring-cloud-netflix-ribbon:2.2.2.RELEASE
|    |    |    \--- org.springframework.cloud:spring-cloud-netflix-archaius:2.2.2.RELEASE
|    |    +--- org.springframework.boot:spring-boot-starter-aop:2.2.5.RELEASE -> 2.2.6.RELEASE
|    |    |    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    |    |    +--- org.springframework:spring-aop:5.2.5.RELEASE (*)
|    |    |    \--- org.aspectj:aspectjweaver:1.9.5
|    |    \--- io.github.openfeign.form:feign-form-spring:3.8.0
|    |         +--- io.github.openfeign.form:feign-form:3.8.0
|    |         |    \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
|    |         +--- org.springframework:spring-web:5.1.5.RELEASE -> 5.2.5.RELEASE (*)
|    |         +--- commons-fileupload:commons-fileupload:1.4
|    |         |    \--- commons-io:commons-io:2.2 -> 2.5
|    |         \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
|    +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.5.RELEASE (*)
|    +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*)
|    +--- io.github.openfeign:feign-core:10.7.4
|    +--- io.github.openfeign:feign-slf4j:10.7.4
|    |    +--- io.github.openfeign:feign-core:10.7.4
|    |    \--- org.slf4j:slf4j-api:1.7.13 -> 1.7.30
|    \--- io.github.openfeign:feign-hystrix:10.7.4
|         +--- io.github.openfeign:feign-core:10.7.4
|         +--- com.netflix.archaius:archaius-core:0.7.6
|         |    +--- com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2
|         |    +--- commons-configuration:commons-configuration:1.8
|         |    |    +--- commons-lang:commons-lang:2.6
|         |    |    \--- commons-logging:commons-logging:1.1.1 -> 1.2
|         |    +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.30
|         |    +--- com.google.guava:guava:16.0 -> 28.2-jre (*)
|         |    +--- com.fasterxml.jackson.core:jackson-annotations:2.4.3 -> 2.10.3
|         |    +--- com.fasterxml.jackson.core:jackson-core:2.4.3 -> 2.10.3
|         |    \--- com.fasterxml.jackson.core:jackson-databind:2.4.3 -> 2.10.3 (*)
|         \--- com.netflix.hystrix:hystrix-core:1.5.18
|              +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30
|              +--- com.netflix.archaius:archaius-core:0.4.1 -> 0.7.6 (*)
|              +--- io.reactivex:rxjava:1.2.0 -> 1.3.8
|              \--- org.hdrhistogram:HdrHistogram:2.1.9 -> 2.1.11
+--- org.springframework.data:spring-data-commons:2.2.6.RELEASE (*)
+--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT (*)
+--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT (*)
+--- io.springfox:springfox-swagger-ui:3.0.0-SNAPSHOT
+--- javax.persistence:persistence-api:1.0.2
+--- io.projectreactor:reactor-core:3.3.4.RELEASE (*)
+--- com.github.fge:json-patch:1.9
|    \--- com.github.fge:jackson-coreutils:1.6
|         +--- com.github.fge:msg-simple:1.1
|         |    +--- com.github.fge:btf:1.2
|         |    |    \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2
|         |    \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2
|         +--- com.fasterxml.jackson.core:jackson-databind:2.2.3 -> 2.10.3 (*)
|         +--- com.google.guava:guava:16.0.1 -> 28.2-jre (*)
|         \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2
+--- com.microsoft.azure:azure-cosmosdb-spring-boot-starter:2.2.4 (*)
+--- com.microsoft.azure:azure-keyvault-secrets-spring-boot-starter:2.2.4
|    +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.6.RELEASE (*)
|    +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.6.RELEASE (*)
|    +--- com.microsoft.azure:azure-spring-boot:2.2.4 (*)
|    +--- com.microsoft.azure:azure-client-authentication:1.6.13
|    |    +--- com.microsoft.azure:azure-client-runtime:1.6.13 (*)
|    |    +--- commons-codec:commons-codec:1.11 -> 1.13
|    |    +--- com.microsoft.azure:adal4j:1.6.4 (*)
|    |    \--- com.microsoft.azure:azure-annotations:1.7.0
|    \--- commons-io:commons-io:2.3 -> 2.5
+--- com.microsoft.azure:azure-cosmos:3.7.2 (*)
+--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.6.0 (*)
+--- com.squareup.okhttp3:okhttp:3.12.2 (*)
+--- com.google.guava:guava:28.2-jre (*)
+--- org.jsmpp:jsmpp:2.3.7
|    \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
+--- org.springframework.boot:spring-boot-starter-cache:2.2.6.RELEASE
|    +--- org.springframework.boot:spring-boot-starter:2.2.6.RELEASE (*)
|    \--- org.springframework:spring-context-support:5.2.5.RELEASE
|         +--- org.springframework:spring-beans:5.2.5.RELEASE (*)
|         +--- org.springframework:spring-context:5.2.5.RELEASE (*)
|         \--- org.springframework:spring-core:5.2.5.RELEASE (*)
+--- javax.cache:cache-api:1.1.1
\--- org.ehcache:ehcache:3.8.1
     +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
     \--- org.glassfish.jaxb:jaxb-runtime:2.3.1 -> 2.3.2
          +--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 -> 2.3.3 (*)
          +--- org.glassfish.jaxb:txw2:2.3.2
          +--- com.sun.istack:istack-commons-runtime:3.0.8
          |    \--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2
          +--- org.jvnet.staxex:stax-ex:1.8.1
          |    +--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2
          |    \--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 -> 2.3.3 (*)
          +--- com.sun.xml.fastinfoset:FastInfoset:1.2.16
          \--- jakarta.activation:jakarta.activation-api:1.2.1 -> 1.2.2

@allenhumphreys, @blacelle - I can repro this using the dependencies in the maven tree. Looking into this.

The above PR fixing this has been merged.
\cc @allenhumphreys, @blacelle

@anuchandy any idea when a new version will be released?

@timwhit - we have a patch release planned around mid of May. I'll update this thread once bits are out.

@anuchandy Please confirm the library/version supposed to fix this.

The version "com.azure:azure-security-keyvault-secrets:4.1.3" include the fix.

We are seeing the same error with some of our tests that exercise cosmosdb sdk, using com.azure:azure-cosmos:4.0.1 with jdk 1.8 and jdk 11. I'm trying to narrow down a test case, but it happens intermittently, about 50% of our test runs. Let me know if you have any advice of possible fix for cosmos sdk apis?

@tysonnorris thanks for reaching out. Can you open a new issue targetting cosmos-SDK with the findings so far from your side?

@tysonnorris I just want to provide bit more details on reason I requested a new issue.

The original problem specifically was due to key-vault library not closing stream in the pipeline.

The ByteBuf leak could happen if the stream is not closed in the library or in the user app. We need to investigate the issue you are seeing in the context of cosmos-sdk, I'm opening a new issue and will tag you in it.

Sorry that this issue's title is misleading, I just corrected it to avoid any confusion in the future.

Was this page helpful?
0 / 5 - 0 ratings