Azure-sdk-for-java: [BUG] cosmos sdk: intermittent leaks detected via netty leak detector

Created on 4 Aug 2020  Â·  28Comments  Â·  Source: Azure/azure-sdk-for-java

Describe the bug
A clear and concise description of what the bug is.

We intermittently receive LEAK errors from netty like [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. when using ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);

Exception or Stack Trace
Add the exception log and stack trace if available

 [ERROR] LEAK: ByteBuf.release() was not called before### DELETE
 it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:285)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
#2:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
    io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:616)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:258)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:225)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
#3:
    io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
    io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:940)
    io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:996)
    io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:239)
    io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:225)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
#4:
    Hint: 'reactor.left.httpCodec' will handle the message from this point.
    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
#5:
    io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
    io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1536)
    io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:167)
    io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:283)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1380)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
#6:
    io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
    io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1536)
    io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:167)
    io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:283)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1380)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
Created at:
    io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
    io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
    io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
    io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2198)
    io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1375)
    io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    java.base/java.lang.Thread.run(Thread.java:830)
: 2 leak records were discarded because they were duplicates
: 25 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

To Reproduce
Steps to reproduce the behavior:

I created a branch of the getting started sample to demonstrate:
https://github.com/tysonnorris/azure-cosmos-java-getting-started/tree/netty_leakdetector

Run it using:

  • openjdk 11
  • mvn clean install exec:java@leak

Code Snippet
Add the code snippet that causes the issue.

https://github.com/tysonnorris/azure-cosmos-java-getting-started/blob/netty_leakdetector/src/main/java/com/azure/cosmos/sample/async/AsyncMainLeak.java is a modified version of the AsyncMain.java sample

Expected behavior
A clear and concise description of what you expected to happen.

No leaks. We are upgrading from cosmos sdk 2.x where we had some leak issues long ago, so have tests that pass reliably with the paranoid level leak detection level, and those tests don't pass with the cosmos 4.3 sdk.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

  • OS: [e.g. iOS] macos
  • IDE : [e.g. IntelliJ] intellij with openjdk 1.11, also cli with openjdk 1.11
  • Version of the Library used com.azure:azure-cosmos:4.3.0

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • [x] Bug Description Added
  • [x] Repro Steps Added
  • [x] Setup information Added
Client Cosmos bug v4-item customer-reported question
Source
picture tysonnorris

All 28 comments

@David-Noble-at-work @xinlian12 - for your information.

kushagraThapar picture kushagraThapar on 4 Aug 2020

@tysonnorris - are you using DIRECT mode or GATEWAY mode ?
We suspect this issue is related to Http Client, and I am wondering if you can also reproduce this on just GATEWAY mode?
So as to verify our speculation.

kushagraThapar picture kushagraThapar on 5 Aug 2020

We are using GATEWAY mode in our app and my sample code is as well. We cannot easily switch to DIRECT mode so I haven’t tested that.

On Aug 4, 2020, at 8:30 PM, Kushagra Thapar notifications@github.com wrote:


@tysonnorris - are you using DIRECT mode or GATEWAY mode ?
We suspect this issue is related to Http Client, and I am wondering if you can also reproduce this on just GATEWAY mode?
So as to verify our speculation.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

tysonnorris picture tysonnorris on 5 Aug 2020

@tysonnorris - thanks for confirming that, will investigate this soon.

kushagraThapar picture kushagraThapar on 5 Aug 2020

@tysonnorris - I tried running AsyncMainLeak class from your forked repo, but I am not seeing any leaks. Is it consistent on your end, I tried running it 3-4 times?

kushagraThapar picture kushagraThapar on 7 Aug 2020

@kushagraThapar sorry I should have mentioned that it is NOT consistent, but generally I don't have problems seeing it at least once in 10 runs. Our more complicated test suites run into it always, but not in the same spot.

tysonnorris picture tysonnorris on 7 Aug 2020

No worries, let me try on the repro then, thanks!

kushagraThapar picture kushagraThapar on 8 Aug 2020

Hi @kushagraThapar - Any luck reproducing this issue?

tysonnorris picture tysonnorris on 12 Aug 2020

@tysonnorris - not yet, I have tried it multiple times. I am using macOS, which OS are you using ?

kushagraThapar picture kushagraThapar on 12 Aug 2020

I'm on recent macbook/macos as well, using jdk 1.8, 11, or 13. I've just run it multiple times in intellij and at cli, and can reproduce the LEAK output, at least every 3rd run, often more frequent. FWIW, our target jdk is openjdk 11.

I did make one change to the code just now, to NOT specify a preferred region, and the write region for the db we are testing is located in East US:
image

Just wondering if the db region is having something to do with reproducing it? I'm running my laptop in US West regional area (but the write region is US East).

tysonnorris picture tysonnorris on 13 Aug 2020

@tysonnorris - I don't think that is the issue.
I am also using open jdk 11, I pulled your latest commit and tried running the sample again multiple times today but unfortunately can't hit the issue.
Just to avoid any confusion :), are you hitting the issue just on this sample app - and no other apps running in parallel ?

kushagraThapar picture kushagraThapar on 13 Aug 2020

Yes I'm just running 

mvn clean install exec:java@leak

and hit it regularly, but not constantly. Here are the mvn/java versions:

-> % mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 13.0.2, vendor: N/A, runtime: /usr/local/Cellar/openjdk/13.0.2+8_2/libexec/openjdk.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.6", arch: "x86_64", family: "mac"
tnorris@Tysons-MacBook-Pro [11:19:44] [~/git/forks/tysonnorris/azure-cosmos-java-getting-started] [netty_leakdetector]
-> % java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
tnorris@Tysons-MacBook-Pro [11:19:48] [~/git/forks/tysonnorris/azure-cosmos-java-getting-started] [netty_leakdetector]
-> %
tysonnorris picture tysonnorris on 13 Aug 2020

I'm going to try to make a jar/container as well to run elsewhere, in case there is some issue with accessing cosmosdb over our corp vpn.

tysonnorris picture tysonnorris on 13 Aug 2020
👍1

@tysonnorris - earlier I was running the code through intellij, I ran it again around 10 times from command line using 

mvn clean install exec:java@leak

Still can't repro it..

Here is my mvn/java versions.

image

kushagraThapar picture kushagraThapar on 13 Aug 2020

I created a Dockerfile to run repeatedly:
https://github.com/tysonnorris/azure-cosmos-java-getting-started/blob/netty_leakdetector/Dockerfile

Running this as a pod in an k8s cluster on azure in east us region, I get much fewer, but not none, cases where the LEAK error appears. e.g. running for last 15 min:

-> % kubectl logs cosmosleaktest | grep LEAK
[2020-08-13T19:08:23.550Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:13:01.546Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:13:01.546Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

I have pod setup like

spec:
  containers:
    - name: cosmosleaktest
      image: tysonnorris/cosmos-leaktest
      command:
        - ./runtest.sh
      env:
        - name: ACCOUNT_KEY
          value: <my key>
        - name: ACCOUNT_HOST
          value: <my host>
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
tysonnorris picture tysonnorris on 13 Aug 2020
👍1

I will try it out today, thanks for the docker file.

kushagraThapar picture kushagraThapar on 13 Aug 2020

FWIW, I ran the dockerfile for a few hours in 2 clusters, one in east us azure region, and one in southeast asia azure region.
The southeast asia one showed only one error, but the east us one showed many. I'm not sure why they would be different.

-> % kubectl logs cosmosleaktest | grep LEAK                                                        
[2020-08-13T19:08:23.550Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:13:01.546Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:13:01.546Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:19:21.966Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:22:19.974Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:22:32.729Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:28:39.152Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:29:17.131Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:29:17.132Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:33:56.042Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:33:56.043Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:35:36.857Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:35:36.860Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:36:14.598Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:36:14.600Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:41:17.495Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:41:17.496Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:42:46.215Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:44:39.945Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:45:56.404Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:46:09.144Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:46:09.144Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:46:34.899Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:47:25.359Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:49:31.036Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:49:31.036Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:51:24.000Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:51:24.001Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:52:26.339Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:52:26.341Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:52:38.919Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:52:38.921Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:53:04.039Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:53:04.041Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:55:22.636Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:57:04.101Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:58:32.802Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:58:32.803Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T19:58:45.399Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:01:17.400Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:01:17.401Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:01:30.040Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:03:11.165Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:03:11.166Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:08:52.132Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:11:24.467Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:15:50.338Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:18:22.139Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:18:22.141Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:19:00.542Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:24:16.277Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:26:34.557Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:26:34.558Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:27:25.200Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:28:16.400Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:28:41.602Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:29:06.941Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:29:19.646Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:29:19.695Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:32:41.428Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:32:41.431Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:33:19.264Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:34:34.818Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:35:00.146Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:35:00.147Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:38:09.314Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:38:09.316Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:41:55.142Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:41:55.143Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:42:07.806Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:42:07.808Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:43:23.195Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:44:38.545Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:44:51.101Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:44:51.103Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:45:03.695Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:45:03.698Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:45:29.008Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:45:29.011Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:48:00.035Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:49:15.436Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:49:15.437Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:53:53.250Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:55:33.888Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:55:59.248Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:55:59.249Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T20:58:05.340Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:02:43.350Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:03:47.390Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:03:47.392Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:06:07.209Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:06:07.211Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:06:58.034Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:07:10.648Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:09:29.131Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:09:29.134Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:10:20.142Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:10:32.852Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:10:32.853Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:10:58.139Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:10:58.140Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:13:55.223Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:13:55.225Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:16:01.602Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:16:01.603Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:16:14.218Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:16:14.220Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:19:24.000Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:19:24.000Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:21:43.094Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:23:51.213Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:25:33.205Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:25:33.208Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:26:23.750Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:26:23.750Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:26:36.495Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:26:36.497Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:27:01.796Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:33:33.653Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:33:33.655Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:33:58.922Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:33:58.924Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:36:30.300Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:36:30.301Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:40:30.156Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:40:30.157Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:40:30.195Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:44:17.331Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:46:11.301Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:46:11.305Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:47:14.539Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:47:14.540Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:49:08.599Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:50:37.007Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:50:37.011Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:51:27.638Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:52:06.196Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:52:06.199Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:56:56.495Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:56:56.498Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:57:09.243Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:57:09.243Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:57:46.913Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:57:46.916Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:59:03.226Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T21:59:03.229Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:00:06.365Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:03:15.920Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:03:15.923Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:05:34.217Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:05:34.220Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:07:02.839Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:07:02.840Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:07:41.101Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:07:41.104Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:09:47.357Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:09:47.397Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:11:03.441Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:11:28.518Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:13:59.703Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:14:24.739Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:14:24.741Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:14:37.500Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:14:37.501Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:15:02.701Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:20:55.913Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:20:55.915Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:25:59.831Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:31:03.926Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:31:54.700Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:41:23.617Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:43:04.813Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[2020-08-13T22:44:58.848Z] [ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
tysonnorris picture tysonnorris on 14 Aug 2020

@tysonnorris - is the East US region cross region work load ?
What is the difference between these two regions from cosmos db account perspective ?
Is any of them multi-master account or setup in a different way than the other ?

kushagraThapar picture kushagraThapar on 14 Aug 2020

Not sure what you mean - in both cases, I just deployed the docker image (from the dockerfile linked above) as a pod in a kubernetes cluster. One kubernetes cluster is in east us 2 (sorry, not east us as previously mentioned), and one is in southeast asia. In both cases, the cosmosdb used is configured with write region as east us, and read regions north europe, southeast asia, and australia east.

The only "cross region" aspect is that the test pod is running in a different region than the cosmosdb write/read regions.

tysonnorris picture tysonnorris on 14 Aug 2020

@tysonnorris - I see, thanks for the information, I will try testing it against an account with different read and write regions, may be that will help.
And since, you are using the quickstart sample, I guess we can rule out a bad reactor-netty version conflict as well ?

kushagraThapar picture kushagraThapar on 14 Aug 2020

Yes - I tried to make the failing sample based on the quickstart sample to rule out dependency versions etc. If there are other versions you'd like me to try, I'm happy to do that.

tysonnorris picture tysonnorris on 14 Aug 2020

@kushagraThapar any luck reproducing this with the Dockerfile?

tysonnorris picture tysonnorris on 20 Aug 2020

@tysonnorris - no luck so far, I tried reproducing the issue on a windows machine as well, but couldn't reproduce it.
Do you have any other suggestions on how to go about this issue ?

kushagraThapar picture kushagraThapar on 24 Aug 2020

@kushagraThapar
Could you please try the below steps to reproduce the issue ?

Steps to reproduce the behavior:

  1. Create an Azure VM in East US 2 with Linux.
  2. Install docker
  3. Use the code from above referenced issue - https://github.com/tysonnorris/azure-cosmos-java-getting-started/tree/netty_leakdetector
  4. Build the docker image -> docker build -t cosmos-sdk-leak:latest .
  5. docker run -e ACCOUNT_HOST='YOUR_ACCOUNT_HOST' -e ACCOUNT_KEY='YOUR_ACCOUNT_KEY' -it cosmos-sdk-leak

Normally the exception arrives within 5-10 runs (runtest.sh executes a while loop)

Observation :

  1. performed the above steps from my local Macbook and able to reproduce.
  2. performed the above steps from Azure VM in West region and not able to reproduce. (However my region of interest is US East 2, but did test via West also to compare behavior)

Thanks!

ac4922 picture ac4922 on 28 Aug 2020

@ac4922 - thanks for the above steps, I will perform them today and will provide you an update.

kushagraThapar picture kushagraThapar on 28 Aug 2020

@tysonnorris @ac4922 - As part of debugging this issue, I am not able to use the DockerFile as it is, because it is always trying to resolve the dependencies from central maven repo.
However, I want docker image to be built and run using the local m2 cache and run using the dependencies present in local m2 cache (because I am testing some un-released version of azure-cosmos jar, which is present in my local m2 cache).

Do you know what changes I need to make to the DockerFile ?

kushagraThapar picture kushagraThapar on 1 Sep 2020

One more thing, not sure if you got the update, but the above issue only happens on Docker container when the memory is limited to 2 GB.

If we increase the memory to higher number (memory to 6 GB) - the issue is not reproducible, which is why I am not able to test it on my mac, which has more memory.

kushagraThapar picture kushagraThapar on 1 Sep 2020

This has been released in azure-cosmos v4.4.0 https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/cosmos/azure-cosmos/CHANGELOG.md#440-2020-09-12

kushagraThapar picture kushagraThapar on 14 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

valmol picture valmol  Â·  4Comments
frank-fischer-by picture frank-fischer-by  Â·  3Comments
abelmariam picture abelmariam  Â·  4Comments
ronny-sphera picture ronny-sphera  Â·  3Comments
hemanttanwar picture hemanttanwar  Â·  3Comments