Skywalking: [Agent] can not connect to the collector maybe cause memory leak

Created on 21 Jan 2019 · 9Comments · Source: apache/skywalking

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Bug

Which version of SkyWalking, OS and JRE?
- version: v5.0.0-GA
- OS: centos 6.5
- JRE: java version "1.8.0_144"
Which company or project?

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.

background
java, spring cloud, micro-service
what happened
several services unavailable on different machine and almost in same time.
java service log says

java.lang.OutOfMemoryError: GC overhead limit exceeded

skywalking log says

00:02:21.725 [SkywalkingAgent-2-GRPCChannelManager-0] ERROR o.a.s.a.d.i.g.i.ManagedChannelOrphanWrapper - *~*~*~ Channel ManagedChannelImpl{logId=1042418, target=xxxxx:xxxxx} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:410)
        at org.apache.skywalking.apm.agent.core.remote.GRPCChannel.<init>(GRPCChannel.java:46)
        at org.apache.skywalking.apm.agent.core.remote.GRPCChannel.<init>(GRPCChannel.java:31)
        at org.apache.skywalking.apm.agent.core.remote.GRPCChannel$Builder.build(GRPCChannel.java:95)
        at org.apache.skywalking.apm.agent.core.remote.GRPCChannelManager.run(GRPCChannelManager.java:99)
        at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
00:03:21.724 [SkywalkingAgent-2-GRPCChannelManager-0] ERROR o.a.s.a.d.i.g.i.ManagedChannelOrphanWrapper - *~*~*~ Channel ManagedChannelImpl{logId=1042467, target=xxxxxx:xxxxx} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.

java abnormal objects use 'jmap -histo:live {pid} > ~/histolive'

  num     #instances         #bytes  class name
 ----------------------------------------------
 [xxx] [xxx@xxxxxxxxxxxx ~]$ grep '39013' histolive 
 4:         39013       20598600  [Lorg.apache.skywalking.apm.dependencies.io.netty.handler.codec.http2.HpackHeaderField;
 11:         39013        8426808  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
 30:         39013        3745248  org.apache.skywalking.apm.dependencies.io.grpc.internal.TransportTracer
 40:         39013        2808936  org.apache.skywalking.apm.dependencies.io.netty.channel.DefaultChannelHandlerContext
 47:         39013        2496832  org.apache.skywalking.apm.dependencies.io.grpc.internal.DelayedClientTransport
 48:         39013        2496832  org.apache.skywalking.apm.dependencies.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory
 57:         39013        1872624  org.apache.skywalking.apm.dependencies.io.grpc.internal.Rescheduler
 94:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule
 95:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusTracingModule
 96:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
 97:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
 98:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.ServiceConfigInterceptor
 99:         39013        1248416  org.apache.skywalking.apm.dependencies.io.netty.channel.AdaptiveRecvByteBufAllocator
 128:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.AtomicBackoff
 129:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CallCredentialsApplyingTransportFactory
 130:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule$1
 131:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule$StatsClientInterceptor
 132:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusTracingModule$1

skywalking agent code fragment

package org.apache.skywalking.apm.agent.core.remote;
public class GRPCChannelManager ... {
  ...
  @Override
   public void run() {
       logger.debug("Selected collector grpc service running, reconnect:{}.", reconnect);
       if (reconnect) {
           if (RemoteDownstreamConfig.Collector.GRPC_SERVERS.size() > 0) {
               String server = "";
               try {
                   int index = Math.abs(random.nextInt()) % RemoteDownstreamConfig.Collector.GRPC_SERVERS.size();
                   server = RemoteDownstreamConfig.Collector.GRPC_SERVERS.get(index);
                   String[] ipAndPort = server.split(":");

                   managedChannel = GRPCChannel.newBuilder(ipAndPort[0], Integer.parseInt(ipAndPort[1]))
                       .addManagedChannelBuilder(new StandardChannelBuilder())
                       .addManagedChannelBuilder(new TLSChannelBuilder())
                       .addChannelDecorator(new AuthenticationDecorator())
                       .build();

                   if (!managedChannel.isShutdown() && !managedChannel.isTerminated()) {
                       reconnect = false;
                       notify(GRPCChannelStatus.CONNECTED);
                   } else {
                       notify(GRPCChannelStatus.DISCONNECT);
                   }
                   return;
               } catch (Throwable t) {
                   logger.error(t, "Create channel to {} fail.", server);
                   notify(GRPCChannelStatus.DISCONNECT);
               }
           }

           logger.debug("Selected collector grpc service is not available. Wait {} seconds to retry", Config.Collector.GRPC_CHANNEL_CHECK_INTERVAL);
       }
   }
  ...
}

if grpc original channel can not connect the collector...yes, the collector of this env being dead more than 10 days... :(
1. original channel will not shutdown
2. a new original channel will be created
3. memory leak
4. until gc can not reclaim memory, out of memory

agent bug high priority

Source

xuet0ng

👍1

Most helpful comment

Fixed.

wu-sheng on 23 Jan 2019

❤1 👍1

All 9 comments

Could you try that, shutdown the old one before creating a new one. Could that solve this leak?

Look forward your feedback and update.

wu-sheng on 21 Jan 2019

original jar
- start & connect collector success
  
  ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 1252: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2978: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3361: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3362: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4125: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4126: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4127: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4128: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4129: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5484: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5485: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5486: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5487: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5488: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 1246: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2985: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3372: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3373: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4130: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4131: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4132: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4133: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4134: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5485: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5486: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5487: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5488: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5489: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
- disconnect network & wait for 3 mins
  
  ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 1246: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2992: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3387: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3388: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4152: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4153: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4154: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4155: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4156: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5511: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5512: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5513: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5514: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5515: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 882: 2 432 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2017: 2 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 2352: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 2353: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2691: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2692: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2693: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2694: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2695: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 3407: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 3408: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 3409: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 3410: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 3411: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 732: 3 648 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1676: 3 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1876: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1877: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2159: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2160: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2161: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2162: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2163: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2714: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2715: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2716: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2717: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2718: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 663: 4 864 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1437: 4 160 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1578: 4 128 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1579: 4 128 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1910: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1911: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1912: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1913: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1914: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2408: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2409: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2410: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2411: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2412: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 601: 5 1080 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1290: 5 200 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1458: 5 160 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1459: 5 160 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1727: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1728: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1729: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1730: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1731: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2090: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2091: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2092: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2093: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2094: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 559: 6 1296 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1186: 6 240 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1343: 6 192 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1344: 6 192 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1554: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1555: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1556: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1557: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1558: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 1942: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 1943: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 1944: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 1945: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 1946: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 524: 7 1512 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1098: 7 280 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1267: 7 224 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1268: 7 224 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1441: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1442: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1443: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1444: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1445: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 1820: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 1821: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 1822: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 1823: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 1824: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
modified code if reconnect & old channel shutdownNow
- start & connect collector success
  
  ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 1250: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2999: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3385: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3386: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4140: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4141: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4142: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4143: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4144: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5495: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5496: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5497: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5498: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5499: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 1251: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2977: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3367: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3368: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4136: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4137: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4138: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4139: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4140: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5499: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5500: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5501: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5502: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5503: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
- disconnect network & wait for 3 mins
  
  ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 732: 3 648 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1675: 3 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1874: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1875: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2150: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2151: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2152: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2153: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2154: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2708: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2709: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2710: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2711: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2712: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 878: 2 432 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1999: 2 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 2338: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 2339: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2681: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2682: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2683: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2684: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2685: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 3399: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 3400: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 3401: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 3402: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 3403: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 732: 3 648 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1675: 3 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1874: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1875: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2150: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2151: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2152: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2153: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2154: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2706: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2707: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2708: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2709: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2710: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
中文了...这是本地电脑上的验证结果..
1. 不修改...一直能链接上collector..链接3分钟...ManagedChannelImpl的对象数一直是1
2. 不修改...先链接collector..再断网..断网3分钟...ManagedChannelImpl的对象数7...中途几次查看对象数目一直在飙升
3. 修改...先链接collector..再断网..断网3分钟...ManagedChannelImpl的对象数3...中途几次查看对象数目1,2,3,2,3...有shutdown关闭释放的迹象，猜测但可能存在shutdown慢或者shutdown失败...

xuet0ng on 22 Jan 2019

修改方法是...reconnect的时候，将原来channel.shutdownNow()

xuet0ng on 22 Jan 2019

@xuet0ng Do you want to raise a pull request to fix this? You are the one found this and found the solution. You should take the credit.

wu-sheng on 22 Jan 2019

@wu-sheng Thanks for the invitation, but unfortunately, sorry and I am afraid I can't do this. Not familiar with Skywalking ,the first time I began being familiar with it was the day before yesterday downloaded the code to locate the memory overflow problem . And the project in my company is too tight now, can't guarantee my energy to complete this thing.

xuet0ng on 23 Jan 2019

Sure. I could submit a pr to fix this. Thank you to find the issue and recheck the possible fix.

wu-sheng on 23 Jan 2019

by the way, tried and tested shutdownNow old channel if reconnect...
1st, start & connect the collector normally
2nd, disconnect the network and wait for 10 mins
3rd, reconnect the network, reconnect collector again and wait for 10 mins

in 2nd leaked instances increased very slowly
in 3rd instances reduced to 2, I assumed that "to 1", interesting...

it works, but can not solve all the problems, but maybe it is acceptable.

xuet0ng on 23 Jan 2019

There may be still some uncertain scenario. I will try to improve that codes. And for agent-backend, I prefer to put a load balance between then, and payload of collector should be better, and we could leave the reconnect to grpc itself.

wu-sheng on 23 Jan 2019

Fixed.

wu-sheng on 23 Jan 2019

❤1 👍1

Was this page helpful?

0 / 5 - 0 ratings