Arthas: Arthas Attach 后报错,已经占用的端口未释放

Created on 27 Dec 2018  ·  2Comments  ·  Source: alibaba/arthas

  • [x] 我已经在 issues 里搜索,没有重复的issue。

环境信息

  • arthas-boot.jar 或者 as.sh 的版本: 3.0.5
  • Arthas 版本: 3.0.5
  • 操作系统版本: Red Hat 4.4.7-11
  • 目标进程的JVM版本: Java HotSpot(TM) 64-Bit Server 1.7.0_80

重现问题的步骤

启动一个应用,该应用设置的PermGen space不足以加载应用中所有的类,但启动初期只加载少部分类,不会OutOfMemoryError。用Arthas去Attach这个进程,产生错误
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.7.0_80]
at java.lang.ClassLoader.defineClass(ClassLoader.java:800) ~[na:1.7.0_80]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.7.0_80]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) ~[na:1.7.0_80]
at java.net.URLClassLoader.access$100(URLClassLoader.java:71) ~[na:1.7.0_80]
at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[na:1.7.0_80]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_80]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_80]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_80]
at com.taobao.arthas.agent.ArthasClassloader.loadClass(ArthasClassloader.java:26) ~[arthas-agent.jar:na]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_80]
at com.taobao.arthas.core.shell.term.impl.Helper.loadKeymap(Helper.java:20) ~[arthas-core.jar:3.0.5]
at com.taobao.arthas.core.shell.term.impl.TelnetTermServer$1.accept(TelnetTermServer.java:51) ~[arthas-core.jar:3.0.5]
at com.taobao.arthas.core.shell.term.impl.TelnetTermServer$1.accept(TelnetTermServer.java:48) ~[arthas-core.jar:3.0.5]
at io.termd.core.telnet.TelnetTtyConnection.checkAccept(TelnetTtyConnection.java:174) ~[arthas-core.jar:3.0.5]
at io.termd.core.telnet.TelnetTtyConnection.onOpen(TelnetTtyConnection.java:165) ~[arthas-core.jar:3.0.5]
at io.termd.core.telnet.TelnetConnection.onInit(TelnetConnection.java:63) ~[arthas-core.jar:3.0.5]
at io.termd.core.telnet.netty.TelnetChannelHandler.channelActive(TelnetChannelHandler.java:52) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) ~[arthas-core.jar:3.0.5]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1422) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) ~[arthas-core.jar:3.0.5]
at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:941) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:518) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) ~[arthas-core.jar:3.0.5]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) ~[arthas-core.jar:3.0.5]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[arthas-core.jar:3.0.5]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[arthas-core.jar:3.0.5]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) ~[arthas-core.jar:3.0.5]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[arthas-core.jar:3.0.5]

此后再用arthas去attach这台物理机上其他正常的进程,产生错误
[ERROR] Target process 24782 is not the process using port 3658, you will connect to an unexpected process.
[ERROR] If you still want to attach target process 24782, Try to set a different telnet port by using --telnet-port argument.
[ERROR] Or try to shutdown the process 20077 using the telnet port first.

此时去看这两个端口
[root@xxx arthas]# lsof -i:3658
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 20077 root 202u IPv4 1547440072 0t0 TCP xxx :ps-ams (LISTEN)
[root@xxx arthas]# lsof -i:8563
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 20077 root 227u IPv4 1547440081 0t0 TCP xxx :8563 (LISTEN)
正是被之前报java.lang.OutOfMemoryError: PermGen space时的那个进程所占据,没有释放。

期望的结果

Attach 失败后能后及时释放端口,并移除所有在目标进程上的产生的影响。

question-answered

Most helpful comment

启动时指定不同的端口

java -jar arthas-boot.jar --telnet-port 9999 --http-port -1 ,然后选择进程B。这样子两个进程都分别有运行Arthas。

All 2 comments

启动时指定不同的端口

java -jar arthas-boot.jar --telnet-port 9999 --http-port -1 ,然后选择进程B。这样子两个进程都分别有运行Arthas。

@humiao8sz 因为OOM而类加载失败,这个不好处理。因为即使catch了这个异常,netty在shutdown时,也有可能要加载新的类,然后还是OOM而失败。

Was this page helpful?
0 / 5 - 0 ratings

Related issues

hengyunabc picture hengyunabc  ·  3Comments

kylixs picture kylixs  ·  4Comments

joey-happy picture joey-happy  ·  3Comments

hengyunabc picture hengyunabc  ·  5Comments

EternalHao picture EternalHao  ·  4Comments