Arthas: can't run on openjdk

[aaronwong1989]

• [x] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

• Arthas version: 3.0.3
• Operating System version: CentOS Linux release 7.5.1804 (Core)
• Java version of target JVM: java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
• Java version of JVM used to attach: java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64

Steps to reproduce this issue

[root@javawa ~]# ./as.sh 24110
Arthas script version: 3.0.4
Calculating attach execution time...
Attaching to 24110 using version 3.0.4...
Start arthas failed, exception stack trace:
java.lang.InternalError: instrument library is missing in target VM
    at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:105)
    at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:84)
    at com.taobao.arthas.core.Arthas.<init>(Arthas.java:25)
    at com.taobao.arthas.core.Arthas.main(Arthas.java:96)
Caused by: com.sun.tools.attach.AgentLoadException: Failed to load agent library
    at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
    ... 3 more
attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.
[root@javawa ~]# cat /root/logs/arthas/arthas.log
cat: /root/logs/arthas/arthas.log: No such file or directory

Expected Result

Actual Result

What actually happens?

If there is an exception, please attach the exception trace:

Just put your stack trace here!

[hengyunabc]

Try to run bash -x as.sh, or try https://alibaba.github.io/arthas/en/install-detail.html#advanced-manual-boot

[aaronwong1989]

i have change my jvm to oraclejdk1.8 and the problem still exists.
when run sh -x as.sh
i got

++ ps -p 24110 -o user=
+ local target_user=root
+ '[' root '!=' root ']'
+ echo 'Calculating attach execution time...'
Calculating attach execution time...
+ attach_jvm 3.0.4
+ local arthas_version=3.0.4
+ local arthas_lib_dir=/root/.arthas/lib/3.0.4/arthas
+ echo 'Attaching to 24110 using version 3.0.4...'
Attaching to 24110 using version 3.0.4...
+ '[' 127.0.0.1 = 127.0.0.1 ']'
+ [[ 3.0.4 > 3.0 ]]
+ /usr/lib/jvm/oracle-jdk-1.8.0_181/bin/java -Djava.awt.headless=true -Xbootclasspath/a:/usr/lib/jvm/oracle-jdk-1.8.0_181/lib/tools.jar -jar /root/.arthas/lib/3.0.4/arthas/arthas-core.jar -pid 24110 -target-ip 127.0.0.1 -telnet-port 3658 -http-port 8563 -core /root/.arthas/lib/3.0.4/arthas/arthas-core.jar -agent /root/.arthas/lib/3.0.4/arthas/arthas-agent.jar
Start arthas failed, exception stack trace:
java.lang.InternalError: instrument library is missing in target VM
    at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:105)
    at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:84)
    at com.taobao.arthas.core.Arthas.<init>(Arthas.java:25)
    at com.taobao.arthas.core.Arthas.main(Arthas.java:96)
Caused by: com.sun.tools.attach.AgentLoadException: Failed to load agent library
    at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
    ... 3 more
+ exit 1
+ '[' 1 -ne 0 ']'
+ exit_on_err 1 'attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.'
+ [[ ! -z attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions. ]]
+ echo 'attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.'
attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.
+ exit 1

[aaronwong1989]

@hengyunabc

[root@javawa arthas]# ll
total 20072
-rw-r--r-- 1 root root  9777184 Sep 28 16:26 arthas-3.0.4-bin.zip
-rw-r--r-- 1 root root     8015 Sep 14 18:52 arthas-agent.jar
-rw-r--r-- 1 root root   103660 Sep 14 18:52 arthas-client.jar
-rw-r--r-- 1 root root 10590270 Sep 14 18:52 arthas-core.jar
-rw-r--r-- 1 root root     2949 Sep 14 18:51 arthas-spy.jar
-rw-r--r-- 1 root root     2430 Sep 14 18:49 as.bat
-rwxr-xr-x 1 root root    14456 Sep 14 18:49 as.sh
-rwxr-xr-x 1 root root      633 Sep 14 18:49 install-local.sh
[root@javawa arthas]# pwd
/root/.arthas/lib/3.0.4/arthas

[hengyunabc]

@aaronwong1989 Try to run:

find /usr/lib/jvm/oracle-jdk-1.8.0_181 | grep 'instrument'

In my ubuntu, result is:

# find /xxx/jdk/jdk1.8.0_171/ | grep 'instrument'
/xxx/jdk/jdk1.8.0_171/jre/lib/amd64/libinstrument.so

[aaronwong1989]

@hengyunabc
yes，libinstrument.so appears

[hengyunabc]

It seems that the target JVM can not load the instrument library.

1. Check if the target JVM version >= 6 (not the JVM that starts arthas)
2. Check if there is libinstrument.so under the target JVM (not the JVM that starts arthas)
3. Try on other machines

[aaronwong1989]

@hengyunabc
It works, thanks very much.

Problem appears when The target JVM is java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64， and The JVM that starts arthas is oracle-jdk-1.8.0_181 .

Initially, there was only one JVM( java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64) on my server, and arthas does not works. then I installed oracle-jdk-1.8.0_181 and set $JAVA_HOME to it, but I did not resatrt the target java process.

There are 2 java process on my server:

* [1]: 11901 /usr/lib/jenkins/jenkins.war
  [2]: 24110 ajax-srv-0.0.1.jar

after restart jenkins (do not restart ajax-srv-0.0.1.jar)，arthas can monitor Jenkins but can't works on process [2]. then I restart ajax-srv-0.0.1.jar, and it works !

the conclusion seems likes : arthas can't run on openjdk ，at least on my machine.

more information

find /usr/lib/jvm/oracle-jdk-1.8.0_181 |grep 'instrument'
/usr/lib/jvm/oracle-jdk-1.8.0_181/jre/lib/amd64/libinstrument.so

find /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 |grep 'instrument'
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/amd64/libinstrument.so

It seems that it has nothing to do with "libinstrument.so".

[aaronwong1989]

Corrections ：Arthas can't monitor java processes that run on The openjdk

[hengyunabc]

@aaronwong1989 有没有比较确定的办法来重现这个问题，我本地开发用的就是 opentjdk，测试mac/linux opent jdk都没有问题。

[oakax]

@hengyunabc 生产环境也是openjdk，第一次运行arthas也是这个问题，重启程序之后再启动arthas就能attach上了...

[jacksparrow414]

和 @oakax 一样，生成环境也是openjdk，第一次运行就会出现这个问题，重启应用程序在启动arthas就可以了