Arthas: 无法trace 和watch加载skywalking7 的jar中的类方法

Created on 24 Apr 2020 · 16Comments · Source: alibaba/arthas

[ ] 我已经在 issues 里搜索，没有重复的issue。

环境信息

arthas-boot.jar 或者 as.sh 的版本： 3.2.0
Arthas 版本: 3.2.0
操作系统版本: centos7 x64 1810
目标进程的JVM版本: 1.8
执行arthas-boot的版本: 3.2

重现问题的步骤

trace 类方法报错

期望的结果

What do you expected from the above steps？

展示调用栈及耗时

实际运行的结果

实际运行结果，最好有详细的日志，异常栈。尽量贴文本。

Error during processing the command: null

class类被skywalking做了增强.

jad class结果:

@RestController
public class ItemController
implements ItemApi,
EnhancedInstance,
EnhancedInstance {
@Autowired
private IHolidayItemFacade holidayItemFacade;
private volatile Object _$EnhancedClassField_ws;
public static volatile /* synthetic / InstMethodsInter delegate$7gn76d1;
public static volatile / synthetic / InstMethodsInter delegate$lv6d7p1;
public static volatile / synthetic / ConstructorInter delegate$27t3dk1;
private static final / synthetic / Method cachedValue$BGsZ60W7$h0kumm0;
private static final / synthetic / Method cachedValue$BGsZ60W7$mheqqj2;
private static final / synthetic / Method cachedValue$BGsZ60W7$7b1jhl2;
private static final / synthetic / Method cachedValue$BGsZ60W7$h90el41;
private static final / synthetic / Method cachedValue$BGsZ60W7$o2lomh2;
private static final / synthetic / Method cachedValue$BGsZ60W7$2cp92d3;
private static final / synthetic / Method cachedValue$BGsZ60W7$trthg52;
private static final / synthetic / Method cachedValue$BGsZ60W7$etp7162;
private static final / synthetic / Method cachedValue$BGsZ60W7$20eg8u0;
private static final / synthetic / Method cachedValue$BGsZ60W7$5ftnfq2;
private static final / synthetic / Method cachedValue$BGsZ60W7$pg6p5u1;
private static final / synthetic / Method cachedValue$BGsZ60W7$p6onuj0;
private static final / synthetic / Method cachedValue$BGsZ60W7$avq2ph1;
private static final / synthetic / Method cachedValue$BGsZ60W7$qkdlei1;
private static final / synthetic / Method cachedValue$BGsZ60W7$jb5tsd2;
private static final / synthetic / Method cachedValue$BGsZ60W7$5tcatp1;
private static final / synthetic / Method cachedValue$BGsZ60W7$nird923;
private static final / synthetic / Method cachedValue$BGsZ60W7$n3hau21;
private static final / synthetic / Method cachedValue$BGsZ60W7$hu9rep3;
private static final / synthetic / Method cachedValue$BGsZ60W7$kootqq0;
private static final / synthetic / Method cachedValue$BGsZ60W7$png6703;
private static final / synthetic / Method cachedValue$BGsZ60W7$ijv26b1;
private static final / synthetic / Method cachedValue$BGsZ60W7$trbqar3;
private static final / synthetic / Method cachedValue$BGsZ60W7$j92umg1;
private static final / synthetic / Method cachedValue$BGsZ60W7$itutci2;
private static final / synthetic / Method cachedValue$BGsZ60W7$r72ggd2;
public static volatile / synthetic / InstMethodsInter delegate$8v6lb91;
public static volatile / synthetic / InstMethodsInter delegate$a9tapa0;
public static volatile / synthetic / ConstructorInter delegate$i1erl51;
private static final / synthetic / Method cachedValue$v34oxCGE$h0kumm0;
private static final / synthetic / Method cachedValue$v34oxCGE$mheqqj2;
private static final / synthetic / Method cachedValue$v34oxCGE$7b1jhl2;
private static final / synthetic / Method cachedValue$v34oxCGE$h90el41;
private static final / synthetic / Method cachedValue$v34oxCGE$o2lomh2;
private static final / synthetic / Method cachedValue$v34oxCGE$2cp92d3;
private static final / synthetic / Method cachedValue$v34oxCGE$trthg52;
private static final / synthetic / Method cachedValue$v34oxCGE$etp7162;
private static final / synthetic / Method cachedValue$v34oxCGE$20eg8u0;
private static final / synthetic / Method cachedValue$v34oxCGE$5ftnfq2;
private static final / synthetic / Method cachedValue$v34oxCGE$pg6p5u1;
private static final / synthetic / Method cachedValue$v34oxCGE$p6onuj0;
private static final / synthetic / Method cachedValue$v34oxCGE$avq2ph1;
private static final / synthetic / Method cachedValue$v34oxCGE$qkdlei1;
private static final / synthetic / Method cachedValue$v34oxCGE$jb5tsd2;
private static final / synthetic / Method cachedValue$v34oxCGE$5tcatp1;
private static final / synthetic / Method cachedValue$v34oxCGE$nird923;
private static final / synthetic / Method cachedValue$v34oxCGE$n3hau21;
private static final / synthetic / Method cachedValue$v34oxCGE$hu9rep3;
private static final / synthetic / Method cachedValue$v34oxCGE$kootqq0;
private static final / synthetic / Method cachedValue$v34oxCGE$png6703;
private static final / synthetic / Method cachedValue$v34oxCGE$ijv26b1;
private static final / synthetic / Method cachedValue$v34oxCGE$trbqar3;
private static final / synthetic / Method cachedValue$v34oxCGE$j92umg1;
private static final / synthetic / Method cachedValue$v34oxCGE$itutci2;
private static final / synthetic */ Method cachedValue$v34oxCGE$r72ggd2;

public ItemController() {
    this(null);
    delegate$i1erl51.intercept(this, new Object[0]);
}

private /* synthetic */ ItemController(auxiliary.u2IDERyW u2IDERyW2) {
}

}

```
把异常信息贴到这里

2020-04-22 15:55:51 [arthas-binding-thread] INFO c.t.arthas.core.util.ArthasBanner -arthas version: 3.2.0
2020-04-22 15:55:52 [nioEventLoopGroup-2-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x7adbd703] REGISTERED
2020-04-22 15:55:52 [nioEventLoopGroup-2-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x7adbd703] BIND: /127.0.0.1:3658
2020-04-22 15:55:52 [nioEventLoopGroup-2-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x7adbd703, L:/127.0.0.1:3658] ACTIVE
2020-04-22 15:55:52 [nioEventLoopGroup-3-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x0304f924] REGISTERED
2020-04-22 15:55:52 [nioEventLoopGroup-3-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x0304f924] BIND: /127.0.0.1:8563
2020-04-22 15:55:52 [nioEventLoopGroup-3-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x0304f924, L:/127.0.0.1:8563] ACTIVE
2020-04-22 15:55:52 [arthas-binding-thread] INFO c.t.a.core.server.ArthasBootstrap -as-server listening on network=127.0.0.1;telnet=3658;http=8563;timeout=6000;
2020-04-22 15:55:52 [arthas-binding-thread] INFO c.t.a.core.server.ArthasBootstrap -as-server started in 1044 ms
2020-04-22 15:55:52 [nioEventLoopGroup-2-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x7adbd703, L:/127.0.0.1:3658] READ: [id: 0xb03c0d6d, L:/127.0.0.1:3658 - R:/127.0.0.1:49094]
2020-04-22 15:55:52 [nioEventLoopGroup-2-1] INFO i.n.handler.logging.LoggingHandler -[id: 0x7adbd703, L:/127.0.0.1:3658] READ COMPLETE
2020-04-22 15:55:52 [nioEventLoopGroup-2-2] INFO c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc
2020-04-22 15:56:27 [as-command-execute-daemon] ERROR c.t.a.c.s.system.impl.ProcessImpl -Error during processing the command:
java.lang.ClassFormatError: null
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at com.taobao.arthas.core.advisor.Enhancer.enhance(Enhancer.java:303)
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.enhance(EnhancerCommand.java:112)
at com.taobao.arthas.core.command.monitor200.EnhancerCommand.process(EnhancerCommand.java:67)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:371)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

not-a-bug question-answered

Source

tz826

Most helpful comment

提交了一个PR，增加ByteBuddy class cache特性，解决retransform失败的问题。
https://github.com/apache/skywalking/pull/4858

https://github.com/hengyunabc/skywalking-error-demo 这个原型验证通过，可以使用Arthas注入并执行trace命令。

[arthas@42451]$ trace *TestController *
trace *TestController *
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 7) cost in 164 ms, listenerId: 1
`---ts=2020-06-04 15:38:58;thread_name=http-nio-8080-exec-1;id=8e;is_daemon=true;priority=5;TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@c190cfc
    `---[3.650681ms] com.example.demo.TestController:test()
        +---[0.069111ms] com.example.demo.TestController$auxiliary$VUHAiX6C:<init>() #-1
        `---[3.096252ms] org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter:intercept() #-1
            +---[0.019431ms] com.example.demo.TestController:getSkyWalkingDynamicField()
            `---[0.105957ms] com.example.demo.TestController:test$original$83418DTC$accessor$dyYqLLCC()
                `---[0.053324ms] com.example.demo.TestController:test$original$83418DTC() #-1
                    `---[0.010328ms] com.example.demo.TestController:test$original$83418DTC()

kylixs on 4 Jun 2020

👍4

All 16 comments

skywalking 应该是给类增加了 field，导致的问题。skywalking实现机制的问题。

hengyunabc on 24 Apr 2020

skywalking 应该是给类增加了 field，导致的问题。skywalking实现机制的问题。

该如何解决呢，才能正常使用controller

RainElohim on 22 May 2020

@RainElohim 试下这个版本: https://github.com/alibaba/arthas/issues/1184

hengyunabc on 22 May 2020

@RainElohim 试下这个版本: #1184

尝试了一下，还是不可以，报错跟上面楼主的一样，并且用了#1184这里的包之后，执行trace命令后，再执行其他命令会提示Timer already cancelled. 然后无法退出了就，再开个终端用arthas进入同一个进程后执行命令也是这个，不能正常使用了 @hengyunabc 辛苦

RainElohim on 22 May 2020

@RainElohim 具体的Timer already cancelled日志是怎样的？是从哪里打印出来的？可以稳定重现不？

hengyunabc on 22 May 2020

@RainElohim 具体的Timer already cancelled日志是怎样的？是从哪里打印出来的？可以稳定重现不？

arthas错误日志里没有关于这个的，是从arthas控制台打印出来的，就是输入arthas命令后出现的，在我这，按我上面的操作稳定复现，我换了两个进程都是
@hengyunabc

RainElohim on 22 May 2020

可能和 skywalking 生成的字节码里，用了一些类似lambda的代码有关。

对于下面的controller，skywalking在每次reTransform之后，字节码里的函数名都不一样。

@Controller
public class TestController {
    @RequestMapping("/test")
    @ResponseBody
    public String test() {
        return "test";
    }
}

第一次jad结果：

@Controller
public class TestController
implements EnhancedInstance,
EnhancedInstance {
    private volatile Object _$EnhancedClassField_ws;
    public static volatile /* synthetic */ InstMethodsInter delegate$dnkuls0;
    public static volatile /* synthetic */ InstMethodsInter delegate$4sa87m1;
    public static volatile /* synthetic */ ConstructorInter delegate$qe910b1;
    private static final /* synthetic */ Method cachedValue$0cEL6pOE$lt7gtc1;
    public static volatile /* synthetic */ InstMethodsInter delegate$mniv240;
    public static volatile /* synthetic */ InstMethodsInter delegate$2tpiec0;
    public static volatile /* synthetic */ ConstructorInter delegate$qemr0h1;
    private static final /* synthetic */ Method cachedValue$WIXtGH4B$lt7gtc1;

    public TestController() {
        this(null);
        delegate$qemr0h1.intercept((Object)this, new Object[0]);
    }

    private /* synthetic */ TestController(auxiliary.0X7gQAYw x7gQAYw) {
    }

    @RequestMapping(value={"/test"})
    @ResponseBody
    public String test() {
        return (String)delegate$2tpiec0.intercept((Object)this, new Object[0], (Callable)new auxiliary.dAQJyyYq(this), cachedValue$WIXtGH4B$lt7gtc1);
    }

    private /* synthetic */ String test$original$OZfDOP8V() {
        return "test";
    }

    static {
        ClassLoader.getSystemClassLoader().loadClass("org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus").getMethod("initialize", Class.class, Integer.TYPE).invoke(null, TestController.class, -708451193);
        cachedValue$WIXtGH4B$lt7gtc1 = TestController.class.getMethod("test", new Class[0]);
    }

    final /* synthetic */ String test$original$OZfDOP8V$accessor$WIXtGH4B() {
        return this.test$original$OZfDOP8V();
    }
}

第二次：

@Controller
public class TestController
implements EnhancedInstance,
EnhancedInstance {
    private volatile Object _$EnhancedClassField_ws;
    public static volatile /* synthetic */ InstMethodsInter delegate$dnkuls0;
    public static volatile /* synthetic */ InstMethodsInter delegate$4sa87m1;
    public static volatile /* synthetic */ ConstructorInter delegate$qe910b1;
    private static final /* synthetic */ Method cachedValue$0cEL6pOE$lt7gtc1;
    public static volatile /* synthetic */ InstMethodsInter delegate$8mqi1i0;
    public static volatile /* synthetic */ InstMethodsInter delegate$igef700;
    public static volatile /* synthetic */ ConstructorInter delegate$hqtp9d0;
    private static final /* synthetic */ Method cachedValue$6iYRJM1N$lt7gtc1;

    public TestController() {
        this(null);
        delegate$hqtp9d0.intercept((Object)this, new Object[0]);
    }

    private /* synthetic */ TestController(auxiliary.YL8stOUn yL8stOUn) {
    }

    @RequestMapping(value={"/test"})
    @ResponseBody
    public String test() {
        return (String)delegate$igef700.intercept((Object)this, new Object[0], (Callable)new auxiliary.5212MTs4(this), cachedValue$6iYRJM1N$lt7gtc1);
    }

    private /* synthetic */ String test$original$FiEFBU6M() {
        return "test";
    }

    static {
        ClassLoader.getSystemClassLoader().loadClass("org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus").getMethod("initialize", Class.class, Integer.TYPE).invoke(null, TestController.class, -1004178659);
        cachedValue$6iYRJM1N$lt7gtc1 = TestController.class.getMethod("test", new Class[0]);
    }

    final /* synthetic */ String test$original$FiEFBU6M$accessor$6iYRJM1N() {
        return this.test$original$FiEFBU6M();
    }
}

可以看到，(Callable)new auxiliary.5212MTs4(this) 这里总是变化，这里可能是jvm内部verify失败的原因。

参考： lambda表达式导致arthas无法redefine的问题 https://zhuanlan.zhihu.com/p/91756092

hengyunabc on 23 May 2020

可能和 skywalking 生成的字节码里，用了一些类似lambda的代码有关。

对于下面的controller，skywalking在每次reTransform之后，字节码里的函数名都不一样。

@Controller
public class TestController {
  @RequestMapping("/test")
  @ResponseBody
  public String test() {
      return "test";
  }
}

第一次jad结果：

@Controller
public class TestController
implements EnhancedInstance,
EnhancedInstance {
    private volatile Object _$EnhancedClassField_ws;
    public static volatile /* synthetic */ InstMethodsInter delegate$dnkuls0;
    public static volatile /* synthetic */ InstMethodsInter delegate$4sa87m1;
    public static volatile /* synthetic */ ConstructorInter delegate$qe910b1;
    private static final /* synthetic */ Method cachedValue$0cEL6pOE$lt7gtc1;
    public static volatile /* synthetic */ InstMethodsInter delegate$mniv240;
    public static volatile /* synthetic */ InstMethodsInter delegate$2tpiec0;
    public static volatile /* synthetic */ ConstructorInter delegate$qemr0h1;
    private static final /* synthetic */ Method cachedValue$WIXtGH4B$lt7gtc1;

    public TestController() {
        this(null);
        delegate$qemr0h1.intercept((Object)this, new Object[0]);
    }

    private /* synthetic */ TestController(auxiliary.0X7gQAYw x7gQAYw) {
    }

    @RequestMapping(value={"/test"})
    @ResponseBody
    public String test() {
        return (String)delegate$2tpiec0.intercept((Object)this, new Object[0], (Callable)new auxiliary.dAQJyyYq(this), cachedValue$WIXtGH4B$lt7gtc1);
    }

    private /* synthetic */ String test$original$OZfDOP8V() {
        return "test";
    }

    static {
        ClassLoader.getSystemClassLoader().loadClass("org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus").getMethod("initialize", Class.class, Integer.TYPE).invoke(null, TestController.class, -708451193);
        cachedValue$WIXtGH4B$lt7gtc1 = TestController.class.getMethod("test", new Class[0]);
    }

    final /* synthetic */ String test$original$OZfDOP8V$accessor$WIXtGH4B() {
        return this.test$original$OZfDOP8V();
    }
}

第二次：

@Controller
public class TestController
implements EnhancedInstance,
EnhancedInstance {
    private volatile Object _$EnhancedClassField_ws;
    public static volatile /* synthetic */ InstMethodsInter delegate$dnkuls0;
    public static volatile /* synthetic */ InstMethodsInter delegate$4sa87m1;
    public static volatile /* synthetic */ ConstructorInter delegate$qe910b1;
    private static final /* synthetic */ Method cachedValue$0cEL6pOE$lt7gtc1;
    public static volatile /* synthetic */ InstMethodsInter delegate$8mqi1i0;
    public static volatile /* synthetic */ InstMethodsInter delegate$igef700;
    public static volatile /* synthetic */ ConstructorInter delegate$hqtp9d0;
    private static final /* synthetic */ Method cachedValue$6iYRJM1N$lt7gtc1;

    public TestController() {
        this(null);
        delegate$hqtp9d0.intercept((Object)this, new Object[0]);
    }

    private /* synthetic */ TestController(auxiliary.YL8stOUn yL8stOUn) {
    }

    @RequestMapping(value={"/test"})
    @ResponseBody
    public String test() {
        return (String)delegate$igef700.intercept((Object)this, new Object[0], (Callable)new auxiliary.5212MTs4(this), cachedValue$6iYRJM1N$lt7gtc1);
    }

    private /* synthetic */ String test$original$FiEFBU6M() {
        return "test";
    }

    static {
        ClassLoader.getSystemClassLoader().loadClass("org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus").getMethod("initialize", Class.class, Integer.TYPE).invoke(null, TestController.class, -1004178659);
        cachedValue$6iYRJM1N$lt7gtc1 = TestController.class.getMethod("test", new Class[0]);
    }

    final /* synthetic */ String test$original$FiEFBU6M$accessor$6iYRJM1N() {
        return this.test$original$FiEFBU6M();
    }
}

可以看到，(Callable)new auxiliary.5212MTs4(this) 这里总是变化，这里可能是jvm内部verify失败的原因。

参考： lambda表达式导致arthas无法redefine的问题 https://zhuanlan.zhihu.com/p/91756092

我的jdk是：
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

但是还出现这个问题，那么有没有什么更具体的解决方案

RainElohim on 25 May 2020

@RainElohim 我觉得这个是 skywalking的实现的问题。我测试用 btrace也无办法正常工作。测试脚本：

import org.openjdk.btrace.core.annotations.BTrace;
import org.openjdk.btrace.core.annotations.Injected;
import org.openjdk.btrace.core.annotations.OnMethod;
import org.openjdk.btrace.core.annotations.ProbeClassName;
import org.openjdk.btrace.core.annotations.ProbeMethodName;
import org.openjdk.btrace.core.annotations.Self;
import org.openjdk.btrace.core.annotations.ServiceType;
import org.openjdk.btrace.services.impl.Printer;

@BTrace
public class AllMethods {
    @Injected(ServiceType.RUNTIME)
    private static Printer printer;

    @OnMethod(
            clazz = "com.example.demo.TestController",
            method = "test"
    )
    public static void m(@Self Object o, @ProbeClassName String probeClass, @ProbeMethodName String probeMethod) {
        printer.println("this = " + o);
        printer.print("entered " + probeClass);
        printer.println("." + probeMethod);
    }
}

不清楚为什么 skywalking 的实现里每次函数名都变化。

hengyunabc on 25 May 2020

@RainElohim 另外，每执行一次 jad，即触发一次 java.lang.instrument.Instrumentation.retransformClasses(Class<?>...) ，就会产生一个新的 auxiliary类，显然是有问题的。

执行多次jad之后，用sc命令可以查找到。

[arthas@82549]$ sc com.example.demo.TestController*
com.example.demo.TestController
com.example.demo.TestController$auxiliary$4ZbpDXX9
com.example.demo.TestController$auxiliary$5RcFgXtT
com.example.demo.TestController$auxiliary$DcNiUd0D
com.example.demo.TestController$auxiliary$EgzoyDwo
com.example.demo.TestController$auxiliary$FgWo5SOK
com.example.demo.TestController$auxiliary$J7p46QSU
com.example.demo.TestController$auxiliary$PYsPbDsI
com.example.demo.TestController$auxiliary$Pc3KSnn2
com.example.demo.TestController$auxiliary$PnenKrBq
com.example.demo.TestController$auxiliary$Y6Hg5yqk
com.example.demo.TestController$auxiliary$YS5oCqQ3
com.example.demo.TestController$auxiliary$aNrdhsqk
com.example.demo.TestController$auxiliary$bEhoPh9t
com.example.demo.TestController$auxiliary$eMmayhvM

hengyunabc on 25 May 2020

@RainElohim 另外，每执行一次 jad，即触发一次 java.lang.instrument.Instrumentation.retransformClasses(Class<?>...) ，就会产生一个新的 auxiliary类，显然是有问题的。

执行多次jad之后，用sc命令可以查找到。
[arthas@82549]$ sc com.example.demo.TestController*
com.example.demo.TestController
com.example.demo.TestController$auxiliary$4ZbpDXX9
com.example.demo.TestController$auxiliary$5RcFgXtT
com.example.demo.TestController$auxiliary$DcNiUd0D
com.example.demo.TestController$auxiliary$EgzoyDwo
com.example.demo.TestController$auxiliary$FgWo5SOK
com.example.demo.TestController$auxiliary$J7p46QSU
com.example.demo.TestController$auxiliary$PYsPbDsI
com.example.demo.TestController$auxiliary$Pc3KSnn2
com.example.demo.TestController$auxiliary$PnenKrBq
com.example.demo.TestController$auxiliary$Y6Hg5yqk
com.example.demo.TestController$auxiliary$YS5oCqQ3
com.example.demo.TestController$auxiliary$aNrdhsqk
com.example.demo.TestController$auxiliary$bEhoPh9t
com.example.demo.TestController$auxiliary$eMmayhvM
感谢解答，辛苦了。我去skywalking下面提问下，看看有没有解决办法，https://github.com/apache/skywalking/issues/4820

RainElohim on 25 May 2020

执行多次jad之后，用sc命令可以查找到。

[arthas@82549]$ sc com.example.demo.TestController*
com.example.demo.TestController
com.example.demo.TestController$auxiliary$4ZbpDXX9
com.example.demo.TestController$auxiliary$5RcFgXtT
com.example.demo.TestController$auxiliary$DcNiUd0D
com.example.demo.TestController$auxiliary$EgzoyDwo
com.example.demo.TestController$auxiliary$FgWo5SOK
com.example.demo.TestController$auxiliary$J7p46QSU
com.example.demo.TestController$auxiliary$PYsPbDsI
com.example.demo.TestController$auxiliary$Pc3KSnn2
com.example.demo.TestController$auxiliary$PnenKrBq
com.example.demo.TestController$auxiliary$Y6Hg5yqk
com.example.demo.TestController$auxiliary$YS5oCqQ3
com.example.demo.TestController$auxiliary$aNrdhsqk
com.example.demo.TestController$auxiliary$bEhoPh9t
com.example.demo.TestController$auxiliary$eMmayhvM

那边说是arthas运行时的问题。。。具体看apache/skywalking#4820

RainElohim on 25 May 2020

和arthas无关，提供一个中文 only 的demo： https://github.com/hengyunabc/skywalking-error-demo

hengyunabc on 25 May 2020

https://github.com/raphw/byte-buddy/issues/829
可能是与ByteBuddy的 MethodDelegation方式有关系

参考这篇
https://www.infoq.com/articles/Easily-Create-Java-Agents-with-ByteBuddy/

kylixs on 2 Jun 2020

提交了一个PR，增加ByteBuddy class cache特性，解决retransform失败的问题。
https://github.com/apache/skywalking/pull/4858

https://github.com/hengyunabc/skywalking-error-demo 这个原型验证通过，可以使用Arthas注入并执行trace命令。

[arthas@42451]$ trace *TestController *
trace *TestController *
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 7) cost in 164 ms, listenerId: 1
`---ts=2020-06-04 15:38:58;thread_name=http-nio-8080-exec-1;id=8e;is_daemon=true;priority=5;TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@c190cfc
    `---[3.650681ms] com.example.demo.TestController:test()
        +---[0.069111ms] com.example.demo.TestController$auxiliary$VUHAiX6C:<init>() #-1
        `---[3.096252ms] org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter:intercept() #-1
            +---[0.019431ms] com.example.demo.TestController:getSkyWalkingDynamicField()
            `---[0.105957ms] com.example.demo.TestController:test$original$83418DTC$accessor$dyYqLLCC()
                `---[0.053324ms] com.example.demo.TestController:test$original$83418DTC() #-1
                    `---[0.010328ms] com.example.demo.TestController:test$original$83418DTC()

kylixs on 4 Jun 2020

👍4

SkyWalking: Compatible with other javaagent bytecode processing
https://github.com/apache/skywalking/blob/master/docs/en/FAQ/Compatible-with-other-javaagent-bytecode-processing.md

kylixs on 30 Sep 2020

Was this page helpful?

0 / 5 - 0 ratings