Dubbo: 奇怪的异常（2.5.3）

[AtomGuo]

不是必现，但是偶尔会持续几分钟，dubbo是2.5.3版本，困扰好久了

java.lang.NullPointerException: null
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:34)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy1.getAtp(proxy1.java)
at com.mryx.grampus.service.impl.GmsShopGoodsServiceImpl.getCategoryGoods(GmsShopGoodsServiceImpl.java:101)
at com.mryx.grampus.controller.ProductController.getGoods(ProductController.java:178)
at sun.reflect.GeneratedMethodAccessor375.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

[chickenlj]

dubbo什么版本？异常是伴随timeout一起出现的吗？如果是的话可能是业务参数的toString()抛出NPE

[qinliujie]

2.5.4已经被修复

[suhlmsn]

牛逼

[tooapi]

怎么解决的 怎么出现的 为什么错误行在 34 什么问题 我这边也出现了 版本 2.4.10

[qinliujie]

@yanzhijun 可以贴一下你的异常堆栈么？

[JarkimZhu]

@qinliujie 你好，我们现在使用的2.5.3版本，升级到最新版会有很大风险，请问2.5.3有没有办法规避啊？

[qinliujie]

@JarkimZhu 目前我们这边也没有定位到具体原因是什么。。主要是我这边复现不了。。。不清楚是不是某个字节码增强了还是怎么回事

[JarkimZhu]

@chickenlj @qinliujie 我看了DefaultFuture是new出来的，应该没有用什么字节码增强技术啊，com.alibaba.dubbo.common.bytecode.proxy2和org.springframework.aop.framework.CglibAopProxy会影响吗？

[qinliujie]

Dump 动态字节码

1､编写一个 Filter

import sun.jvm.hotspot.oops.InstanceKlass;
import sun.jvm.hotspot.tools.jcore.ClassFilter;

/**
 * @author qinliujie
 * @date 2017/09/26
 */
public class Filter implements ClassFilter {
    public boolean canInclude(InstanceKlass instanceKlass) {
        String klassName = instanceKlass.getName().asString();
        return klassName.contains("DefaultFuture");
    }
}

注意不要加 包名

2､在消费者端用 javac 编译

javac -cp "$JAVA_HOME/lib/sa-jdi.jar" Filter.java

3､编写 attach agent，跟 Filter 一个目录

/**
 * @author qinliujie
 * @date 2017/09/26
 */
public class Demo {
    public static void main(String[] args){
        System.setProperty("sun.jvm.hotspot.tools.jcore.filter","Filter");
        ClassDump.main(args);
    }
}

4､编译

javac -cp "$JAVA_HOME/lib/sa-jdi.jar" Demo.java

5､执行命令

sudo java -cp ".:$JAVA_HOME/lib/sa-jdi.jar" Demo 26763(消费端进程号)

6､把拿到的字节码发回给我 [email protected]

[qinliujie]

@AtomGuo @yanzhijun @JarkimZhu 麻烦各位遇到问题的，按上面的步骤发送一下相应的字节码给我吧

[qinliujie]

目前已经定位到问题，麻烦各位确认一下是否用到了

// Method com/tingyun/agent/proxy/AgentProxy.getAgent:()Lcom/tingyun/agent/proxy/Agent;

public java.lang.Object get(int) throws com.alibaba.dubbo.remoting.RemotingException;
    descriptor: (I)Ljava/lang/Object;
    flags: ACC_PUBLIC
    Code:
      stack=6, locals=12, args_size=2
         0: aconst_null
         1: astore_2
         2: getstatic     #417                // Field com/tingyun/agent/proxy/AgentProxy.instrumentation:Lcom/tingyun/agent/proxy/Instrumentation;
         5: aload_0
         6: sipush        7127
         9: aconst_null
        10: bipush        6
        12: invokeinterface #423,  5          // InterfaceMethod com/tingyun/agent/proxy/Instrumentation.createTracer:(Ljava/lang/Object;ILjava/lang/String;I)Lcom/tingyun/agent/proxy/ExitTracer;
        17: astore_2
        18: goto          22
        21: pop
        22: invokestatic  #427                // Method com/tingyun/agent/proxy/AgentProxy.getAgent:()Lcom/tingyun/agent/proxy/Agent;
        25: invokeinterface #433,  1          // InterfaceMethod com/tingyun/agent/proxy/Agent.getAction:()Lcom/tingyun/agent/proxy/Action;
        30: invokeinterface #439,  1          // InterfaceMethod com/tingyun/agent/proxy/Action.getTracedMethod:()Lcom/tingyun/api/agent/TracedMethod;
        35: checkcast     #441                // class com/tingyun/agent/proxy/TracedMethod
             306: athrow
      Exception table:
         from    to  target type
             2    22    21   Class java/lang/Throwable
           112   161   174   Class java/lang/InterruptedException
           112   161   186   any
           174   188   186   any
            73   253   256   Class java/lang/Exception
             2   294   294   Class java/lang/Throwable
      LineNumberTable:
        line 34: 22
        line 88: 79
        line 89: 84
        line 91: 89
        line 92: 97
        line 93: 102
        line 95: 112
        line 96: 120
        line 97: 137
        line 98: 158
        line 104: 161
        line 105: 171
        line 101: 174
        line 102: 176
        line 104: 186
        line 106: 201
        line 107: 209
        line 110: 243
      LocalVariableTable:
        Start  Length  Slot  Name   Signature
          176      10     9     e   Ljava/lang/InterruptedException;
          102     141     7 start   J
           79     172     6  this   Lcom/alibaba/dubbo/remoting/exchange/support/DefaultFuture;
           79     172     5 timeout   I
          258      19    11     e   Ljava/lang/Exception;
           22     272     0  this   Lcom/alibaba/dubbo/remoting/exchange/support/DefaultFuture;
           22     272     1 timeout   I
           42     252     3    tm   Lcom/tingyun/agent/proxy/TracedMethod;
           73     221     4 result   Ljava/lang/Object;
    Exceptions:
      throws com.alibaba.dubbo.remoting.RemotingException

[qinliujie]

感谢 @JarkimZhu 大力配合