Dubbo: nacos v1.1.4, dubbo v2.7.6.SNAPSHOT "No provider available from registry xxx"

[vergilyn]

• [ ] I have searched the issues of this repository and believe that this is not a duplicate.
• [ ] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

• Dubbo version: 2.7.6-SNAPSHOT(aliyun maven镜像，也用源码的 2.7.6.release分支测试过)
• Operating System version: win10
• Java version: 1.8.0_171

Steps to reproduce this issue

测试代码：https://github.com/apache/dubbo-samples/tree/master/java/dubbo-samples-nacos/dubbo-samples-nacos-registry

dubbo版本修改为"2.7.6.SNAPSHOT"

Pls. provide [GitHub address] to reproduce this issue.

Expected Result

Actual Result

(IP地址手动改成了 127.0.0.1)

Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'annotatedConsumer': Injection of @Reference dependencies is failed; nested exception is java.lang.IllegalStateException: Failed to check the status of the service org.apache.dubbo.samples.api.GreetingService. No provider available for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 from the url nacos://localhost:8848/org.apache.dubbo.registry.RegistryService?application=nacos-registry-demo-consumer&dubbo=2.0.2&init=false&interface=org.apache.dubbo.samples.api.GreetingService&methods=sayHello&pid=19028&register.ip=127.0.0.1&release=2.7.6-SNAPSHOT&revision=1.0.0&side=consumer&sticky=false&timeout=3000&timestamp=1584584637337&version=1.0.0 to the consumer 127.0.0.1 use dubbo version 2.7.6-SNAPSHOT
    at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor.postProcessPropertyValues(AbstractAnnotationBeanPostProcessor.java:146)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1268)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
    at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:84)
    at org.apache.dubbo.samples.ConsumerBootstrap.main(ConsumerBootstrap.java:32)
Caused by: java.lang.IllegalStateException: Failed to check the status of the service org.apache.dubbo.samples.api.GreetingService. No provider available for the service org.apache.dubbo.samples.api.GreetingService:1.0.0 from the url nacos://localhost:8848/org.apache.dubbo.registry.RegistryService?application=nacos-registry-demo-consumer&dubbo=2.0.2&init=false&interface=org.apache.dubbo.samples.api.GreetingService&methods=sayHello&pid=19028&register.ip=127.0.0.1&release=2.7.6-SNAPSHOT&revision=1.0.0&side=consumer&sticky=false&timeout=3000&timestamp=1584584637337&version=1.0.0 to the consumer 127.0.0.1 use dubbo version 2.7.6-SNAPSHOT
    at org.apache.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:349)
    at org.apache.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:258)
    at org.apache.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:158)
    at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor.getOrCreateProxy(ReferenceAnnotationBeanPostProcessor.java:274)
    at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor.doGetInjectedBean(ReferenceAnnotationBeanPostProcessor.java:143)
    at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor.getInjectedObject(AbstractAnnotationBeanPostProcessor.java:359)
    at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor$AnnotatedFieldElement.inject(AbstractAnnotationBeanPostProcessor.java:539)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
    at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor.postProcessPropertyValues(AbstractAnnotationBeanPostProcessor.java:142)
    ... 12 more

备注

1. nacos中已存在 provider注册的服务 serviceName = "providers:org.apache.dubbo.samples.api.GreetingService:1.0.0:"

2. 启动consumer时，NacosRegistry#doSubscribe(...)为了兼容会创建2个 serviceNames

3. providers:org.apache.dubbo.samples.api.GreetingService:1.0.0:
4. providers:org.apache.dubbo.samples.api.GreetingService:1.0.0

并且进行了 NacosRegistry#subscribeEventListener(...) 监听和订阅 nacos。

但是因为 provider 其实并不存在 providers:org.apache.dubbo.samples.api.GreetingService:1.0.0，
所以（nacos 回调的 instances = 0）NacosRegistry#notifySubscriber(URL , NotifyListener , Collection<Instance> )中会创建一个 empty 的 protocol。
又因为只有1个provider，所以RegistryDirectory#refreshInvoker()中满足了 forbidden 的条件，所以invokers被destroy....

[tswstarplanet]

这个应该不是empty协议的问题，因为ZookeepRergistry里也是这么做的

[vergilyn]

这个应该不是empty协议的问题，因为ZookeepRergistry里也是这么做的

理解不能..😅
debug 确实是因为 dubbo 2.7.6为了兼容历史，创建了一个未被提供的 serviceName = "providers:org.apache.dubbo.samples.api.GreetingService:1.0.0"。（被正确提供的 serviceName 后面多一个 冒号）

// org.apache.dubbo.registry.nacos.NacosRegistry#getServiceNames0
private Set<String> getServiceNames0(URL url) {
        NacosServiceName serviceName = createServiceName(url);

        final Set<String> serviceNames;

        if (serviceName.isConcrete()) { // is the concrete service name
            serviceNames = new LinkedHashSet<>();
            serviceNames.add(serviceName.toString());

            /* vergilyn-question, 2020-03-18 >>>> 注释，由于"no provider available"
             *   https://github.com/apache/dubbo/issues/5871
             *   https://github.com/apache/dubbo/issues/5885
             */
            // Add the legacy service name since 2.7.6
            // serviceNames.add(getLegacySubscribedServiceName(url));
        } else {
            serviceNames = filterServiceNames(serviceName);
        }

        return serviceNames;
    }

然后由于nacos确实不存在这个serviceName，其 instances == empty，刚好满足创建 empty protocol的条件。接着也满足 integration.RegistryDirectory#refreshInvoker(...) forbidden 的条件，所以最终并没有可用的invokers.

ps. 大佬通过 dubbo-sample 修改dubbo版本复现bug了吗？

[laddcn]

看提交记录是这部分代码是解决 https://github.com/apache/dubbo/issues/5442 引入的

[laddcn]

我提了个解决方法 https://github.com/apache/dubbo/pull/5902 ，现在可以用

[laddcn]

看起来问题的原因是，为了解决 https://github.com/apache/dubbo/issues/5442 里面描述的问题，为了兼容老版本的nacos服务（老版本的serviceName应该是不带 :: 的，比如 providers:org.apache.dubbo.samples.api.GreetingService:1.0.0），消费者会同时监听新老2个版本的serviceName。在生产者不存在多dubbo版本的情况下，一般只会有一个serviceName在Nacos中是存在的（要么老，要么新）。这样就会导致另外一个serviceName对应的健康invoker示例数为0，进而导致服务会被forbidden，该接口整体不可用，发生该issue中的bug。我现在在消费者订阅的时候，会对兼容模式的invoker示例数进行判断，如果新老serviceName有不存在生产者的情况，就不订阅对应的serviceName。现在看是能解决问题的，下周再验证完善下。第一感觉Nacos注册中心里面的代码还不够健壮，等有时间好好研究研究