Axonframework: Failed to start bean 'queryHandlerSubscriber'; nested exception is io.grpc.StatusRuntimeException: UNAVAILABLE

[rzcastilho]

Basic information

• Axon Server version: 4.4.3
• Axon Framework version: 4.4.2
• JDK version: 1.8.0_202

Steps to reproduce

Run a client application that connects to Axon Server in a Kubernetes environment. To reproduce you have to update de deployment or scale up the application, after a few times, the error showed in the log below happens and the application cannot starts.

Expected behaviour

Starts application without "Failed to start bean 'queryHandlerSubscriber'" error.

Actual behaviour

The error "Failed to start bean 'queryHandlerSubscriber'; nested exception is io.grpc.StatusRuntimeException: UNAVAILABLE" is returned, and that application cannot start.

LOG:

2020-10-22 14:11:53  - [] INFO o.s.s.q.LocalDataSourceJobStore - JobStoreCMT initialized.
2020-10-22 14:11:56  - [] INFO o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-10-22 14:12:03  - [] WARN ConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'queryHandlerSubscriber'; nested exception is io.grpc.StatusRuntimeException: UNAVAILABLE
2020-10-22 14:12:03  - [] INFO o.s.s.c.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'
2020-10-22 14:12:03  - [] INFO o.s.s.q.SchedulerFactoryBean - Shutting down Quartz Scheduler
2020-10-22 14:12:05  - [] INFO j.LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
2020-10-22 14:12:08  - [] INFO ConditionEvaluationReportLoggingListener -

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-10-22 14:12:08  - [] ERROR o.s.b.SpringApplication - Application run failed
org.springframework.context.ApplicationContextException: Failed to start bean 'queryHandlerSubscriber'; nested exception is io.grpc.StatusRuntimeException: UNAVAILABLE
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162) ~[spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) [spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
    at br.com.kroton.captacao.simulacao.Application.main(Application.java:18) [classes!/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:?]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:?]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [app.jar:?]
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [app.jar:?]
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE
    at io.axoniq.axonserver.connector.impl.AxonServerManagedChannel$FailingCall.sendMessage(AxonServerManagedChannel.java:428) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.grpc.stub.ClientCalls$CallToStreamObserverAdapter.onNext(ClientCalls.java:346) ~[grpc-stub-1.28.1.jar!/:1.28.1]
    at io.axoniq.axonserver.connector.impl.SynchronizedRequestStream.lambda$onNext$0(SynchronizedRequestStream.java:77) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.axoniq.axonserver.connector.impl.SynchronizedRequestStream.inLock(SynchronizedRequestStream.java:95) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.axoniq.axonserver.connector.impl.SynchronizedRequestStream.onNext(SynchronizedRequestStream.java:77) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.lambda$sendInstruction$6(QueryChannelImpl.java:271) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.axoniq.axonserver.connector.impl.ObjectUtils.doIfNotNull(ObjectUtils.java:56) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.sendInstruction(QueryChannelImpl.java:264) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at io.axoniq.axonserver.connector.query.impl.QueryChannelImpl.registerQueryHandler(QueryChannelImpl.java:240) ~[axonserver-connector-java-4.4.1.jar!/:4.4.1]
    at org.axonframework.axonserver.connector.query.AxonServerQueryBus.subscribe(AxonServerQueryBus.java:186) ~[axon-server-connector-4.4.2.jar!/:4.4.2]
    at org.axonframework.queryhandling.annotation.AnnotationQueryHandlerAdapter.subscribe(AnnotationQueryHandlerAdapter.java:120) ~[axon-messaging-4.4.2.jar!/:4.4.2]
    at org.axonframework.queryhandling.annotation.AnnotationQueryHandlerAdapter.lambda$subscribe$0(AnnotationQueryHandlerAdapter.java:92) ~[axon-messaging-4.4.2.jar!/:4.4.2]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_202]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_202]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_202]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_202]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_202]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_202]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_202]
    at org.axonframework.queryhandling.annotation.AnnotationQueryHandlerAdapter.subscribe(AnnotationQueryHandlerAdapter.java:94) ~[axon-messaging-4.4.2.jar!/:4.4.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
    at org.axonframework.spring.config.AbstractAnnotationHandlerBeanPostProcessor$AdapterIntroductionInterceptor.invoke(AbstractAnnotationHandlerBeanPostProcessor.java:287) ~[axon-spring-4.4.2.jar!/:4.4.2]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    at br.com.kroton.captacao.simulacao.query.SimulacaoBeneficioProjection$$EnhancerBySpringCGLIB$$2c4b8203.subscribe(<generated>) ~[classes!/:?]
    at org.axonframework.spring.config.QueryHandlerSubscriber.lambda$start$0(QueryHandlerSubscriber.java:73) ~[axon-spring-4.4.2.jar!/:4.4.2]
    at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608) ~[?:1.8.0_202]
    at org.axonframework.spring.config.QueryHandlerSubscriber.start(QueryHandlerSubscriber.java:73) ~[axon-spring-4.4.2.jar!/:4.4.2]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) ~[spring-context-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
    ... 22 more

[abuijze]

Thanks for reporting the issue.

The issue is actually in the AxonServer Connector, but it's good to have a reference to it here as well, as most of the connector's functionality is hidden behind Axon Framework.

There is PR to resolve this issue here: https://github.com/AxonIQ/axonserver-connector-java/pull/24

[smcvb]

Given the issue resides in the axonserver-connector-java project, that makes this issue a pointer to upgrade to this newer version of the connector once it has been released.

[smcvb]

Since the axonserver-connector-java issue #24 has been merged, as well as that the version has been incremented for Axon Framework in commit 1da001ef66c1a9a2f1dae65f69123498f9220f00, I feel we can close this issue as resolved.