Spring-cloud-netflix: Zuul Forwarding Error although Eureka instances are up

[tminussi]

My current architecture consists of an Eureka Server, a Zuul Gateway and one Microservice (let's call it Service A) which usually has 2 instances running. This error does not happen if I start all the instances upfront but if I kill one of the instances of Service A instead.

So to reproduce the problem, I start all my services (Eureka, Zuul and Service A) and start sending requests to Zuul, which will proxy incoming requests to any running instance of Service A. Up to this point, Ribbon is load balancing the requests properly and both instances are responding.

Scenario 1: Now, I kill one of the instances of Service A and start sending requests through Zuul again. Every two requests, one will fail due to the Round Robin strategy (which hits the dead instance), so I would say this behaviour, although annoying (because Eureka takes 30+ seconds to recognise a given instance is dead), is still expected and acceptable.

Scenario 2: After going through Scenario 1, I kill the remaining instance for Service A and start two instances again. I wait until Eureka registers both instances (to make sure they are registered, I go to Eureka's homepage, click in one of the just started instances of Service A and I successfully send requests to it) and then I finally I send a request to Zuul so it can proxy to one of the live instances of Service A. Zuul will now start behaving weirdly and throwing Internal Server Errors (GENERAL). It will take (usually) more than one minute for Zuul to recognise that there are two instances of Service A running again. If I send too many requests, I will get a SHORTCIRCUIT error, which kind of makes sense because Zuul thinks both instances are down.

I have googled around, asked @spencergibb some questions on gitter (I was advised to include Spring Retry on the classpath, which I did) and still no luck. I have tried many different configurations and the behaviour is always the same.

Even though I think this is a problem with Zuul, it could also have something to do with Eureka (although I was able to access the instances through Eureka's interface without any issues) not being able to proxy Zuul requests when looking up for the service name.

This is how the instances looked like in Eureka when the error was thrown:

And this was Zuul's response:

In the end, I think that changing the LB strategy on Zuul could fix the problem, but I did not find a way to do it (only found how to do it when Eureka is not being used)

I will leave the stack trace here as well as the project I am testing. Again, I tried many different configurations, so some of them may not make sense anymore, so if you can please tell me what could be wrong I would be more than happy to try.

Project: https://github.com/tminussi/spring-cloud-masterpiece

Stacktrace:

com.netflix.zuul.exception.ZuulException: Forwarding error
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:189) ~[spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:164) ~[spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:112) ~[spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:117) ~[zuul-core-1.3.1.jar:1.3.1]
    at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193) ~[zuul-core-1.3.1.jar:1.3.1]
    at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157) ~[zuul-core-1.3.1.jar:1.3.1]
    at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118) ~[zuul-core-1.3.1.jar:1.3.1]
    at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.3.1.jar:1.3.1]
    at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.3.1.jar:1.3.1]
    at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.3.1.jar:1.3.1]
    at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:165) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44) [spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) [spring-webmvc-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90) [spring-boot-actuator-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:155) [spring-boot-actuator-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:123) [spring-boot-actuator-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108) [spring-boot-actuator-2.0.4.RELEASE.jar:2.0.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.32.jar:8.5.32]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: station
    at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:483) ~[ribbon-loadbalancer-2.2.5.jar:2.2.5]
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184) ~[ribbon-loadbalancer-2.2.5.jar:2.2.5]
    at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) ~[ribbon-loadbalancer-2.2.5.jar:2.2.5]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.subscribe(Observable.java:10247) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.subscribe(Observable.java:10214) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.observables.BlockingObservable.single(BlockingObservable.java:341) ~[rxjava-1.2.0.jar:1.2.0]
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112) ~[ribbon-loadbalancer-2.2.5.jar:2.2.5]
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:186) ~[spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:51) ~[spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302) ~[hystrix-core-1.5.12.jar:1.5.12]
    at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298) ~[hystrix-core-1.5.12.jar:1.5.12]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.unsafeSubscribe(Observable.java:10151) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.subscribe(Observable.java:10247) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.Observable.subscribe(Observable.java:10214) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51) ~[rxjava-1.2.0.jar:1.2.0]
    at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411) ~[rxjava-1.2.0.jar:1.2.0]
    at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378) ~[hystrix-core-1.5.12.jar:1.5.12]
    at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344) ~[hystrix-core-1.5.12.jar:1.5.12]
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:159) ~[spring-cloud-netflix-zuul-2.0.0.RELEASE.jar:2.0.0.RELEASE]
    ... 65 common frames omitted

Thank you for your attention

[ryanjbaxter]

I have potential reason for why you could be running into
Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: station

in scenario 2.

The above exception means that Ribbon has not server list for the service station. This could happen in scenario 2 depending on the timing of when you shut down the 2 running instances and then start them back up. If Zuul happens to refresh the server list after you shut down the 2 running instances but before you start them back up then it will think there are no running instances of station until its next refresh from Eureka. In the mean time any requests made to station will result in the above error. There is really nothing we can do about this.

Also in addition to putting spring retry on the classpath of zuul you will need to set zuul.retryable=true. This should eliminate scenario 1 from ever happening.

[tminussi]

@ryanjbaxter thanks for getting back to me.

For dev environments, I have been playing with some properties that have lowered the renewal threshold in order to cover scenario 2, so Zuul won't take as long to realise that the instances are back online. At least I now know why the instances were not being recognized by Zuul. Thanks!

Regarding zuul.retryable=true the behaviour is quite funny, not to say weird. Let's see if you can help me out here: when I add this property to Zuul, I get a 404 when sending a request to station. If I remove the property and reboot Zuul, then I am able to hit station successfully. If you run these tests yourself, you may get a 500 which is OK, because I have an instance of Redis exposed in localhost and you probably don't. The route you must call is http://{zuul-dns}:{zuul-port}/weather?city=anythingYouWish

Right now I am at work and I don't have a instance of Redis running locally either, so I also get the 500, but at least it shows that Zuul is proxying it successfully downstream.

I am attaching two screenshots that show the problem. Please let me know if you want me to create another issue or if we can continue to troubleshoot it on this very same thread.

(500 because Redis is not running, but Zuul is hitting station. zuul.retryable=false here.)

(Interesting 404 upon adding zuul.retryable=true)

Thanks for your time,

Thales

[ryanjbaxter]

I am not too familiar with Koltin so I have not tried running the apps. Is there any exception from either Zuul or the station service when the 404 is returned?

[tminussi]

Not really. None of the services print anything when the 404 error is thrown.

I can come up with a Java based application if you want to.

[ryanjbaxter]

I think the problem is your path property, I think you meant the value to be /weather/**.

[tminussi]

Yep, that did the trick. I am still hitting /weather?city=someCity although I now have /weather/** on Zuul. Any idea why that happens though?

Initially I had set weather/* because I really did not want my calls to be /weather/?city=someCity, but adding /weather/* still lets me send the request as the above.

Zuul is now proxying calls successfully to both Services and if an instance of the same service is killed, it will stop sending requests to it.

Can't thank you enough. The issue may be closed. Sorry for any inconvenience.

[ryanjbaxter]

Looks like it was because of a weird pattern matching problem