Spring-cloud-config: Questions on config client's retry behavior

[george24601]

Context

Our current step is 3 config server instances registered on eureka. Other services use eureka client and spring.cloud.config.serviceId to locate the config server instance.

During the failover test, seems that client retries against the same config server URI. That is, when one config server instance is down, and I fetch configs from that config server instance, client will try reconnecting to that instance instead of using another config server instance.

Questions

1. Is such behavior intended? The implementation of ConfigServicePropertySourceLocator suggests so.
2. If answer to 1) is yes, is it possible to use client-side load balancing, e.g., ribbon, on the retry logics? If so, is there any implementation I can use for reference?

Of course, I can also put config server instances behind a load balancer.

[ryanjbaxter]

There is retry logic build into the config client
http://cloud.spring.io/spring-cloud-static/Dalston.SR4/single/spring-cloud.html#config-client-retry

But the config client does not have a dependency on Ribbon so there is no load balancing built in. As you mention the best option would be to possibly put the config server instances behind a load balancer.

[george24601]

Thanks for the reply!

I was following the section 10.2 Discovery First Bootstrap on that page. It says by using Eureka to discover config server instances by service id,

The benefit is that the Config Server can change its co-ordinates, as long as the Discovery Service is a fixed point.

My understanding is that if I use @EnableDiscoveryClient, config client will try a different config server instance if it is not able to connect to the current config server instance? If so, when will such switch happen? After the retry in ConfigServicePropertySourceLocator?

The ultimate goal is that config client can connect and fetch configuration, even though some config clients are healthy and some not.

[ryanjbaxter]

I dont think that is the way it works. ConfigServiceInstanceProvider just chooses the first instance in the list
https://github.com/spring-cloud/spring-cloud-config/blob/master/spring-cloud-config-client/src/main/java/org/springframework/cloud/config/client/ConfigServerInstanceProvider.java#L28

When a retry happens it continues to retry the same instance.

[george24601]

After more research, I see what confused me at the beginning. Here are my findings

1. By default, an unavailable instance will be removed from eureka registry after leaseExpirationDurationInSeconds=90 seconds. During that 90 seconds, Eureka server still treats that instance as UP, and the traffic to the now-dead instance will fail. Such behaviour is standard in load balancers and is intended.

2. However, during my failover test, I killed off a few instances and triggered Eureka's self-preservation mode. In this mode, Eureka always returns the now-dead instance and will not retire it. That is why the config client has been retrying against a dead-instance.

For the record, here are my solutions:

1. The original idea of a client side load balancer is not really needed - Eureka itself is enough.
2. Lower renewalPercentThreshold. 85 Percent is too high for non-prod/performance environment
3. Lower leaseExpirationDurationInSeconds so that instances can be evicted sooner.