Spring-cloud-netflix: Eureka Server takes around 10 minutes to update the service-urls since refreshed

[fahimfarookme]

I'm deploying eureka and config server on an on-premise cloud. I follow a config first approach where the defaultZone is configured with config-server in config-repo (because DNS is not an option for us). However this mandates a strict deployment order i.e. config server must be up and running before starting eureka. In our case we do not grantee a deployment order due to some infra policies. So the only option available is to invoke the /refresh endpoint once the config-server is started. Also as I tested, the property eureka.client.eureka-service-url-poll-interval-seconds is only applicable for DNS.

• Tried invoking /refresh endpoint of eureka / microservices, but the defaultZone is not updated.
• This is because Spring cloud defines EurekaClient bean in refresh scope, however it injects EurekaClientConfig bean which is not in refresh scope.

@Bean(destroyMethod = "shutdown")
@ConditionalOnMissingBean(value = EurekaClient.class, search = SearchStrategy.CURRENT)
@org.springframework.cloud.context.config.annotation.RefreshScope
@Lazy
public EurekaClient eurekaClient(ApplicationInfoManager manager, EurekaClientConfig config, EurekaInstanceConfig instance) {
    manager.getInfo(); // force initialization
    return new CloudEurekaClient(manager, config, this.optionalArgs,
            this.context);
}

Since the EurekaClient is already on refresh scope, I believe it would be beneficial if the EurekaInstanceConfig bean could also be defined in refresh scope so that /refresh could update the service-url list.

[ryanjbaxter]

I think this makes sense. Want to try submitting a PR for it?

[fahimfarookme]

@ryanjbaxter absolutely!!! Let me file a PR

[fahimfarookme]

@ryanjbaxter Further analysis revealed that even-though the EurekaClient bean is not defined in refresh scope, the service-url is getting updated in the Eureka Server Dashboard when I hit the /refresh endpoint, however it takes around 10 minutes for it to reflect the new service-urls and that's where I was misled. To summarize my finding;

• Hitting the /refresh endpoint updates the @ConfigurationProperties beans even if they are not in @RefreshScope. PoC here to confirm this behavior. During /refresh i.e. EnvironmentChangeEvent, @ConfigurationProperties beans are reinitialized by ConfigurationPropertiesRebinder. This means that @ConfigurationProperties beans are in refresh scope by default.

However there are 2 more issues.

Issue 1 (not relevant for Spring Cloud Netflix)

Consider the following sequence.

• Config-server is down/ not started while the eureka-server (or any other config-client) is starting
• Hit /refresh endpoint of config-client once the config-server is started

This does not reinitialize the environment with properties in config-repo, hence service-url is not picked from config-repo. However if the config-server is already up while the config-client is starting, then hitting the /refresh endpoint reinitialize the environment. PoC here to confirm this behavior. I believe this should be addressed in spring-cloud-config and will raise a ticket for this issue.

Issue 2

It takes around 10 mins for Eureka Server to reflect the new service-urls (i.e. eureka.client.service-url.defaultZone) after hitting the /refresh endpoint. i.e.

• Started the Eureka server while the config server is UP and running.
• Changed the service-urls in config repo and hit the /refresh endpoint of Eureka Server
• New service-url is not reflected in the eureka dashboard or PeerEurekaNodes class as soon as environment is refreshed.

Solutions

This is because PeerEurekaNodes.updatePeerEurekaNodes() is invoked by a scheduler which runs in every 10 minutes by default. i.e. configured by EurekaServerConfigBean.peerEurekaNodesUpdateIntervalMs = 10 * MINUTES. Provided that the /refresh endpoint is invoked, waiting for 10 minutes to reflect the service-urls in Eureka Server can lead ASM teams to come to wrong conclusions. Setting the property eureka.server.peer-eureka-nodes-update-interval-ms to a lower value could solve the problem, however that will introduce unnecessary network traffic in the system since the service-url list is rarely updated. Also as I analyzed, we cannot define EurekaServerContext bean in @RefreshScope, because that will reinitialize the scheduler objects and results in an IllegalState.

Therefore I would suggest to call PeerEurekaNodes.updatePeerEurekaNodes() upon EnvironmentChangeEvent. I've tested this solution successfully and could see the DS replicas updated in Eureka Dashboard as soon as the environment is refreshed. Will try submitting a PR shortly.

i.e.

@Bean
@ConditionalOnMissingBean
public PeerEurekaNodes peerEurekaNodes(...) {
    return new RefreshablePeerEurekaNodes(...);
}

static class RefreshablePeerEurekaNodes extends PeerEurekaNodes
        implements ApplicationListener<EnvironmentChangeEvent> {
        // constrcutor

    @Override
    public void onApplicationEvent(final EnvironmentChangeEvent event) {
        updatePeerEurekaNodes(resolvePeerUrls());
    }
}