Spring-cloud-netflix: Eureka: eurekaClient shutdown on context /pause but won't restart on /resume

[brenuart]

EurekaServiceRegistry.unregister() shuts down the EurekaClient when asked to unregister the current registration.
The EurekaAutoServiceRegistration is a SmartLifecycle whose stop() method is invoked when the current context is closed. When called, it tries to unregister the application.

--> there is no need to shutdown the Eureka client at this point - unregistering the application is enough.

Problem is nothing will restart the Eureka client when the context is (re-)started, during a /resume for instance.

The CloudEurekaClient is registered as a @Bean(destroyMethod = "shutdown") in the application context. So its shutdown method will be automatically invoked when the application context is destroyed.

--> we could remove the call to eurekaClient.shutdown() from EurekaServiceRegistry and let Spring invoke the destroy method when it feels appropriate. However, this approach is causing the following exception:

Invocation of destroy method failed on bean with name 'scopedTarget.eurekaClient': org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'eurekaInstanceConfigBean': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

[brenuart]

Note: the proposed change may break this code: https://github.com/spring-cloud/spring-cloud-netflix/blob/master/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/serviceregistry/EurekaRegistration.java#L103-L108

[ryanjbaxter]

What version of Spring Cloud are you using? It sounds similar to #1857.

[brenuart]

It can be reproduced with any version up to Dalston.SR1 included.
I created a simple application using the Spring Initializr by including the features Web, Eureka and Actuator. Here is a short excerpt of the logs:

Once the application is started, try to _pause_ it via a call to POST /pause:

(1) 2017-07-10 22:53:26.286  INFO 3844 --- [nio-8080-exec-1] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
(2) 2017-07-10 22:53:26.287  INFO 3844 --- [nio-8080-exec-1] o.s.c.n.e.s.EurekaServiceRegistry        : Unregistering application unknown with eureka with status DOWN
(3) 2017-07-10 22:53:26.287  WARN 3844 --- [nio-8080-exec-1] com.netflix.discovery.DiscoveryClient    : Saw local status change event StatusChangeEvent [timestamp=1499720006287, current=DOWN, previous=UP]
(4) 2017-07-10 22:53:26.287  INFO 3844 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_UNKNOWN/macfury: registering service...
(5) 2017-07-10 22:53:26.287  INFO 3844 --- [nio-8080-exec-1] com.netflix.discovery.DiscoveryClient    : Shutting down DiscoveryClient ...
(6) 2017-07-10 22:53:26.288  INFO 3844 --- [nio-8080-exec-1] com.netflix.discovery.DiscoveryClient    : Unregistering ...
(7) 2017-07-10 22:53:26.304  INFO 3844 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_UNKNOWN/macfury - registration status: 204
(8) 2017-07-10 22:53:26.317  INFO 3844 --- [nio-8080-exec-1] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_UNKNOWN/macfury - deregister  status: 200
(9) 2017-07-10 22:53:26.321  INFO 3844 --- [nio-8080-exec-1] com.netflix.discovery.DiscoveryClient    : Completed shut down of DiscoveryClient

The application's status is changed to DOWN and the new status is propagated to the Eureka server (see thread InstanceInfoReplicator-0 on lines 4 and 7).
The Eureka Client is shutdown at line 5 and is completely dead at line 9.

Notice that the replication of the instance status and the shutdown of the client happen on TWO different threads. In this example we are lucky the unregister action (line 8) happened after the status update (line 7)...

Now attempt to resume the application with a call to POST /resume:

2017-07-10 22:53:43.935  INFO 3844 --- [nio-8080-exec-2] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
2017-07-10 22:53:43.935  INFO 3844 --- [nio-8080-exec-2] o.s.c.n.e.s.EurekaServiceRegistry        : Registering application unknown with eureka with status UP

The EurekaServiceRegistry simply update the InstanceStatus - but nothing is listening to these changes anymore since he EurekaClient is dead - and nothing will restart it.
Nothing will happen anymore beyond this point: the application won't be registered with the Eureka server anymore nor any heartbeat will be sent.

[ryanjbaxter]

What looks like happens is the DiscoveryClient bean (in this case CloudDiscoveryClient) is left in a "shutdown" (isShutdown is true) state when the pause even occurs. When the resume even occurs we go through the right steps to reregister the client but since the DiscoveryClient thinks it is shutdown nothing happens. I don't see a public way of changing the shutdown state at the moment but I will look into it.

[ryanjbaxter]

Actually I wonder why deregister actually causes the client to shutdown, @spencergibb do you have any idea? https://github.com/spring-cloud/spring-cloud-netflix/blob/1.3.x/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/serviceregistry/EurekaServiceRegistry.java#L71

[brenuart]

According to me, it deregister should not shutdown the client at this point (see my original comments). The client should be closed only when the application is shutdown by the same component that used to start it. The lifecycle of the client would be cleaner this way.

[vpetit35]

I have the same issue. Any workaround in sight ?

[ryanjbaxter]

the fix will be in the Edgware release

[dulimitta]

I am getting the same issue in Edgeware release as well.
"Error creating bean with name 'eurekaAutoServiceRegistration'"

[ryanjbaxter]

@dulimitta if you believe there is still an issue, please open a new issue with a project and steps to reproduce the problem

[dchack]

I use Edgware.SR3, but still have this exception when shutdown project.
use @FeignClient it must have exception.
@ryanjbaxter Is there a way to fix it?

[hyang80-ford]

@dchack
I've added below code to fix this issue:

@Component
public class FeignBeanFactoryPostProcessor implements BeanFactoryPostProcessor {

@Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
    if (containsBeanDefinition(beanFactory, "feignContext", "eurekaAutoServiceRegistration")) {
        BeanDefinition bd = beanFactory.getBeanDefinition("feignContext");
        bd.setDependsOn("eurekaAutoServiceRegistration");
    }
}

private boolean containsBeanDefinition(ConfigurableListableBeanFactory beanFactory, String... beans) {
    return Arrays.stream(beans).allMatch(b -> beanFactory.containsBeanDefinition(b));
}

}

The code depend on this url: https://github.com/dengly/spring-cloud-study/issues/5

[ghost]

I've created new issue here: https://stackoverflow.com/questions/53955613/working-with-eureka-clients-programmatically-issue-completed-shut-down-of-disc

Can somebody please guide ?

[spencergibb]

Please stop spamming multiple issues