Spring-cloud-netflix: Eureka client local cache doesn't refresh in a "discovery first bootstrap" scenario

[bberto]

Describe the bug
I've set up 3 spring boot apps:

1. _discovery_ (eureka server) listening on port 9000
2. _config_ server on 9001
3. _admin_ application (spring admin) on port 9002

On _admin_ application I configured bootstrap.yml to retrieve settings from _config_ server using _discovery_, like this:

spring:
  cloud:
    discovery:
      blocking.enabled: true
      reactive.enabled: false        
    config:
      enabled: true
      discovery:
        enabled: true
        service-id: config
      fail-fast: false
      retry:
        initial-interval: 2000
        multiplier: 1.5
        max-attempts: 10
        max-interval: 30000

When starting apps in order with some delay (_discovery_ then _config_, then after a while _admin_) everythings seems OK.

But if I change topology (for example shutting down _config_), _admin_ doesn't refresh its cache. Can be checked using /actuator/health/discoveryComposite endpoint that still lists _config_, while on _discovery_ is disappeared. Moreover, eureka status remains Eureka discovery client has not yet successfully connected to a Eureka server

Another test: if I start _discovery_ first and then _config_ + _admin_ together (or at least before _config_ become STARTED on Eureka server), _admin_ eureka cache remains empty and I get Could not locate configserver via discovery warn on all subsequent retries.

Doing some debugging, seems to be related to the presence of two differents DiscoveryClient (one in bootstrap and the other in the application context?).

Reproduced with Spring Boot 2.2.2 + Spring Cloud Hoxton.RELEASE. Similar issues were present also on 2.1.10 (not analyzed in depth)

Sample
I committed a repo with demo apps to reproduce the issue.
https://github.com/bberto/springcloud-bootstrap-config

[spencergibb]

If there are very few instances eureka will enter self preservation mode and will not evict instances.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[bberto]

I disabled it with eureka.server.enable-self-preservation=false but I can't notice any difference. I think the root cause is the one reported here https://github.com/spring-cloud/spring-cloud-netflix/issues/3675#issuecomment-565388117

[Manuel-Linkener]

I have the same problem. I run a eureka-server and config-server. Now I have a service with a config client in discovery first mode.

spring:
    config:
      discovery:
        enabled: true

discovery first config client bootstrap.yml

On boot, the discovery client fetches the list of services (to find the config server) and then never updates the list of available services again.

When you hit the actuator health endpoint you see:

{
  "status": "UP",
  "components": {
    "discoveryComposite": {
      "status": "UP",
      "components": {
        "discoveryClient": {
          "status": "UP",
          "details": {
            "services": [
              "config-server"
            ]
          }
        },
        "eureka": {
          "description": "Eureka discovery client has not yet successfully connected to a Eureka server",
          "status": "UP",
          "details": {
            "applications": {
              "CONFIG-SERVER": 1
            }
          }
        }
      }
    }
  }
}

It shows only the services, that where registered at the time the service booted (in my case there was just the config-server) and never changes again.

Now I deploy another service, that does not use service discovery for the config client and the ....eureka.description changes after a while to Remote status from Eureka server and the list of services updates. But if you start the service with a config client in discovery first mode, the list of services never updates.

I am not sure if this is a spring cloud netflix issue, looks more like a config client (or service discovery client) issue, but since I found this issue, I post it here.

[OlgaMaciaszek]

Putting it on hold - we'll be discussing it with Netflix to try to come up with a solution.

[aveiros]

Is the best work around for now to exclude EurekaDiscoveryClientConfigServiceAutoConfiguration ?

[OlgaMaciaszek]

Changes need to be made in order to avoid EurekaClient for registration in SC Config.
For a temporary workaround, please see this one proposed by @k631583871.

Closing in favour of SC Config #514.

[Manuel-Linkener]

I just tested it again with Spring Cloud Hoxton.SR5 and Spring Boot 2.3.0 the issue is fixed 🎉🎉🎉 Thank you very much @OlgaMaciaszek and the guys in https://github.com/spring-cloud/spring-cloud-netflix/issues/3795 for fixing this!