Spring-boot-admin: EventStream fails with Spring Boot 2.3.5

[kdvolder]

Very similar to this issue: https://github.com/codecentric/spring-boot-admin/issues/672

My spring-boot-admin server application was created using 'start.spring.io' and the instructions from the docs:
https://codecentric.github.io/spring-boot-admin/current/

My setup has 'no security'. Here's a link to my test application:

https://github.com/kdvolder/spring-boot-docker-fortune-teller/tree/cd7741d7c61de9142c58321da5bc4af61e0327e0

Description of the problem:

• server seems to run mostly fine. But on occasion we get the 'Server Conection Failed' warning banner. If we keep reloading the page, it seems to eventually clear itself.

Things also seem to be working mostly fine as long as the services shown in the main page remain stable. But if we start or stop services then the error often reappears.

So a method that I have to more or less reliably reproduce this is as follows:

• start all the apps in my test application. (I run them from STS boot dash, but you can also run them on the CLI, I suggeest running each app in its own terminal so you can control easily when to start/stop them individually. Command to start them is `./mvnw -Dmaven.test.skip=true -Dspring.profiles.active=local spring-boot:run
• There are 4 apps in total fortune-eureka, fortune-admin-server, fortune-ui, fortune-service.

When all apps are running access admin-server on port 8000 (http://localhost:8000).

Initially the error may appear, refreshing the page a few times will 'fix it'.

Now stop the 'fortune-service'.

Very often I immediately see the error reapear in boot-admin ui (which is still open in browser).

If it does not appear by itself, then usually appears when I refresh the page.

Again refreshing the page a few times eventually fixes it.

Start 'fortune-service' again and the error comes back.

This process can be repeated and will reproduce the error almost every time.

I've also seen this error stacktrace in the of spring-boot-admin server. This error doesn't allways get logged but it seems connected.

Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
    at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) ~[netty-transport-native-unix-common-4.1.53.Final.jar:4.1.53.Final]
    at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) ~[netty-transport-native-unix-common-4.1.53.Final.jar:4.1.53.Final]
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.53.Final-linux-x86_64.jar:4.1.53.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.53.Final.jar:4.1.53.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.53.Final.jar:4.1.53.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.53.Final.jar:4.1.53.Final]
    at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]

Note: Perhaps this is a 'Application Error/problem', as sugested in the other bug as well. This part of the trace:

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) 

Makes me think maybe it has something to do with more than one thing trying to use the same 'singleTread' at the same time and so one of them is 'rejected'.

If it is indeed some kind of configuration problem... perhaps it would be good to document the need for configuring this (whatever 'this' is) in docs somewhere.

[kdvolder]

Looking at the error more closely, I guess it more likely is just a natural consequence of stopping one of the apps than a real problem.

I've tried a bit yesterday to debug but finding it very hard to make any headway on this. Struggling with:

• The reactive code in the backend, which really doesn't lend itself to debugging with breakpoints and stepping through code to try understand what is happening.
  
  
  
  • The only error I see that seems directly connected is in the client-side browser app. But the error has no details all it basically says is 'error' no explanation. I looked in the 'network' tab of Chrome devtools but see no errors there at all.
  
  
  • There are no errors logged on the server-side. I set the logging level for spring boot admin to 'debug' but output doesn't seem to indicate anything going wrong.
  
  

Any help/pointers on how to debug it would be appreciated.

[kdvolder]

I tried to use the sample from here: https://github.com/codecentric/spring-boot-admin/tree/master/spring-boot-admin-samples/spring-boot-admin-sample-eureka

That doesn't have the problem. So wanted to try and narrow down what is the difference.

To make both of them as much similar as possible I made few changes on both side:

• spring-boot-admin-sample-eureka

  • profile: insecure
  • commented out spring-security dependency (as I don't have that).
  • commented out security related beans that don't compile anymore

• my own sample:

  • copied the application.yml exactly from admin-eureka-sample
  • made the annotation on main class exactly same as on admin-eureka-sample

This made no difference.

Finally changed version of boot from 2.3.5 (what I had) to 2.3.4 (what boot-admin-sample-eureka has)... and seems like the problem went away.

[kdvolder]

Okay so I change spring-boot version from 2.3.4 to 2.3.5 in the parent pom of spring-boot-admin and now the eureka sample has same problem as my own. So it looks like there's some change in spring boot or its dependencies that's causing this.

[joshiste]

Looking at the stacktrace, my guess would be that netty is causing this issue. Did you tried to just downgrade the netty version?

[kdvolder]

Did you tried to just downgrade the netty version?

I just tried it and it didn't help.

I.e. I restored boot version of my test app back to 2.3.5 and added:

    <properties>
        <netty.version>4.1.52.Final</netty.version>
            ...
    </properties>

With that the error still/again happens.

[kdvolder]

Looking at the stacktrace

At first I thought the stacktrace might have been related but I'm not sure now. It looks like just an error connecting to one of the apps boot-admin is keeping tabs on. I just stopped that app, so it is probably somewhat to be expected to get an error like that, rather than indicating a real problem.

[joshiste]

For me it doesn't work with Spring Boot 1.3.5 either. But with the current snapshot of 2.3.6 it works again.
So it looks to me like dependency issue.