Spring-boot: ApplicationFailedEvent should cause management context to close

[Koizumi85]

Hi,

assuming the following config:

server:
  port: 8080
  address: 10.22.33.44

management:
  address: localhost
  port: 8089

with this config, the tomcat can successfully bind the management-port, but fails to start (in this case on purpose, but could happen through misconfiguration) the regular Tomcat connector, because it can't bind to the given address.
But if you call the /health endpoint, it prints the following output:

{
   status: "UP",
      diskSpace: {
         status: "UP",
         total: 218638835712,
         free: 161173192704,
         threshold: 10485760
      }
}

in my opinion it should say "status: DOWN", because the actual application is not reachable.
Possibly it would be a good idea to implement something like a "WebServerHealthIndicator", which checks the state of all configured tomcat connectors.

[snicoll]

If you ask me it shouldn't show anything. If the main container has failed to start the management endpoint shouldn't even be available. Thanks for sharing the use case!

[dsyer]

If the main container has failed to start the management endpoint shouldn't even be available.

That's debatable I guess. In fact I would claim the opposite - it's quite nice to be able to see management endpoints even if the main app failed to start, and that's a feature I'd quite like to support even if the management port is not declared as different.

[joshiste]

In my opinion the whole process/jvm should stop if the main container can't bind its port (as it is the same when the management port is not different)

[ksperling]

I agree with @joshiste. Having the application exit when it can't fully start/configure itself should be the default behaviour, and matches what a vanilla ApplicationContext does. This works well when running the application from a service manager (systemd, upstart, runit, ...) without extra configuration.

Having to implement additional interactions with the health endpoints just to detect some types of startup errors seems very overcomplicated for the default cause (which is not to say that if somebody has that kind of setup in place Spring shouldn't be configurable to work in that way as well.)

[PedroAlvarado]

Allow me to stand next to @dsyer on this debate and share some thoughts.

If the main container has failed to start the management endpoint shouldn't even be available.

Spring Boot is designed to run applications of many kinds such as batch, console and reactive applications. Servlet-based applications are one more kind of application. The monitoring and management of applications is a cross-cutting concern. Therefore, loose coupling between the actuator and the running application is arguably a healthy design decision.

Having to implement additional interactions with the health endpoints just to detect some types of startup errors seems very overcomplicated

The usage of the health endpoint to monitor applications is a recommended practice for production systems. With this reasonable practice in place and in the presence of a startup and/or other application issues, the active monitoring systems will signal the issues and call for action. Moreover, the process by which operators become aware of application issues can be consistent regardless of whether an issue is startup related or, say, configuration related(e.g. bad redis, rabbitmq, rdbms, consul, eureka configuration).

Possibly it would be a good idea to implement something like a "WebServerHealthIndicator", which checks the state of all configured tomcat connectors.

The case where the main servlet container failed to start is analogous to a data source not being available. That is, there is a critical internal application component that is not available for use. Therefore, the health endpoint needs to report a "DOWN" state. Adding a ServletContainerHealthIndicator similar in nature to the RefreshScopeHealthIndicator can be a way of addressing this issue.

---

All that said, I've always found the availability options to be one Spring Boot's greatest strengths. A fail-fast option can satisfy @joshiste and @ksperling request.

[joshiste]

Possibly it would be a good idea to implement something like a "WebServerHealthIndicator", which checks the state of all configured tomcat connectors.

Afaik, this is not easy possible due to the fact that the "main" tomcat failed and hence the connector is terminated and not accessible. And trying a tcp connect to the port wouldn't suffer because it might be accepted by the other application blocking the port.

The case where the main servlet container failed to start is analogous to a data source not being available.

I totally disagree on this point. A datasource is an external resource your application depends on. If the datasource becomes available again the application (hopefulley) reconnects without any further ado.
If the tomcat cannot bind the port operators restart the application after the port has become available or fixing the configuration - you can't recover from that failure without restarting.

[wilkinsona]

In some ways this is a duplicate of https://github.com/spring-projects/spring-boot/issues/5480. The two issues have competing goals and it's possible that fixing 5480 will mean that the whole app is shut down when the main container fails to start.

[ksperling]

It only makes sense for the container to stay running if there is a chance that the problem will actually fix itself. This is the case for the DataSource example, but I don't see that being the case for the embedded servlet container, at least not how it's implemented now.

For one thing, this would actually require the servlet container to attempt to retry starting periodically, but you would also need to distinguish between permanent errors like misconfiguration and transient ones like a port that is in use -- and even that condition is really only transient when the port is in use by another instance of the same service that is expected to stop soon, as opposed to some arbitrary other service that is using the same port due to misconfiguration of either service. Once you convince yourself that Spring should keep retrying to bind the port, you might as well argue that it should keep retrying on things like missing or invalid configuration files in the hopes of those files appearing.

The fact is that there will always be conditions where Spring won't be able to start up, and any monitoring system already has to handle that case anyway, no matter if the health endpoint says 'DOWN' or is unavailable outright.

If you look at what other network services do (think sshd, Apache httpd, etc) the vast majority will treat failure to bind a port as fatal and exit. I was going to say ntpd is the only example I can think of that handles binding ports as network interfaces come and go, but I just tried it and even ntpd exits immediately if the port is already in use by another process.