Kube-router: /healthz returns OK when kube-router is not okay

[XANi]

kube-router claims it is OK

$ curl localhost:20244/healthz
OK

even when it can't even get into the required API endpoints:

journalctl -a -e -n3 -u kube-router |cat
-- Logs begin at Fri 2019-05-10 15:29:48 CEST, end at Fri 2019-05-10 15:39:09 CEST. --
May 10 15:39:09 kube3 kube-router[6001]: E0510 15:39:09.265533    6001 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: nodes is forbidden: User "kube-router" cannot list resource "nodes" in API group "" at the cluster scope
May 10 15:39:09 kube3 kube-router[6001]: E0510 15:39:09.268237    6001 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: namespaces is forbidden: User "kube-router" cannot list resource "namespaces" in API group "" at the cluster scope
May 10 15:39:09 kube3 kube-router[6001]: E0510 15:39:09.269948    6001 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "kube-router" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope

[MarkDeckert]

I'm not advocating any particular resolution but let's just imagine a scenario where the healthcheck is configured to fail due to a missing ClusterRoleBinding or ClusterRole.

1. The liveness check will see this and restart the pod
2. The healthcheck will fail again
3. Repeat

I'm not sure how this crash loop will resolve the underlying issue, which is the fact that the RoleBindings and/or ClusterRoleBindings are misconfigured or missing. Of course in this situation if the official yaml deployment is used and successful, you will not have missing permissions so this scenario would not happen.

I'm actually battling the opposite issue with another k8s-related project where there is a startup check for Cluster-wide access to something, where I specifically as the user/admin do not want it to have such access, and instead would prefer to give it only specific namespace access. There is no resolution to it since the application just crashes.

[XANi]

Crash loop at least gives immediate feedback that something is wrong (and in our case we just wanted to hook up our monitoring system to something that would show error if something is wrong with a given component)

[andrewsykim]

This isn't ideal, queueing this up for v0.5.0

[aauren]

The current functionality allows kube-router to continue working, similar to many other kubernetes components, when the kube-apiserver is unavailable. With the current functionality, it will continue to run based upon last known state until such a time as the kube-apiserver is reachable again.

In the event, that kube-router cannot reach the kube-apiserver, then restarting/crash looping won't really do much help and only has the potential to propogate more issues into the cluster since it is providing a foundational network layer for the kubernetes cluster which kube-apiserver may be trying to run within.

[XANi]

I don't think you understand problem here @aauren . I run kube-router. I add a monitoring check to check whether there is everything fine with it like any good ops person. The check says OK. The kube router is not okay. Stuff doesn't work. Nobody knows it doesn't work until failure causes problem elsewhere.

If /healtz is required for kubernetes to return okay that's fine but I'd like to have way of checking actual status of the service, whether that be url parameter or another path

[mrueg]

I assume what you are looking for is a readiness endpoint, which in comparison to a liveness endpoint (=internal components and configuration are healthy) takes the external services into account (= service is able to receive and serve traffic)?

[aauren]

I would add to that, that it would be good to use metrics for observing the state of kube-router as well. Admittedly, there is a bit left to be desired by the current metrics emitted from kube-router and there are none currently that indicate the status of it's connection to the kube-apiserver. If you wanted to add metrics that indicate this state we'd be open to such a PR.

However, kube-router is not unhealthy when it loses its connection to the kube-apiserver, it is just stale until such a time as it is able to resume it's connection and catch up. The Kubernetes api libraries take care of all of that functionality in the exact same was as all of the delivered kubernetes control and worker components.

[XANi]

However, kube-router is not unhealthy when it loses its connection to the kube-apiserver, it is just stale until such a time as it is able to resume it's connection and catch up

Stale is as good as down if you're deploying often. And in my particular case the connection problem was config issue so it was bad from the start, there was never "good" state in runtime of the app so it should never even report it is ready.

Just endpoint metrics in form of "here is list of the components , their state and whether they are okay" would IMO be pretty useful in debbuging any issues and could double down as endpoint for montioring