kube-router stop working when Kubernetes masters are down

[tkoeck]

Hi,

we are in the process for testing a Kubernetes upgrade from 0.11 to 0.12. For that we have to upgrade all masters at once.

Now we have the strange behavior that the kube-router are crashing after the three masters are not available anymore. We could replicate the behavior with

• kube-router 0.1
• kube-route 0.3.1

As far as I have udnerstood services/pods that are running on the pod (e.g. webserver) should still be accessable without masters.

Or is it the right behavior that kube-router is crashing on the other nodes if the masters aren't there anymore?

I haven't found further information in the kube-router documentation. And I am puzzled by that behavior. Do you have any ideas?

[wu0407]

The kube-router is not crash, but the liveness probe with /healthz failed, which process test in 0.3.2.
Kubelet restart the pod, but failed to synchronize informers cache, so start kube-router failed.

[wu0407]

Pods still running when Kubernetes masters are down, so the pods should still work, but no route to pods ip. I will fix to this bug

[tkoeck]

Some pods aren't accessible anymore after the masters were down for 1 minute. We had checked the kube-router pods on the nodes. They were always restarting after a short time.

We have noticed that behavior because we had to upgrade all three masters at once (etcd3 upgrade breaking changes) and wanted to have a zero down upgrade.

A bug fix would be preferable. Not for us but for all other guys who still have to upgrade their etcd Kubernetes.

[wu0407]

you can disable liveness probe before upgrade etcd.

[wu0407]

It like the bug is vendor path not update on build 0.3.2 docker image.

The log with 0.3.2 image output when apiserver is down.

E1116 09:03:17.475938       1 network_routes_controller.go:291] Error adding BGP export policies: Failed to get pod CIDR allocated for the node due to: Failed to identify the node by NODE_NAME, hostname or --hostname-override

The code of network_routes_controller.go:291 on tag v0.3.2
https://github.com/cloudnativelabs/kube-router/blob/4afd6d6d2ab9c94abc5985c30c56ca2605a70a3f/pkg/controllers/routing/network_routes_controller.go#L291

This change relate with https://github.com/cloudnativelabs/kube-router/commit/8fe9f70dd59a7fc5c0de7ffcd07b75e7b0b99a8c

[aauren]

@murali-reddy I believe from the error message above that @wu0407 is correct, that this is related to the PodCIDR fix that was merged last week (https://github.com/cloudnativelabs/kube-router/pull/813). I assume that this could be considered fixed now and closed?

[murali-reddy]

@aauren right. I doble checked to see if there are any code paths where pod CIDR is being read from API server. There are just two instaces at the start of the controllers. We can close this issue.

@tkoeck please try v1.0.0-rc1 and reopen if you run into this issue again.