Kube-router: Running without master

[zerkms]

I was under impression that key kubernetes components are designed to be able to run without master.

I implied that it would just keep the same state but at least it would run it for as long as possible.

What I observe is that kube-router dies approximately 3 minutes after the master is not available and removes routes (delivered via BGP, in a full-mesh configuration), not all though (!).

This leads to partial-to-full cluster failure

These are the last entries the kube-router container outputs before quitting and effectively killing the rest of the cluster

E0128 23:03:21.610345       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.NetworkPolicy: Get https://10.250.45.112:6443/apis/networking.k8s.io/v1/networkpolicies?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:03:21.610484       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Pod: Get https://10.250.45.112:6443/api/v1/pods?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:03:21.610540       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Endpoints: Get https://10.250.45.112:6443/api/v1/endpoints?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:03:21.610584       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Service: Get https://10.250.45.112:6443/api/v1/services?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:03:21.610634       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: Get https://10.250.45.112:6443/api/v1/namespaces?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
Failed to run kube-router: Failed to synchronize cache: 1m0s timeout

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                     PORTS               NAMES
44daf4e427d7        cloudnativelabs/kube-router   "/usr/local/bin/kube…"   4 minutes ago       Exited (1) 3 minutes ago                       k8s_kube-router_kube-router-qdbs7_kube-system_52f0f933-1ac2-11e9-8cbc-0afb69308c7a_32

Sometimes - the quit is cleaner, when a particular instance of kube-router is lucky enough to clean up after themselves:

time="2019-01-28T23:31:10Z" level=warning msg="received notification" Code=6 Communicated-Reason= Data="[]" Key=10.250.44.218 Subcode=2 Topic=Peer
time="2019-01-28T23:31:10Z" level=info msg="Peer Down" Key=10.250.44.218 Reason="notification-received code 6(cease) subcode 2(administrative shutdown)" State=BGP_FSM_ESTABLISHED Topic=Peer
E0128 23:31:10.757682       1 health_controller.go:118] Network Routing Controller heartbeat missed
I0128 23:31:11.914479       1 kube-router.go:178] Shutting down the controllers
I0128 23:31:11.914520       1 health_controller.go:161] Shutting down health controller
E0128 23:31:11.914574       1 health_controller.go:150] Health controller error: http: Server closed
I0128 23:31:11.914597       1 health_controller.go:178] Shutting down HealthController RunCheck
I0128 23:31:11.914610       1 network_services_controller.go:353] Shutting down network services controller
I0128 23:31:11.914618       1 network_policy_controller.go:152] Shutting down network policies controller
E0128 23:31:13.252717       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Endpoints: Get https://10.250.45.112:6443/api/v1/endpoints?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:31:13.252769       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Node: Get https://10.250.45.112:6443/api/v1/nodes?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:31:13.252797       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Namespace: Get https://10.250.45.112:6443/api/v1/namespaces?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:31:13.252825       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.NetworkPolicy: Get https://10.250.45.112:6443/apis/networking.k8s.io/v1/networkpolicies?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:31:13.252848       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Pod: Get https://10.250.45.112:6443/api/v1/pods?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:31:13.252865       1 reflector.go:205] github.com/cloudnativelabs/kube-router/vendor/k8s.io/client-go/informers/factory.go:73: Failed to list *v1.Service: Get https://10.250.45.112:6443/api/v1/services?resourceVersion=0: dial tcp 10.250.45.112:6443: connect: no route to host
E0128 23:31:16.324373       1 network_routes_controller.go:289] Error adding BGP export policies: Failed to get pod CIDR allocated for the node due to: Failed to identify the node by NODE_NAME, hostname or --hostname-override
E0128 23:31:19.396277       1 bgp_peers.go:42] Failed to list nodes from API server due to: Get https://10.250.45.112:6443/api/v1/nodes: dial tcp 10.250.45.112:6443: connect: no route to host. Can not perform BGP peer sync
E0128 23:31:19.396300       1 network_routes_controller.go:299] Error during periodic sync in network routing controller. Error: Failed to get pod CIDR allocated for the node due to: Failed to identify the node by NODE_NAME, hostname or --hostname-override
E0128 23:31:19.396306       1 network_routes_controller.go:300] Skipping sending heartbeat from network routing controller as periodic sync failed.
I0128 23:31:19.396314       1 network_routes_controller.go:305] Shutting down network routes controller
time="2019-01-28T23:31:19Z" level=info msg="Administrative shutdown" Key=10.250.45.112 State=BGP_FSM_ACTIVE Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Administrative shutdown" Key=10.250.47.169 State=BGP_FSM_ESTABLISHED Topic=Peer
time="2019-01-28T23:31:19Z" level=warning msg="sent notification" Code=6 Communicated-Reason= Data="[]" Key=10.250.47.169 State=BGP_FSM_ESTABLISHED Subcode=2 Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Administrative shutdown" Key=10.250.44.218 State=BGP_FSM_ACTIVE Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Administrative shutdown" Key=10.250.46.255 State=BGP_FSM_ESTABLISHED Topic=Peer
time="2019-01-28T23:31:19Z" level=warning msg="sent notification" Code=6 Communicated-Reason= Data="[]" Key=10.250.46.255 State=BGP_FSM_ESTABLISHED Subcode=2 Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Administrative shutdown" Key=10.250.32.16 State=BGP_FSM_ESTABLISHED Topic=Peer
time="2019-01-28T23:31:19Z" level=warning msg="sent notification" Code=6 Communicated-Reason= Data="[]" Key=10.250.32.16 State=BGP_FSM_ESTABLISHED Subcode=2 Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Peer Down" Key=10.250.47.169 Reason="notification-sent code 6(cease) subcode 2(administrative shutdown)" State=BGP_FSM_ESTABLISHED Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Peer Down" Key=10.250.46.255 Reason="notification-sent code 6(cease) subcode 2(administrative shutdown)" State=BGP_FSM_ESTABLISHED Topic=Peer
time="2019-01-28T23:31:19Z" level=info msg="Peer Down" Key=10.250.32.16 Reason="notification-sent code 6(cease) subcode 2(administrative shutdown)" State=BGP_FSM_ESTABLISHED Topic=Peer

My question is - is it a deliberate decision and works "as designed", or a bug and kube-router should tolerate temporary api server loss?

[murali-reddy]

TLDR: API server is database for the controllers. If api server is not accessible its fatal condition. But,

I implied that it would just keep the same state but at least it would run it for as long as possible.

is desirable to have so kube-router is more graceful. Marking this issue as enhancement to be taken care.

kube-router is built around the concept of controllers. Controller pattern is at the heart of Kubernetes architecture. Typically controllers perform two activities. Reacting to the updates (add/delete/update events) to API objects to sync actual state to desired state. Other functionality is to periodically reconcile actual state to desired state. I would imagine first activity is not an issue as API servers is down as you wont get updates at all. Second activity which is to do periodic reconciliation will be affected as API server cannot be reached. So this is where things go wrong. kube-router can be changed to be little more graceful and keep running till it its able to get updates to the API object.

What I observe is that kube-router dies approximately 3 minutes after the master is not available and removes routes (delivered via BGP, in a full-mesh configuration), not all though (!).
This leads to partial-to-full cluster failure

This can happen even if you are performing rolling-upgrades. Essentially you need to use BGP graceful restart functionality so that routes are not withdrawn when control-plane goes down.

[zerkms]

Right, it looks like --bgp-graceful-restart is a solution.

Thanks for the advice and what you do for the project overall :-)

[aauren]

I believe that this is finally resolved via https://github.com/cloudnativelabs/kube-router/pull/813 and https://github.com/cloudnativelabs/kube-router/pull/856

Please test off of master and let us know if you find any more issues.