Kube-router: CNI subnet showing empty after the node is added to cluster

[cvhbsk]

I initialised master with --pod-network-cidr 10.60.3.128/26 and using kube-router with all features enabled using the below configuration.

https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter-all-features.yaml

When i am adding a node to cluster the cni configuration is overwriting with the following content due to which kube-router is failing to run on node.

### /etc/cni/net.d/10-kuberouter.conf

{"bridge":"kube-bridge","ipam":{"subnet":"","type":"host-local"},"isDefaultGateway":true,"name":"kubernetes","type":"bridge"}

Error logs from router

I1212 19:18:16.301932       1 network_policy_controller.go:106] Starting network policy controller
E1212 19:18:16.327767       1 network_routes_controller.go:82] Failed to get pod CIDR from CNI conf file: Failed to get IPAM details from the CNI conf file: invalid CIDR address:
.
.
E1212 19:18:16.360121       1 network_routes_controller.go:102] Failed initial ipset setup: Failed to sync Pod Subnets ipset: ipset v6.32: Syntax error: cannot parse : resolving to IPv4 address failed
.
.
.
E1212 19:18:17.233656       1 network_routes_controller.go:202] Error synchronizing ipsets: Failed to sync Pod Subnets ipset: ipset v6.32: Syntax error: cannot parse : resolving to IPv4 address failed
.
.
.
panic: runtime error: index out of range
goroutine 55 [running]:
github.com/cloudnativelabs/kube-router/app/controllers.(*NetworkRoutingController).advertiseRoute(0xc420648120, 0x25, 0x0)
        /home/travis/gopath/src/github.com/cloudnativelabs/kube-router/app/controllers/network_routes_controller.go:359 +0x60b
github.com/cloudnativelabs/kube-router/app/controllers.(*NetworkRoutingController).Run(0xc420648120, 0xc4206006c0, 0xc4202ece70)
        /home/travis/gopath/src/github.com/cloudnativelabs/kube-router/app/controllers/network_routes_controller.go:243 +0xbea
created by github.com/cloudnativelabs/kube-router/app.(*KubeRouter).Run
        /home/travis/gopath/src/github.com/cloudnativelabs/kube-router/app/server.go:145 +0x5e0

[bzub]

Hi @cvhbsk. Can you check that you are passing --allocate-node-cidrs=true and for example --cluster-cidr=10.60.3.128/26 in your kube-controller-manager command arguments? It's mentioned in https://github.com/cloudnativelabs/kube-router/tree/master/Documentation#requirements

[cvhbsk]

@bzub Yes i verified the equirements section and confirm that kube-controller-manager is having the arguments --allocate-node-cidrs=true and --cluster-cidr=10.60.3.128/26

Also, kube-apiserver and kubelet are running with --allow-privileged=true

[bzub]

Is this affecting a brand new cluster, or newly added nodes to an existing cluster? How many nodes?

I ask because --cluster-cidr=10.60.3.128/26 is fairly small (64 IPs) and would get divided up between nodes. Even with one node, kube-controller-manager may still give it a smaller IP range to work with because it expects more nodes to be added later.

Can you try with a larger cluster-cidr? You may be able to see what's going on in the kube-controller-manager logs. Also check kubelet logs on the node that isn't getting a pod-cidr.

[cvhbsk]

It is new cluster with only 4nodes and no additional deployments or services running.

Currently I have only 1 master node and 1 worker node.

Before adding the worker node in the cluster, I verified the CNI config which looks like below.

/etc/cni/net.d/10-kuberouter.conf

    {
      "name":"kubernetes",
      "type":"bridge",
      "bridge":"kube-bridge",
      "isDefaultGateway":true,
      "ipam": {
        "type":"host-local"
      }
    }

Once I add the worker node to the cluster, it is replaced with the below content by kube-router which is deployed on worker node.

    {
      "name":"kubernetes",
      "type":"bridge",
      "bridge":"kube-bridge",
      "isDefaultGateway":true,
      "ipam": {
        "subnet":"",
        "type":"host-local"
      }
    }

The reson for using --cluster-cidr=10.60.3.128/26 is to avoid ip conflicts with physical network (10.60.3.0/24).

[murali-reddy]

Can you share output of kubectl get nodes -o json | jq '.items[] | .spec' which shoud give pod CIDR allocated for each node.

[cvhbsk]

@murali-reddy

Please find the below output of kubectl get nodes -o json | jq '.items[] | .spec' from master node

{
  "externalID": "os-10-60-3-12.example.com"
}
{
  "externalID": "os-10-60-3-9.example.com",
  "podCIDR": "10.60.3.128/26",
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "node-role.kubernetes.io/master",
      "timeAdded": null
    }
  ]
}

[cvhbsk]

Below is the ConfigMap of kube-router kubectl get cm -n kube-system kube-router-cfg -o yaml from master

apiVersion: v1
data:
  cni-conf.json: |
    {
      "name":"kubernetes",
      "type":"bridge",
      "bridge":"kube-bridge",
      "isDefaultGateway":true,
      "ipam": {
        "type":"host-local"
      }
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"cni-conf.json":"{\n  \"name\":\"kubernetes\",\n  \"type\":\"bridge\",\n  \"bridge\":\"kube-bridge\",\n  \"isDefaultGateway\":true,\n  \"ipam\": {\n    \"type\":\"host-local\"\n  }\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"k8s-app":"kube-router","tier":"node"},"name":"kube-router-cfg","namespace":"kube-system"}}
  creationTimestamp: 2017-12-12T16:21:02Z
  labels:
    k8s-app: kube-router
    tier: node
  name: kube-router-cfg
  namespace: kube-system
  resourceVersion: "436"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-router-cfg
  uid: 726c1a47-df58-11e7-b24f-fa163ea81b68

[murali-reddy]

{
  "externalID": "os-10-60-3-12.sjc2.turn.com"
}
{
  "externalID": "os-10-60-3-9.sjc2.turn.com",
  "podCIDR": "10.60.3.128/26",
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "node-role.kubernetes.io/master",
      "timeAdded": null
    }
  ]
}

Like @bzub mentioned master has taken out entire subnet. You should try with larger cluster-cidr.

[bzub]

You could potentially use --node-cidr-mask-size argument to kube-controller-manager, it defaults to /24 which is why you have issues.

--node-cidr-mask-size=27 2 nodes, 30 IPs each
--node-cidr-mask-size=28 4 nodes, 14 IPs each

Last note on this topic @cvhbsk you mentioned the network was small to avoid IP conflicts. Generally, you would use a network that isn't in use/routable outside of the cluster.

[cvhbsk]

Thanks @murali-reddy @bzub