Rke: Support kube-router

Created on 5 Feb 2018 · 17Comments · Source: rancher/rke

Is it possible to install kube-router instead of kube-proxy and Calico?

Release Candidates kinenhancement

Source

myselfghost

👍38

Most helpful comment

https://gist.github.com/bensallen/10ab4740d9e5042d936ebb4971f2de4f

Slight modification of @PhilippeChepy's example, updated and merged with differences from https://github.com/cloudnativelabs/kube-router/blob/master/daemonset/generic-kuberouter-all-features.yaml.

bensallen on 11 Jun 2019

👍2

All 17 comments

Hi, I would love to be able to use kube-router instead of a network overlay as well :-)

rdegez on 29 Jun 2018

👍1

Same thing here :) I am most interested into non-encapsulated network for performance reasons

abuisine on 27 Oct 2018

Very interested on this too!

LunaticZorr on 21 Nov 2018

Same reason with @abuisine

myclau on 17 Jan 2019

Also interested, posting for updates.

yamashi on 3 Apr 2019

@yamashi you don't need to comment to receive updates - this just spams everyone else who's subscribed. Instead, just click the Subscribe button under Notifications on the right of an issue.

Sorry for the additional noise.

pdf on 3 Apr 2019

👍1

Its possible to use kube-router with something like that :

```apiVersion: v1
kind: ConfigMap
metadata:
name: kube-router-cfg
namespace: kube-system
labels:
tier: node
k8s-app: kube-router
data:
cni-conf.json: |
{
"name":"kubernetes",
"type":"bridge",
"bridge":"kube-bridge",
"isDefaultGateway":true,
"ipam": {
"type":"host-local"
}
}
kubeconfig: |
apiVersion: v1
kind: Config
clusterCIDR: "<< cluster network >>"
clusters:
- name: cluster
cluster:
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
server: https://<< cluster IP>>:6443/
users:
- name: kube-router
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
contexts:
- context:
cluster: cluster
user: kube-router
name: kube-router-context

current-context: kube-router-context

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
k8s-app: kube-router
tier: node
name: kube-router
namespace: kube-system
spec:
template:
metadata:
labels:
k8s-app: kube-router
tier: node
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: kube-router
containers:
- name: kube-router
image: cloudnativelabs/kube-router:v0.2.5
imagePullPolicy: Always
args:
- --run-router=true
- --run-firewall=true
- --run-service-proxy=true
- --kubeconfig=/var/lib/kube-router/kubeconfig
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
livenessProbe:
httpGet:
path: /healthz
port: 20244
initialDelaySeconds: 10
periodSeconds: 3
resources:
requests:
cpu: 250m
memory: 250Mi
securityContext:
privileged: true
volumeMounts:
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: cni-conf-dir
mountPath: /etc/cni/net.d
- name: kubeconfig
mountPath: /var/lib/kube-router/
readOnly: true
initContainers:
- name: install-cni
image: busybox
imagePullPolicy: Always
command:
- /bin/sh
- -c
- set -e -x;
if [ ! -f /etc/cni/net.d/10-kuberouter.conf ]; then
TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
cp /etc/kube-router/cni-conf.json $${TMP};
mv $${TMP} /etc/cni/net.d/10-kuberouter.conf;
fi;
wget https://github.com/containernetworking/plugins/releases/download/v0.7.5/cni-plugins-amd64-v0.7.5.tgz -O /tmp/cni-plugins-amd64.tgz;
tar xf /tmp/cni-plugins-amd64.tgz -C /opt/cni/bin/ .;
true
volumeMounts:
- mountPath: /etc/cni/net.d
name: cni-conf-dir
- mountPath: /opt/cni/bin
name: cni-bin-dir
- mountPath: /etc/kube-router
name: kube-router-cfg
hostNetwork: true
tolerations:
# Make sure calico/node gets scheduled on all nodes.
- effect: NoSchedule
operator: Exists
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
- key: "node-role.kubernetes.io/controlplane"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/etcd"
operator: "Exists"
effect: "NoExecute"
volumes:
- name: lib-modules
hostPath:
path: /lib/modules
- name: cni-conf-dir
hostPath:
path: /etc/cni/net.d
- name: cni-bin-dir
hostPath:
path: /opt/cni/bin
- name: kube-router-cfg
configMap:
name: kube-router-cfg
- name: kubeconfig
configMap:
name: kube-router-cfg
items:
- key: kubeconfig

path: kubeconfig

apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-router

namespace: kube-system

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-router
namespace: kube-system
rules:

apiGroups:
- ""
  
  resources:
  - namespaces
  - pods
  - services
  - nodes
  - endpoints
    
    verbs:
  - list
  - get
  - watch
apiGroups:
- "networking.k8s.io"
  
  resources:
  - networkpolicies
    
    verbs:
  - list
  - get
  - watch
apiGroups:
- extensions
  
  resources:
  - networkpolicies
    
    verbs:
  - get
  - list

- watch

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-router
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-router
subjects:

kind: ServiceAccount
name: kube-router
namespace: kube-system

You must of course disable network plugins in your RKE configuration :

network:
plugin: none
```

PhilippeChepy on 2 May 2019

👍1

https://gist.github.com/bensallen/10ab4740d9e5042d936ebb4971f2de4f

bensallen on 11 Jun 2019

👍2

also, the kubeconfig need to copied to
/var/lib/kube-router/kubeconfig

on each node

aderumier on 11 Jun 2019

@PhilippeChepy @bensallen @aderumier

To get this to work, I'm creating a new cluster in Rancher, modifying the YAML file and setting

network:
  plugin: none

I then use the Docker run command given by rancher on the Masters and Workers, but the masters come up until the point where ETCD tries to register, and I only get the following:

2019-06-10 15:32:52.872967 I | etcdmain: rejected connection from "10.201.91.193:41101" (error "EOF", ServerName "")
2019-06-10 15:32:52.928073 I | etcdmain: rejected connection from "10.201.91.194:34899" (error "EOF", ServerName "")
2019-06-10 15:32:52.953377 I | etcdmain: rejected connection from "10.201.91.195:35589" (error "EOF", ServerName "")

What procedure do you guys follow to get kube-router in when there's no kubeconfig file yet to get in Rancher?

LunaticZorr on 11 Jun 2019

@LunaticZorr I've only tried this with a RKE provisioned cluster so far.

@aderumier The gist above has an initContainer that copies the kubeconfig from the configMap above to /var/lib/kube-router/kubeconfig on each node if it doesn't already exist.

bensallen on 11 Jun 2019

Does anyone know how to configure RKE to not deploy kube-proxy? With kube-router --run-service-proxy=true its redundant.

bensallen on 27 Jun 2019

@LunaticZorr I've only tried this with a RKE provisioned cluster so far.

@aderumier The gist above has an initContainer that copies the kubeconfig from the configMap above to /var/lib/kube-router/kubeconfig on each node if it doesn't already exist.

seem that it's deploying it only on tagged [worker] nodes.
I had master with only [controlplane,etcd], and kube-router was not deployed on it.

aderumier on 1 Jul 2019

@aderumier Yes, I noticed that too, it only deployed it on the Worker nodes, causing lots of connectivity issues. It should go on all nodes.

Edit:

Maybe changing the tier fixes this?

  labels:
    k8s-app: kube-router
    tier: node

(it's defined twice in the yaml)

LunaticZorr on 2 Jul 2019

@aderumier @LunaticZorr Try settings the tolerations to:

      tolerations:
      - effect: NoSchedule
        operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoExecute
        operator: Exists

This is copied from what RKE uses for Canal.

From https://rancher.com/docs/rke/latest/en/config-options/nodes/, etcd roles are tainted with NoExecute, which the spec from the Gist doesn't tolerate at all. Plus the NoSchedule toleration has different key than what RKE appears to use.

bensallen on 18 Jul 2019

Is there anything happening on this issue regarding official support from Rancher? I see that its in the backlog but have there been any decision if this CNI will be supported?

ventris on 12 Nov 2019

+1 for kube-router support. It should be fairly simple to implement this. It's much less complex when compared to Calico (also doesnt require an additional etcd cluster) and is incredibly stable. We've been running it in kubespray deployed clusters for almost two years now.