Kube-router: Hostport pods take over traffic from services

Created on 16 Dec 2018  路  6Comments  路  Source: cloudnativelabs/kube-router

Kube-Router Version: v0.2.3

Steps:

  1. Create a daemon set with a hostport. This port must match an existing service that is already created.
  2. Wait for pods to start
  3. Try and communicate with the service

Example:

The kubernetes service in the default namespace with a port of 443. Spin up a daemonset with pods that have a host port of 443. All traffic being sent to the kubernetes service is now being sent to the daemonset pods instead of the service pods.

As soon as the host port pods are deleted things go back to normal.

cni-config.json

    {
       "cniVersion":"0.3.0",
       "name":"mynet",
       "plugins":[
          {
             "name":"kubernetes",
             "type":"bridge",
             "bridge":"kube-bridge",
             "isDefaultGateway":true,
             "ipam":{
                "type":"host-local"
             }
          },
          {
             "type":"portmap",
             "capabilities":{
                "snat":true,
                "portMappings":true
             }
          }
       ]
    }
picture rmb938

Most helpful comment

So if I update my hosts to use the newest CNI plugins then I edit /etc/cni/net.d/10-kuberouter.conflist to have this in the portmap configuration

"conditionsV4": ["!", "-d", "100.67.0.0/16"]

where 100.67.0.0/16 is my svc cidr. It seems to work as expected.

However I needed to edit /etc/cni/net.d/10-kuberouter.conflist after kube-router adds the subnet as it seems that if I do it before (i.e via the config map) that option gets removed. So soemthing needs to be changed within kube-router to allow this option.

Actually ignore that last part, I didn't update the config correctly.

picture rmb938 on 17 Dec 2018
👍2

All 6 comments

If this is a master node then yes, you cannot use port 443 as the api-server in most setups listens on the host interface and you would nuke it by setting up a hostport

roffe picture roffe on 17 Dec 2018

you can create a Daemonset with hostport and use a nodeselector / pod affinity / anti-affinity to just have it run on a set of non-master nodes that won't have 443 occupied

roffe picture roffe on 17 Dec 2018

This is not the master node and I don't really appreciate this issue being closed without any confirmation.

This is a legit bug and I can easily repeat it.

rmb938 picture rmb938 on 17 Dec 2018
👍1

Here is some manifests to test with:

apiVersion: v1
kind: Service
metadata:
  name: test-svc
spec:
  ports:
  - port: 5678
    protocol: TCP
  selector:
    app: test
  type: ClusterIP
--- 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: test
  labels:
    app: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: nginx
        image: hashicorp/http-echo
        args:
          - -text="hello world"
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: test-daemonset
  labels:
    app: daemonset
spec:
  selector:
    matchLabels:
      app: daemonset
  template:
    metadata:
      labels:
        app: daemonset
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
          - containerPort: 80
            hostPort: 5678

Pods Launched

kubectl get pods -o wide                                                                                      05:24:47
NAME                    READY     STATUS    RESTARTS   AGE       IP            NODE
test-76b7b64bc7-drwpg   1/1       Running   0          45s       100.66.1.45   192.168.23.166
test-daemonset-cfs9j    1/1       Running   0          7m        100.66.2.42   192.168.23.183
test-daemonset-wqtc4    1/1       Running   0          7m        100.66.1.43   192.168.23.166

Once the daemonset pods are launch, delete the pods and let them relaunch.

Service

kubectl get endpoints test-svc                                                                                05:24:29
NAME       ENDPOINTS          AGE
test-svc   100.66.1.45:5678   8m

Curling the host using the host port

curl 192.168.23.183:1234                                                                                      05:19:27
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

Curling the service inside a pod. This shouldn't return nginx but it does!

/ # curl test-svc:5678
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>

This is what I expect it to return

 curl test-svc:5678
"hello world"
rmb938 picture rmb938 on 17 Dec 2018

It does seem that the portmap plugin https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap has some new options that may solve this.

Specifically these:

conditionsV4, conditionsV6 - array of strings. A list of arbitrary iptables matches to add to the per-container rule. This may be useful if you wish to exclude specific IPs from port-mapping

So if the service cidr is excluded that should solve the issue.

rmb938 picture rmb938 on 17 Dec 2018

So if I update my hosts to use the newest CNI plugins then I edit /etc/cni/net.d/10-kuberouter.conflist to have this in the portmap configuration

"conditionsV4": ["!", "-d", "100.67.0.0/16"]

where 100.67.0.0/16 is my svc cidr. It seems to work as expected.

However I needed to edit /etc/cni/net.d/10-kuberouter.conflist after kube-router adds the subnet as it seems that if I do it before (i.e via the config map) that option gets removed. So soemthing needs to be changed within kube-router to allow this option.

Actually ignore that last part, I didn't update the config correctly.

rmb938 picture rmb938 on 17 Dec 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

SEJeff picture SEJeff  路  6Comments
cvhbsk picture cvhbsk  路  10Comments
johanot picture johanot  路  7Comments
shibd picture shibd  路  4Comments
murali-reddy picture murali-reddy  路  4Comments