Calico: Traffic Shaping in Calico

[rikatz]

This issue is a 'revival' of issue https://github.com/projectcalico/calicoctl/issues/577

Expected Behavior

The expected behavior is to have a possible traffic shaping in Calico, as implemented in Contiv here:

http://contiv.github.io/documents/networking/policies.html

Current Behavior

No Traffic Shaping is supported

Possible Solution

Contiv does implements this using Openvswitch, but I think this is not the right way. Maybe setting this through 'tc command' is a good choice (and also, CoreOS have tc built in, as example!).

Maybe creating some tc rules the same way Calico Creates IPtables rules is enough :)

Context

We're going to create a bunch of network/ippools with Calico. This is useful to us to separate (in the very same container cluster) the prod and non-prod traffic. But also, we would like also to create different services levels per ippool.

Your Environment

• Calico version 2.1
• Orchestrator version Kubernetes
• Operating System and version: CoreOS

[caseydavenport]

I think something like this could be done via tc, but as mentioned in the original issue I'd like to see some more concrete user stories laid out so we can determine what the API should look like.

I think we'll need to roughly follow these steps:

• [ ] Enumerate key user stories / use-cases in this issue.
• [ ] Agree how these should be expressed in the Calico API.
• [ ] Extend libcalico-go with the newly agreed APIs.
• [ ] Likely add support to Felix to implement the new APIs.

[rikatz]

@caseydavenport yeah, I'll try to enumerate them (If you think it's better I can change the original text from the issue)

Use cases

• Admin want's to limit the bandwidth from a defined ippool/network (or maybe this is a new object in Calico).

A network is created for some kind of workload. This workload demands a lot of communication between the VMs/Containers/whatever and Internet. We need to limit this, as our link doesn't get saturated. (TS for Input traffic)

Other: Some networks are created for different workloads. Some containers get inside the 'Gold' network, that have no bandwidth limit, and other in 'silver' workload have a bandwidth limit of 5Mb/s.

Another one: To limit some container causing 'DoS' to other container (like in a PaaS environment) we should be able to limit the bandwidth between containers/VMs from different projects/namespaces (or even between them in the same namespace).

How this is expressed in the Calico API

Actually I don't know. This could be some kind of 'policy' object for Calico, but could impact in existing 'firewall' policies. Maybe this can be only a part of the policy, like 'allow this traffic from here to there, with this bandwidth'.

Thanks :D

[haoyixin]

+1

[xiaods]

+1

[meishd]

+1

[caseydavenport]

FWIW traffic shaping via tc and the CNI shaping plugin will soon be supported in upstream Kubernetes via CNI plugin chaining.

I haven't tried it with Calico yet but I don't see why it wouldn't work. If anyone tried, please report back with your findings!

[valentin2105]

Hi @casey, you got any link about that ? Thanks !

[caseydavenport]

Yup, this is the PR that added it: https://github.com/kubernetes/kubernetes/pull/63194

I don't believe there are any docs on it yet.

[johscheuer]

Hi together, I got some time to test calico and the cni bandwidth plugin. From my tests I can say that everything works (nearly) out of the box:

I needed to adjust the cni config:

  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.0",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
            "type": "host-local",
            "subnet": "usePodCidr"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        { # you need to add this
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }

now you need to install thee bandwidth plugin (this step is required as long as this PR is open: https://github.com/projectcalico/cni-plugin/pull/745):

cd /tmp

curl -sLO https://github.com/containernetworking/plugins/releases/download/v0.8.0/cni-plugins-linux-amd64-v0.8.0.tgz
tar xfvz cni-plugins-linux-amd64-v0.8.0.tgz
sudo mv bandwidth /opt/cni/bin/

Now you can create a very simple test to see if the rules take effect:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: 1M
    kubernetes.io/egress-bandwidth: 1M
  name: iperf-server
  labels:
    app: iperf-server
spec:
  restartPolicy: OnFailure
  containers:
  - name: iperf-server
    image: networkstatic/iperf3
    args: [ "-s" ]
    ports:
    - containerPort: 5201
---
apiVersion: v1
kind: Service
metadata:
  name: iperf-server
  labels:
    app: iperf-server
spec:
  ports:
  - port: 5201
    targetPort: 5201
  selector:
    app: iperf-server
---
apiVersion: v1
kind: Pod
metadata:
  name: iperf-client
  labels:
    app: iperf-client
spec:
  restartPolicy: OnFailure
  containers:
  - name: iperf-server
    image: networkstatic/iperf3
    args: [ "-c",  "iperf-server" ]

with the bandwidth plugin activated:

kubectl logs -f iperf-client
Connecting to host iperf-server, port 5201
[  4] local 192.168.1.87 port 52070 connected to 10.110.212.85 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  40.7 KBytes   333 Kbits/sec    2   1.36 KBytes       
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.36 KBytes       
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.36 KBytes       
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.36 KBytes       
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.36 KBytes       
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.36 KBytes       
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.36 KBytes       
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.36 KBytes       
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.36 KBytes       
[  4]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.36 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  40.7 KBytes  33.3 Kbits/sec    5             sender
[  4]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.

and without:

kubectl logs -f iperf-client
Connecting to host iperf-server, port 5201
[  4] local 192.168.1.89 port 59380 connected to 10.102.83.243 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.12 GBytes  26.8 Gbits/sec  536   1.03 MBytes       
[  4]   1.00-2.00   sec  3.09 GBytes  26.5 Gbits/sec  325    538 KBytes       
[  4]   2.00-3.00   sec  2.81 GBytes  24.1 Gbits/sec  378   1.31 MBytes       
[  4]   3.00-4.00   sec  3.10 GBytes  26.6 Gbits/sec    1   1.31 MBytes       
[  4]   4.00-5.00   sec  2.86 GBytes  24.6 Gbits/sec    0   1.45 MBytes       
[  4]   5.00-6.00   sec  3.11 GBytes  26.7 Gbits/sec  1302    773 KBytes       
[  4]   6.00-7.00   sec  3.13 GBytes  26.9 Gbits/sec    0    861 KBytes       
[  4]   7.00-8.00   sec  2.72 GBytes  23.4 Gbits/sec    0   1.09 MBytes       
[  4]   8.00-9.00   sec  3.20 GBytes  27.5 Gbits/sec    0   1.12 MBytes       
[  4]   9.00-10.00  sec  2.43 GBytes  20.9 Gbits/sec    0   1.43 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  29.6 GBytes  25.4 Gbits/sec  2542             sender
[  4]   0.00-10.00  sec  29.6 GBytes  25.4 Gbits/sec                  receiver

iperf Done.

you can see that the rules take effect. Also you can use tc qdisc show to inspect the rules.

qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev ens4 root 
qdisc fq_codel 0: dev ens4 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn 
qdisc fq_codel 0: dev ens4 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn 
qdisc noqueue 0: dev docker0 root refcnt 2 
qdisc noqueue 0: dev tunl0 root refcnt 2 
qdisc noqueue 0: dev calie15876440c3 root refcnt 2 
qdisc tbf 1: dev cali7f4c826eaba root refcnt 2 rate 1Kbit burst 21473b lat 4123.2s  # this is the iperf server
qdisc ingress ffff: dev cali7f4c826eaba parent ffff:fff1 ---------------- 
qdisc tbf 1: dev 911d root refcnt 2 rate 1Kbit burst 21473b lat 4123.2s 
qdisc noqueue 0: dev calif5ff665b457 root refcnt 2 
qdisc noqueue 0: dev calica9a09d9c09 root refcnt 2 

From my current tests it seems like the bandwidth plugin has no effect to other containers.

It seems like there is still a bug in the Kubernetes implementation (I will file a bug tomorrow, now it's time to sleep) :)

[caseydavenport]

Very cool - thanks @johscheuer for the great write up :)

[johscheuer]

The bug is already fixed in the 1.15 release (sadly not in the 1.14) https://github.com/kubernetes/kubernetes/commit/f4937619a2228a71ac62270d643dc869d3765d99

[ahrkrak]

Great to hear this is confirmed. Would be nice to see this included in Calico docs / standard manifests.

[johscheuer]

I can make the according changes in the docs/manifests. You just need to keep in mind that this currently is only supported with Docker as container runtime (since it's implemented in the dockershim) for other runtimes the CRI must implement it to (I opened a PR/issue for containerd).