Kind: Allow users to configure MTU on kindnet

[faiq]

What would you like to be added:

There should be an easy way to configure MTU values on the bridge that kind net creates.

Why is this needed

We're using KIND heavily for our CI testing purposes. Our CI workflows run on Kubernetes and in order to get kind to run properly we run a pod with a dind container that starts kind and our tests run against this cluster. We were running into many issues running our tests that make network request from this docker in docker in docker environment. After hours of debugging we realized that the MTU was configured too high and once we ran our inner most container with the following args

args: ["-c", "ip link set mtu 1400 dev eth0 && /manager --enable-leader-election"]

the issue was mitigated.

If we were able to pass in some configuration via our kind-config.yaml it would be greatly appreciated!

Please let me know if you'd like me to add in more information.

Thank you for you time and consideration.

[BenTheElder]

cc @aojea
/priority backlog

[aojea]

Our CI workflows run on Kubernetes

what CNI uses that Kubernetes? is it an overlay?

Because you should not have MTU issues if there is no overlay involved, because kindnet does not encapsulate anything, neither docker does.

Anyway, kindnet uses the ptp CNI plugin and it does have an option to configure the MTU :smile:

https://github.com/containernetworking/plugins/tree/master/plugins/main/ptp

[aojea]

Ok, I've tested and the MTU option works, we just need to add "mtu": 1400, to the CNI config manifest

https://github.com/kubernetes-sigs/kind/blob/54471c23b742b75c8c9baaa6979bd63576c618f3/images/kindnetd/cmd/kindnetd/cni.go#L60-L62

@BenTheElder what will be the easier way to make it configurable?

passing it as an environment variable to kindnet and templating it, like we are doing with the POD_SUBNET?
https://github.com/kubernetes-sigs/kind/blob/54471c23b742b75c8c9baaa6979bd63576c618f3/pkg/build/node/cni.go#L157-L158

[faiq]

@aojea I think passing it in via an environment variable would be completely fine :) i didn't realize that it was using the ptp module (or that it even existed) thanks so much for looking into this!

[aojea]

@aojea I think passing it in via an environment variable would be completely fine :) i didn't realize that it was using the ptp module (or that it even existed) thanks so much for looking into this!

@faiq in the meantime you can workaround it manually, processing the file/etc/cni/net.d/10-kindnet.conflist, using docker exec per example, and adding the MTU option.

[faiq]

@aojea I'm looking at this again we're using Calico and running an Overlay network with IPIP encapsulation. We set an mtu on that network of 1480.

Furthermore, we run the following command to start the docker daemon in our entrypoints for our node images

dockerd \
  --cgroup-parent="${CGROUP_PARENT}" \
  --bip="${DOCKERD_BIP:-172.17.1.1/24}" \
  --mtu="${DOCKERD_MTU:-1400}" &

I hope this helps give insight into the problem that we're facing.

I'd like to submit a patch for mtu configuration. I'm going to try to put some stuff together for review

[aojea]

@faiq please bear with me, so we can try to find a standard solution for KIND to this problem:

So, we have the docker bridge that in this case emulates the underlay network of the KIND cluster.
By default the docker network sets the mtu to 1500.
Inside the docker containers (kind nodes) we have the pods, that take the MTU from the CNI plugin (by default we have kindnet)

And from your explanation, you are running kind inside a k8s cluster that uses calico and forces you to have your docker bridge to 1400.

So, I think that the best way to solve this is:

• Add an option to kind to configure the MTU in the docker bridge (we create it since 0.8.1)
• Make kindnet to detect the node interface and autoconfigure the MTU of the pods.

The changes are relatively easy and this way the API has better semantics

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  mtu: 1480

will set the cluster MTU here
https://github.com/kubernetes-sigs/kind/blob/7620a6b488866a57cac5e73fc4540c348c6437d0/pkg/cluster/internal/providers/docker/network.go#L94

adding --mtu networking.mtu

For kindnet instead of taking the MTU value from an environment variable, it can search for the default interface or just for eth0 directly, and take it's mtu,
i.e. https://ispycode.com/GO/Network/System-network-interfaces

@BenTheElder thoughts?

[jieyu]

I like Make kindnet to detect the node interface and autoconfigure the MTU of the pods.

[faiq]

Both of those suggestions seem good to me.

I can adjust the code that I wrote here https://github.com/kubernetes-sigs/kind/pull/1686 to make it modify the mtu value on the docker bridge. Right now my code takes in the value there and sets for tempting.

Having kindnet detect node interface and autoconfigure the MTU sounds straightforward as well.

[aojea]

@faiq I prefer to submit one PR per feature if you don't mind.

Let's address first the kindnet MTU autodiscovery because this will automatically solve your issue, since you are already modifying the docker bridge mtu

Add an option to kind to configure the MTU in the docker bridge (we create it since 0.8.1)

I've talked with Ben offline and I think we should analyze more possible consequences of this change, what happen on upgrades, with different clusters on the same bridge, if the bridge was already created with a different mtu, ... are we going to support different bridges per cluster? we should answer these questions before implementing it

[BenTheElder]

https://github.com/kubernetes-sigs/kind/pull/1690 allows this to be configured on kindnet(d) (by way of the bridge)

_technically_ this is now possible if you create the kind network custom yourself before kind does.

next up we should consider how that might be done better, for example kind should probably copy the mtu from the default bridge when creating the custom bridge (we'd be using the standard network if it had the DNS bits)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[aojea]

I think we can close this issue because of https://github.com/kubernetes-sigs/kind/pull/1690
/close

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

I think we can close this issue because of https://github.com/kubernetes-sigs/kind/pull/1690
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.