Kind: Allow configurable docker network for kind cluster nodes

[foolusion]

It would be nice to be able to specify the network that the cluster uses.

[BenTheElder]

/kind feature
/priority important-longterm
/assign

[BenTheElder]

We definitely want this, I think we're going to put it in the networking config, and start having an automated default.

cc @aojea @neolit123

Strawman:

• if the specified network exists, we use it without creating
• if it doesn't exist, we create it, and label it as belonging to the cluster
• on cluster delete, we list networks labeled with the cluster, and delete only those (so not just whatever the containers use, only if we labeled it)
• we name this field / functionality somehow such that it is clear that this feature is docker specific, leaving room for podman etc. in the immediate future xref #154.

[neolit123]

Strawman:

SGTM

do we need to have the config field?
when a cluster is created we can auto-manage a network with the same name or prefixed similarly?
e.g.
kind-network-kind
kind-network-mycluster

[BenTheElder]

I think we need a field with defaulting to the generated one, that way EG
federation can make multiple clusters on the same network by
actually setting the field

On Thu, May 2, 2019 at 5:13 PM Lubomir I. Ivanov [e-mail]
wrote:

Strawman:

SGTM

do we need to have the config field?
when a cluster is created we can auto-manage a network with the same name
or prefixed similarly?
e.g.
kind-network-kind
kind-network-mycluster

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/273#issuecomment-488876015,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHADK6PWFVIVH5K5RQVFB3PTN7UBANCNFSM4GUPO6FQ
.

[neolit123]

ok, makes sense.

[aojea]

seems that docker has an option to populate the /etc/hosts file, that can be useful to get rid of the loopback address in the resolv.conf and keep the node name resolution

Managing /etc/hosts
Your container will have lines in /etc/hosts which define the hostname of the container itself as well as localhost and a few other common things. The --add-host flag can be used to add additional lines to /etc/hosts.

$ docker run -it --add-host db-static:86.75.30.9 ubuntu cat /etc/hosts
172.17.0.22     09d03f76bf2c
fe00::0         ip6-localnet
ff00::0         ip6-mcastprefix
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
86.75.30.9      db-static

[BenTheElder]

the only problem with --add-host is we don't know the other nodes's IPs when we call docker run, it's a bit chicken and egg :^)

[BenTheElder]

This turned out to have a few more issues that we expected due to non-default docker networks having different behavior. This may slip to 0.5 as we're nearing the 0.4 release, but it's definitely something we want.

[jayunit100]

Hi . ! can someone disambiguate use cases here, between this and the https://github.com/kubernetes-sigs/kind/issues/278 issue ?

[aojea]

@jayunit100 there are different things regarding networking, one is the CNI plugin used by the kubernetes cluster, kind installs its own CNI by default but you can disable it and install your preferred CNI plugin once kind finish creating the cluster.
The other networking part is the one that docker provides, that's where kind spawn the nodes. Currently kind only supports docker, with its networking limitations. Using another bridge in docker has some consequences that break kinds because of different things.

[jayunit100]

So docker0 is only being used for node IP addresses in the use case for this issue? Thanks for clarifying! Was confused :) . Curious what the use case is for not using docker0 at that level ... after all kind as an abstraction for testing k8s is sufficient as long as the k8s specific stuff isn’t impacted by Dockers impl as a hyper visor for virtual nodes, right?

[jayunit100]

Mostly get it now.. maybe change the title of this issue to “use non docker0 interface for kubelet IPs” (although imprecise I think it gets the point across) so that is clear what we mean by cluster :):)... thanks again! The CNI feature for kind is definetly awesome, want to make sure people know that it works as is :).
Ps for context am looking at using kind instead of my vagrant recipes for some aspects of some calico tests .

[aojea]

There are calico folks using kind for testing, as you can see in this slack conversation https://kubernetes.slack.com/archives/CEKK1KTN2/p1570036710217000 , maybe you can bring up this conversation in our slack channel

The main problem of using a custom bridge with docker is that it modifies the DNS behavior, using an embedded dns server https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/

[zephinzer]

this would be nice- currently facing an issue of not being able to resolve an internal image registry which is behind my org's vpn. what work is left regarding this? maybe i can help!

[BenTheElder]

kind uses a specific network now in HEAD (kind) as part of some unrelated work.

as it currently stands kind will not delete any networks, so you can just precreate the kind network with your desired settings.

we need to revisit how that works a bit though WRT IIPv6 in a follow up PR before moving forward.

[BenTheElder]

https://github.com/kubernetes-sigs/kind/pull/1538 will make it possible to do this.

you shouldn't actually need this in nearly all cases though, kind is switching to ensure and use a "kind" network with all of the features of a user defined network.

if you pre-create this network it will use it as you configured, it does not delete networks.

[BenTheElder]

most of the problem initially was just that user defined networks are a breaking change in docker vs the bridge, they have different DNS in ways that don't trivially work with kind.

we've fixed that and always use one now.

the remaining issues are that completely arbitrary networks can be ... very strange.
for now we're provisioning our own under a fixed name unless it already exists.

this network is a standard bridge, with an IPv6 subnet picked out of ULA.