Kind: Use container IP address in server field in kubeconfig for kind cluster

[font]

When working with multiple clusters such as with https://github.com/kubernetes-sigs/federation-v2, we provide the cluster info to the federation controller running in one of the clusters, to talk to all the other clusters. That cluster info comes from the kubeconfig provided by the user. The current kind create cluster command will create a kubeconfig with the server field using localhost as shown below:

$ KUBECONFIG=$(kind get kubeconfig-path) kubectl config view | grep server
    server: https://localhost:38785

This makes it problematic when one kind cluster needs to access the kube API server in another kind cluster because localhost for each kind cluster refers to itself. Since the kind clusters share a common docker network bridge, we can use the IP address for each container. So what is needed is for the container IP address to be used in the server field in the kubeconfig for the kind cluster as shown below:

$ KUBECONFIG=$(kind get kubeconfig-path) kubectl config view | grep server
    server: https://172.17.0.2:38785

The host would still be able to talk to the kube API servers using their container IP addresses. Is this something that is desirable? Is there a reason to use localhost instead?

[munnerz]

On OSX at least, it's not possible to access the docker container network 😞

If we want to make this configurable, there's 3 options I can think of:

• Add a boolean/toggle to the kind Config type
• Add a flag to kind create cluster to toggle the behaviour
• Add a new kind get kubeconfig to grab a kubeconfig for an existing cluster, and add a flag to that

Not sure what's most desirable here? 😄

[BenTheElder]

Each cluster has a different port for the API server so it should still be possible for them to talk to each other (?), the reason for localhost is that windows / mac do not support anything else as far as I know, because the container network is actually in a light VM.

[BenTheElder]

It's also possible that using a single network bridge will be problematic in the future, once we get multiple nodes. Using localhost and a random port makes things pretty portable.

Can federation support keying off of the address + port instead of just the address instead?

It would be quite easy to create a kubeconfig with the container IP address when on linux, but so far I've preferred keeping the environment as consistent as possible. Ideally users should be able to replicate CI very closely on their local machines, which is a major goal for kind.

[font]

Each cluster has a different port for the API server so it should still be possible for them to talk to each other (?), the reason for localhost is that windows / mac do not support anything else as far as I know, because the container network is actually in a light VM.

On Linux, I don't think it's possible while using localhost because 127.0.0.1 is not routable from one container to another i.e. across the network bridge.

Can federation support keying off of the address + port instead of just the address instead?

We key off of the entire server field so we include the port. It's just that using localhost:<port_for_other_kind_cluster_api_server> returns a connection refused error because the local container (referred to by using localhost) is not listening on that port since it's part of the other cluster's container network. So the pod running inside the kind-1 cluster needs to have a routable IP address to the API server running in the kind-2 cluster.

[BenTheElder]

Ah, right. Thanks. Will think about this some more, while you can trivially get the container IP and rewrite it currently, it would be ideal to make this work portably.

[fabriziopandini]

@BenTheElder @font
Currently I'm working on multi node and I want to be sure my effort is compliant with federation test requirements

What I'm proposing for multi-node is to

• always use IP:6443 as API server address for "infra-docker" communication. This allows multiple cluster but gets out of the way random port for node join/API server load balancing
• only when copying the kubeconfig on the host machine, use localhost + random port

What described above is a small variation of current state, and if I got this thread right, it neither fix nor prevent fixing the above issue for federation test. is that right or do you see problems in what I'm proposing?

Instead, if I can give my two cents on how to address federation requirement, I think that a possible solution is to add an option for retrieving the raw kubeconfig (kind get kubeconfig --raw, that will give you exactly the kubeconfig that exist in docker/with IP:6443), that can be reused by federation controller - assuming it will be run in docker too -. Wdyt?

[neolit123]

i think we should make the random port a UX knob.

[BenTheElder]

--raw makes sense, BUT, we need the random port outside the container to handle docker for mac / windows and I'd rather minimize code paths because CI should match local usage for reproducibility as much as possible..

[BenTheElder]

for random port see #178, we should allow setting the address as well, we'll set a random port if you don't specify a port.

[BenTheElder]

neglected to update here: the random port is a config knob now, and it is always 6443 inside the network, we still need to add a command to export the kubeconfig, with a --raw option or similar.

[aojea]

I think that as a workaround this will work
docker exec -it $CLUSTER_NAME-control-plane cat /etc/kubernetes/admin.conf

[BenTheElder]

kind get kubeconfig --internal > internal-kubeconfig after https://github.com/kubernetes-sigs/kind/pull/478