Kind: Documentation for inter-cluster direct pod to pod communication

[howardjohn]

What would you like to be documented:

Capturing Slack conversation:
John Howard 19 hours ago
Is it possible to do something similar to "--enable-ip-alias" in GKE? This creates a "VPC-native cluster" which allows direct pod-pod communications by internal IP between two clusters. I am hoping to do the same on kind, since currently we have a test that does this on GKE (our last test not on kind). Any ideas? I am not super familiar with this area so hopefully what I described is accurate

aojea 19 hours ago
is it possible, but it will need some glue :slightly_smiling_face:

aojea 19 hours ago
my understanding is that you wont that pod from cluster A communicate directly with pod from cluster B, can you confirm? (not familiar with that enable-ip-alias feature) (edited)

John Howard 19 hours ago
Yes, by internal IP.

aojea 10 hours ago
you have to assign different podSubnet to each cluster

aojea 10 hours ago
and then add routes into the nodes

Why is this needed:

For our use case, we have existing tests on GKE with this setup that we want to move over to kind. I am not sure how common this is though.

I will try to find some to get this test working this week, so I may be able to contribute this if I can get it working.

[BenTheElder]

I want to point out first that doing this will require depending on some internal details that are otherwise swappable:

• The CNI / network daemon ("kindnetd") is responsible for pod routing, if we need to switch back to a third party maintained solution (eg calico) in the future, the routing will be different. CNI / Kubernetes do not define joining pod networks between clusters, I'm not sure what implementing this would look like against other CNI. It should be relatively simple against kindnetd.

• currently kind cluster nodes are all on the default docker bridge network, that is likely to change in the future in order to fix things like host restart support (with a per-cluster bridge instead), however we can try to come up with some multi-cluster solution (it will be pretty easy to allow creating them on the same network, however you will likely lose things like host restart again)

[aojea]

This is simple routing, there is no problem creating clusters in different networks, just it needs to add a few routes more in the host and be careful with the iptables rules.

Regarding CNIs, the only problem is with the iptables rules they insert, kindnet is so simple that makes it perfect for this scenario.

So far so good, in order to make this work with current kind version you can create 2 cluster with different podSubnet and serviceSubnet to avoid overlapping. Let's say:

• cluster1 podSubnet: 10.10.0.0/16 and serviceSubnet: 10.255.10.0/24
• cluster2: podSubnet: 10.20.0.0/16 and serviceSubnet: 10.255.20.0/24

Each cluster will have a different podSubnet assigned in each node, i.e.:

• cluster1 will have 10.10.1.0/24 in nodeA1, node A1 has the address 172.17.0.3 (assigned by docker)
• cluster2 will have 10.20.1.0/24 in nodeB1, node B1 has the address 172.17.0.14 (assigned by docker)

In order to enable pods in node B1 to communicate with nodeA1 you just have to:

• Add a route in A1 towards pod subnet in B1 through B1 address: ip route add 10.20.1.0/24 via 172.17.0.14
• Add a route in B1 towards pod subnet in A1 through A1 address: ip route add 10.10.1.0/24 via 172.17.0.3
• Make sure that those subnets are not being masqueraded (I have to check it more in detail, but I think we can pass an option to kindnet to not masq them)

Of course this can be easily automated with a routing protocol , but a simple script will do the work, just iterating through the nodes and inserting the necessary routes and iptables rules.

This is basically how kindnet works :sweat_smile:

[howardjohn]

@aojea seems to work with some minor changes. In step 1 you mention

cluster1 podSubnet: 10.10.0.0/16 and serviceSubnet: 10.255.10.0/24

But then later reference 10.10.1.0/24. I had to change this to either 10.10.0.0/24 or 10.10.1.0/16. Am I doing something wrong there?

Here is my script:

#!/bin/bash

SUBNET_A="10.10.0.0/16"
SUBNET_B="10.20.0.0/16"

CONFIG_A=$(cat <<EOF
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  podSubnet: ${SUBNET_A}
  serviceSubnet: 10.255.10.0/24
EOF
)
CONFIG_B=$(cat <<EOF
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  podSubnet: ${SUBNET_B}
  serviceSubnet: 10.255.10.0/24
EOF
)
kind create cluster --name a --config <(echo "${CONFIG_A}")
kind create cluster --name b --config <(echo "${CONFIG_B}")

DOCKER_IP_A=$(docker inspect -f "{{ .NetworkSettings.IPAddress }}" a-control-plane)
DOCKER_IP_B=$(docker inspect -f "{{ .NetworkSettings.IPAddress }}" b-control-plane)

docker exec a-control-plane ip route add 10.20.0.0/16 via ${DOCKER_IP_B}
docker exec b-control-plane ip route add 10.20.0.0/16 via ${DOCKER_IP_A}

[aojea]

docker exec a-control-plane ip route add 10.20.0.0/16 via ${DOCKER_IP_B}

I didn't explain it correctly, podSubnet is splitted in smaller subnets assigned to each node, this is the value that you have to obtain.
You can obtain it from the cni config in the node, that 10.10.1.0/24 is the subnet assigned to that node from the whole range 10.10.0.0/16

root@kind-control-plane:/# grep subnet /etc/cni/net.d/10-kindnet.conflist 
                                        "subnet": "10.10.1.0/24"

or from the kubernetes API node.Spec.PodCIDR}

[howardjohn]

@aojea thanks for that help, that seems to work! Here is my refined script:

#!/bin/bash

set -eux

SUBNET_A="10.10.0.0/16"
SUBNET_B="10.20.0.0/16"

CONFIG_A=$(cat <<EOF
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  podSubnet: ${SUBNET_A}
  serviceSubnet: 10.255.10.0/24
EOF
)
CONFIG_B=$(cat <<EOF
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  podSubnet: ${SUBNET_B}
  serviceSubnet: 10.255.10.0/24
EOF
)
kind create cluster --name a --config <(echo "${CONFIG_A}") --wait 60s
kind create cluster --name b --config <(echo "${CONFIG_B}") --wait 60s

DOCKER_IP_A=$(docker inspect -f "{{ .NetworkSettings.IPAddress }}" a-control-plane)
DOCKER_IP_B=$(docker inspect -f "{{ .NetworkSettings.IPAddress }}" b-control-plane)

POD_CIDR_A=$(KUBECONFIG="$(kind get kubeconfig-path --name a)" kubectl get node -ojsonpath='{.items[0].spec.podCIDR}')
POD_CIDR_B=$(KUBECONFIG="$(kind get kubeconfig-path --name b)" kubectl get node -ojsonpath='{.items[0].spec.podCIDR}')

docker exec a-control-plane ip route add ${POD_CIDR_B} via ${DOCKER_IP_B}
docker exec b-control-plane ip route add ${POD_CIDR_A} via ${DOCKER_IP_A}

[aojea]

/close

Thanks for sharing

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close

Thanks for sharing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.