Rke: Support IPv6DualStack

[thuck]

RKE version:
v1.0.4

Docker version:
18.09.1

Operating system and kernel:
Debian GNU/Linux 10 (buster), 4.19.0-6-amd64

Type/provider of hosts:
KVM

cluster.yml file:

nodes:
  - address: 192.168.122.254
    internal_address: 192.168.122.254
    role:
      - controlplane
      - etcd
      - worker

ignore_docker_version: false
cluster_name: cluster.local
kubernetes_version: v1.17.2-rancher1-2
addon_job_timeout: 300

services:
  kube-api:
    service_cluster_ip_range: 10.42.0.0/24
    service_node_port_range: 30000-32767
    pod_security_policy: false
    always_pull_images: false
    extra_args:
      audit-log-path: "/var/log/audit/"
      runtime-config: 'authorization.k8s.io/v1beta1=true,apps/v1beta1=true,apps/v1beta2=true,extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true'
      delete-collection-workers: 3
      feature-gates: "SCTPSupport=True,IPv6DualStack=true"

  kube-controller:
    cluster_cidr: "10.43.0.0/24,fc00::/64"
    service_cluster_ip_range: "10.42.0.0/24,fc01::/64"
    extra_args:
      feature-gates: "IPv6DualStack=true"

  kubeproxy:
    cluster_cidr: "10.43.0.0/24,fc00::/64"
    extra_args:
        feature-gates: "IPv6DualStack=true"
        proxy-mode: "ipvs"

  kubelet:
    cluster_domain: cluster.local
    cluster_dns_server: 10.42.0.3
    fail_swap_on: false
    extra_args:
      max-pods: 20
      pod-manifest-path: /etc/kubernetes/manifests
      feature-gates: "IPv6DualStack=true"
      v: 2

authorization:
  mode: rbac

network:
  plugin: calico

ingress:
  provider: none

dns:
  provider: coredns

Steps to Reproduce:
rke up
Results:
Calico fails to create an IPV6 network:
2020-02-04 15:53:28.006 [ERROR][8] startup.go 779: Invalid CIDR specified in CALICO_IPV4POOL_CIDR '10.43.0.0/24,fc00::/64'

Template for calico do not contain proper environment variable:

- name: CALICO_IPV6POOL_CIDR
  value: "fd20::0/112"
- name: IP6
  value: "autodetect"

Calico documenation

[lwolf]

I also tried to deploy dual-stack. Worked around calico issue described above by using network.plugin: none and deploying calico separately.
Faced another issue though: rke doesn't allow to set service_cluster_ip_range for kube-apiserver to more that one subset.

failed set network options: Failed to get Kubernetes Service IP: Failed to get kubernetes service IP from Kube API option [service_cluster_ip_range]: invalid CIDR address: 10.43.0.0/16,2a00:xxxx:xxxx:xxxx::/112

So, currently I have semi-working setup running: pods are dual-stack, but services are not.

the issue is in GetKubernetesServiceIP function here: https://github.com/rancher/rke/blob/4c1d3db2b0c1ff36409c5cff658a6f5a9d1833e0/pki/util.go#L382
it used to validate service IP and can't parse array of values.

[lwolf]

Don't know rke codebase well enough, but this seems to be enough to make it work. At least I was able to deploy dual-stack cluster.

https://github.com/lwolf/rke/commit/b332b568e82591d3df459590487fa3a41b61ed5c

Can make a PR if somebody from rancher could confirm that this looks ok.

[ghakfoort]

any news on this one?

I try to create a dual stack enabled cluster. According to the Kubernetes documentation, a few settings need to be set correctly:

kube-apiserver:
--feature-gates="IPv6DualStack=true"
--service-cluster-ip-range=,
kube-controller-manager:
--feature-gates="IPv6DualStack=true"
--cluster-cidr=,
--service-cluster-ip-range=,
--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6 defaults to /24 for IPv4 and /64 for IPv6
kubelet:
--feature-gates="IPv6DualStack=true"
kube-proxy:
--cluster-cidr=,
--feature-gates="IPv6DualStack=true"

This can be done for the Kube-controller:
services:
kube-controller:
cluster_cidr: "10.42.0.0/24,fc00::/64"
service_cluster_ip_range: "10.43.0.0/24,fc01::/64"
extra_args:
feature-gates: "IPv6DualStack=true"
node-cidr-mask-size-ipv4: 24
node-cidr-mask-size-ipv6: 64

but it fails at the Kube-api:
services:
kube-api:
service_cluster_ip_range: "10.43.0.0/24,fc01::/64"
extra_args:
feature-gates: "IPv6DualStack=true"

I don't seem to be able to configure 2 (ipv4 and ipv6) ip ranges in the service_cluster_ip_range of the kube-api.
If I only configure the ipv4 ip range. "rke up" works correctly. But ofcourse, dual stack isn't configured well. If I manage calico manually, I'm able to assign ip addresses in both stacks to the pods and egress traffic works OK but I need to solve the whole puzzle :)

[JustInVTime]

:+1: on this one. I know it's still an alpha feature in kubernetes, but (experimental) support in RKE would be very nice!

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[n1kofr]

Is there any plan / roadmap on RKE to have this feature available?

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[n1kofr]

Any news for this issue?

[WebSpider]

I'm very interested in this as well, since my cloud provider has support for IPv6 in all relevant areas

[c-mckenzie]

I'd also be very keen to get any update on this issue, especially with the upcoming promotion of dual-stack support in k8s 1.21

[pd-dfinity]

+1 we would have to abandon RKE if IPv6 Dualstack is not supported soon.