Is your feature request related to a problem? Please describe.
Since 1.9 K8s supports ipv6-only but it is still in alpha after 5 minor releases and >1.5 years. In that sense it does not fit in the k3s concept with "no alpha features". However the main reason for the lingering alpha state is lack of e2e testing. This is aggressively addressed now for the upcoming dual-stack support in k8s.
To bring up a ipv6-only k8s cluster is currently not for the faint hearted and I think if the simplicity of k3s can also include ipv6 it would be greatly appreciated. Also dual-stack is on the way IMHO support for ipv6-only is an important pro-active step.
Describe the solution you'd like
A --ipv6 option :smile:
This would setup node addresses, service and pod CIDRs etc with ipv6 addresses but keep the image loading (containerd) configured for ipv4. The image loading should still be ipv4 because the internet and ISP's are still mostly ipv4-only and for ipv6 users the way images gets loaded is of no concern.
A requirement will then be that the nodes running k3s must have a working dual-stack setup (but the k8s cluster would be ipv6-only).
Describe alternatives you've considered
The "not for the faint hearted" does not mean that setting up an ipv6-only k8s cluster is particularly complex, more that most users have a fear of the unknown and that support in the popular installation tools is lacking or not working. To setup k8s for ipv6-only is basically just to provide ipv6 addresses in all configuration and command line options. That may even be possible without modifications to k3s (I have not yet tried). It may be more complex to support the "extras" such as the internal lb and traefik, so I would initially say that those are not supported for ipv6. Coredns with ipv6 in k3s should be supported though (coredns supports ipv6 already).
The flannel CNI-plugin afaik does not support ipv6 (issue). So the --no-flannel flag must be specified and a CNI-plugin with ipv6 support must be used.
Additional context
I will start experimenting with this and possibly come up with some PR's. The ammount of time I can spend may be limited.
I am currently adding k3s in my xcluster environment where I already have ipv6-only support in my own k8s setup.
uablrek
All 24 comments
I'm definitely in favor of this. If you can provide PRs we can figure out what we need to do to support it. Ideally I would like dual stack on by default for k3s. If ipv6 only is a good stepping stone for that then lets do it.
ibuildthecloud
on 31 Mar 2019
First ipv6 try without any modifications to k3s;
Server
k3s server --no-flannel --no-deploy coredns \
--no-deploy servicelb --no-deploy traefik --disable-agent \
--write-kubeconfig /etc/kubernetes/kubeconfig --tls-san 192.168.0.1 \
--cluster-cidr 1000::2:11.0.0.0/112 --service-cidr fd00:4000::/112 \
--node-ip 1000::1:192.168.1.1
When specifying an ipv6 --cluster-cidr 1000::2:11.0.0.0/112 the server this happens;
time="2019-04-03T12:22:46.467223177Z" level=info msg="k3s is up and running"
time="2019-04-03T12:22:48.468064815Z" level=info msg="Handling backend connection request [vm-004]"
time="2019-04-03T12:22:48.474068645Z" level=info msg="Handling backend connection request [vm-002]"
F0403 12:22:48.476902 207 node_ipam_controller.go:98] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size
... (program crash follows...)
The flag --node-cidr-mask-size=120 must be given to kube-controller-manager.
Is there a way to specify extra flags to kube-controller-manager?
While I am at it; I propose that if --no-flannel is specified do not set default --allocate-node-cidrs=true to kube-controller-manager since the CNI may not support it and just ignode the node cidr in the k8s node object. It works anyway but the node object will have an invalid value (my case).
Agent
k3s agent --server https://192.168.1.1:6443 \
--no-flannel --containerd-config-template /etc/containerd.conf \
--node-ip 1000::1:192.168.1.4
As you can see the server url is still ipv4. I think it doesn't matter since the server opens 6443 for ipv6 (:::6443) even in an ipv4-only cluster.
time="2019-04-03T12:22:48.547178739Z" level=error msg="Failed to connect to proxy" error="websocket: close 1006 (abnormal closure): unexpected EOF"
I0403 12:22:48.547264 207 log.go:172] http: proxy error: EOF
W0403 12:22:48.547412 207 node.go:103] Failed to retrieve node info: an error on the server ("") has prevented the request from succeeding (get nodes vm-004)
I0403 12:22:48.547425 207 server_others.go:148] Using iptables Proxier.
W0403 12:22:48.547488 207 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
F0403 12:22:48.547501 207 server.go:396] unable to create proxier: clusterCIDR 1000::2:b00:0/112 has incorrect IP version: expect isIPv6=false
I think the first fault is caused by the crashing server and the second problem is a cause of the first.
uablrek
on 3 Apr 2019
Is there a way to specify extra flags to kube-controller-manager?
Just saw #290
uablrek
on 3 Apr 2019
Applied PR https://github.com/rancher/k3s/pull/309 and started with;
# Server;
k3s server --no-flannel --no-deploy coredns \
--no-deploy servicelb --no-deploy traefik --disable-agent \
--write-kubeconfig /etc/kubernetes/kubeconfig --tls-san 1000::1:192.168.1.1 \
--cluster-cidr 1000::2:11.0.0.0/112 --service-cidr fd00:4000::/112 \
--node-ip 1000::1:192.168.1.1 \
--kube-controller-args node-cidr-mask-size=120
# Agent;
k3s agent --server https://[1000::1:192.168.1.1]:6443 --no-flannel \
--containerd-config-template /etc/containerd.conf --node-ip 1000::1:192.168.1.2 \
--kubelet-args address =:: --kubelet-args healthz-bind-address=::1 \
--kube-proxy-args bind-address=::1
And;
# k3s kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
vm-002 NotReady <none> 7m45s v1.13.5-k3s.1 1000::1:c0a8:102 <none> Xcluster 5.0.0 containerd://1.2.4+unknown
:smile:
There is still a problem with image loading, containerd insist on using ipv6 addresses;
E0408 09:01:40.897457 474 remote_runtime.go:96] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to get sandbox image "k8s.gcr.io/pause:3.1": failed to pull image "k8s.gcr.io/pause:3.1": failed to resolve image "k8s.gcr.io/pause:3.1": no available registry endpoint: failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.1: dial tcp [2a00:1450:4010:c05::52]:443: connect: network is unreachable
I will try to pre-load for testing and also configure containerd. Or, if it comes to that, resort to cri-o which I know can download images over ipv4 in an ipv6-only cluster.
uablrek
on 8 Apr 2019
My mistake. My routing from within the cluster was bad. Now it works;
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
alpine-deployment-797f999977-5spzw 1/1 Running 0 25s 1000::2:b00:302 vm-003 <none> <none>
alpine-deployment-797f999977-npknn 1/1 Running 0 25s 1000::2:b00:402 vm-004 <none> <none>
alpine-deployment-797f999977-pckfb 1/1 Running 0 25s 1000::2:b00:202 vm-002 <none> <none>
alpine-deployment-797f999977-wqfxs 1/1 Running 0 25s 1000::2:b00:303 vm-003 <none> <none>
Note the ipv6 addresses.
So it seems that k3s with PR https://github.com/rancher/k3s/pull/309 has support for ipv6 :grin:
I will now see what needs to be done for coredns and external access (nothing I hope).
uablrek
on 8 Apr 2019
External access to services with externalIPs: works fine.
But access to the kubernetes service from within a pod does not work. The endpoint for the kubernetes service is;
# kubectl get endpoints kubernetes
NAME ENDPOINTS AGE
kubernetes [::1]:6445 90m
This is OK, but on the agent nodes port 6445 is only open for ipv4;
# netstat -putln | grep 6445
tcp 0 0 127.0.0.1:6445 0.0.0.0:* LISTEN 217/k3s
Among other things this prevents coredns to work for cluster addresses since it can't connect to the api-server.
I am unsure where this address is set. Any hint is appeciated.
It's so close...
uablrek
on 8 Apr 2019
uablrek
on 8 Apr 2019
# kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 127.0.0.1, fd00:4000::1, not ::1
uablrek
on 8 Apr 2019
I made a PR https://github.com/rancher/k3s/pull/319 that fixes the problems above but access to the kubernetes api service [fd00:4000::1]:443 does not work from within a pod. From main netns you can connect;
# wget https://[fd00:4000::1]:443/
Connecting to [fd00:4000::1]:443 ([fd00:4000::1]:443)
wget: server returned error: HTTP/1.1 401 Unauthorized
It seems like when the DNAT rule kubernetes-service:443->[::1]:6443 is applied then the packets to the cni bridge interface (cbr0) is dropped. Since flannel does not support ipv6 "bridge" cni is used for ipv6. With tcpdump I can see packets coming to kubernetes-service:443 on the "veth" device from the pod, but they do not appear on the bridge device (cbr0).
I suspect that some sysctl for ipv6 needs to be set (like rp_filter for ipv4), but I have not found anything except forwarding (which is on).
A correstonding trace on ipv4 with flannel shows packets with translated addresses on the cni0 interface, e.g 127.0.0.1.6445 and untranslated addresses on the veth device, e.g. 10.43.0.1.443.
uablrek
on 23 Apr 2019
The problem seem to be that the localhost is set as destination which is detected as a martian destination by the kernel.
However, this also applies for ipv4 but the DNAT to 127.0.0.1:6443 works in k3s. I tried to setup this in a "bare VM" environment but could not get rid of the "martian destination" for ipv4. I could not figure out how k3s makes this work.
@ibuildthecloud Can you please explain how you manage to get rid of the "martian destination" problem for ipv4 in k3s?
Then I hope to be able to do the same thing for ipv6.
uablrek
on 2 May 2019
Looking at https://en.wikipedia.org/wiki/Martian_packet, I am curious if it might have something to do with iptables or the flannel/cni setup.
erikwilson
on 7 May 2019
Hey I'm interested in IPv6 for K3S too because I plan to switch from docker swarm because they still haven't implemented IPv6 on swarm.
Do you have any news about ipv6 on k3s? I saw that project: https://docs.projectcalico.org/v3.7/usage/ipv6 but I don't know if it would work on K3S.
unixfox
on 1 Jun 2019
@unixfox I got other things to do so I couldn't work with ipv6 on k3s. I noticed that my PR for fixing the certificate problem does not work on 0.5.x. I think the problem is small. IFAIK the "martian packet" problem is still a stopper, but I think it's the only one. Almost all ipv6 works with just re-configuration as described above, but access to the API from agents needs a corrected certificate fix and access to the API from pod's are stopped by the martian-packet problem. "normal" ipv6 traffic via services works thogh.
About the CNI-plugin you must select one that supports ipv6, Calico is one as you have seen. The CNI-plugin is started and configured separate from k3s so you must use the instructions for the cni-plugin. You can configure Calico to hand out both ipv4 and ipv6 addresses so you get "dual-stack" on PODs even though k3s can't handle it. Actually I recommend that as a first step if you are using Calico.
uablrek
on 2 Jun 2019
This issue should migrate to; "Support for dual-stack Kubernetes cluster" or be closed.
The ipv6 "martian" problem will probably not be present in a k8s dual-stack cluster since the API-server communication will still be ipv4. So adapt to dual-stack is likely simpler than ipv6-only.
uablrek
on 21 Nov 2019
@uablrek K8s release with dual-stack is out and k3s can install it...is this now confirmed working and supported?
zoza1982
on 5 Dec 2019
I don't know. If flannel is used, then no, since flannel only support ipv4. I am using k8s now since I need dual-stack.
uablrek
on 5 Dec 2019
Removed my previous comment...
Looks like in order to enable this need to do two things,
- Configure Calico & BGP with router
- Configure two Pod/Services within the k3s launch per: https://github.com/rancher/k3s/issues/284#issuecomment-479477634 and https://kubernetes.io/docs/concepts/services-networking/dual-stack/
- It's my understanding that K3S doesn't support gated features in order to remain thin, is that accurate? Will tear down / rebuild my cluster in the next couple of days to see if it actually works.
I've got an extra IPv6 /64 from Comcast that I'll use.
carpenike
on 8 Feb 2020
Opened #1405 to track dual stack
carpenike
on 10 Feb 2020
Isn't now IPV6 only cluster in beta for 1.18: https://github.com/kubernetes/enhancements/issues/508?
unixfox
on 26 Mar 2020
Found the [::1]:6443 martian problem; net.ipv4.conf.all.route_localnet=1 is set by K8s, but there is no corresponding setting for ipv6. Please see; https://github.com/kubernetes/kubernetes/issues/90259
uablrek
on 29 Apr 2020
Trying to build a k3os IPv6 only cluster with no more success, and as I can't spend more time at the moment to try having a full IPv6 only cluster, I published some personal notes about "Unsuccessful attempt to deploy CIRRUS cluster IPv6 only with k3os" on a alternative git site.
see https://framagit.org/snippets/5803.
Hope this can be useful to go further in that quest...
j-landru
on 12 Jun 2020
FYI, I update my personal notes ( https://framagit.org/snippets/5803), as Installing Calico operator helps going further.
It seems that I have now a ipv6 only k3os/k3s cluster !!
# kubectl get nodes,pods -A -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/k3os-cirrus-v2-server-ox0000 Ready master 156m v1.18.3+k3s1 2001:db8:3111:80:5:0:400:0 <none> k3OS v0.11.0-m1 5.4.0-29-generic containerd://1.3.3-k3s2
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tigera-operator pod/tigera-operator-6659cdcd96-5kgpl 1/1 Running 0 149m 2001:db8:3111:80:5:0:400:0 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/metrics-server-7566d596c8-stx49 1/1 Running 0 156m 2001:db8:3111:80:90d5::e800 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/helm-install-traefik-vkbbs 0/1 Completed 1 156m 2001:db8:3111:80:90d5::e803 k3os-cirrus-v2-server-ox0000 <none> <none>
default pod/busybox 0/1 Completed 0 56m 2001:db8:3111:80:90d5::e807 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/local-path-provisioner-6d59f47c7-5fvxv 1/1 Running 1 156m 2001:db8:3111:80:90d5::e801 k3os-cirrus-v2-server-ox0000 <none> <none>
calico-system pod/calico-typha-679cd7855c-qr8p9 1/1 Running 0 133m 2001:db8:3111:80:5:0:400:0 k3os-cirrus-v2-server-ox0000 <none> <none>
calico-system pod/calico-node-zl78s 1/1 Running 0 133m 2001:db8:3111:80:5:0:400:0 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/traefik-758cd5fc85-4p5kz 1/1 Running 0 72m 2001:db8:3111:80:90d5::e804 k3os-cirrus-v2-server-ox0000 <none> <none>
calico-system pod/calico-kube-controllers-5687f44fd5-qxp76 1/1 Running 0 133m 2001:db8:3111:80:90d5::e802 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/svclb-traefik-g4tfl 0/2 CrashLoopBackOff 38 72m 2001:db8:3111:80:90d5::e805 k3os-cirrus-v2-server-ox0000 <none> <none>
I can run IPv6 only pods !
# kubectl run -i -t busybox --image=busybox --restart=Never
If you don't see a command prompt, try pressing enter.
/ # ifconfig
eth0 Link encap:Ethernet HWaddr D6:B7:8D:CC:1A:82
inet6 addr: fe80::d4b7:8dff:fecc:1a82/64 Scope:Link
inet6 addr: 2001:db8:3111:80:90d5::e808/128 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1410 Metric:1
RX packets:7 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:746 (746.0 B) TX bytes:752 (752.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # ip -6 route show
2001:db8:3111:80:90d5::e808 dev eth0 metric 256
fe80::/64 dev eth0 metric 256
default via fe80::ecee:eeff:feee:eeee dev eth0 metric 1024
ff00::/8 dev eth0 metric 256
/ # ip route show
/ # ping6 -c 5 2001:db8:3111:80:90d5::e800
PING 2001:db8:3111:80:90d5::e800 (2001:db8:3111:80:90d5::e800): 56 data bytes
64 bytes from 2001:db8:3111:80:90d5::e800: seq=0 ttl=63 time=0.298 ms
64 bytes from 2001:db8:3111:80:90d5::e800: seq=1 ttl=63 time=0.333 ms
64 bytes from 2001:db8:3111:80:90d5::e800: seq=2 ttl=63 time=0.090 ms
64 bytes from 2001:db8:3111:80:90d5::e800: seq=3 ttl=63 time=0.224 ms
64 bytes from 2001:db8:3111:80:90d5::e800: seq=4 ttl=63 time=0.244 ms
--- 2001:db8:3111:80:90d5::e800 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.090/0.237/0.333 ms
/ # ping6 -c 5 2001:db8:3111:80:5:0:400:0
PING 2001:db8:3111:80:5:0:400:0 (2001:db8:3111:80:5:0:400:0): 56 data bytes
64 bytes from 2001:db8:3111:80:5:0:400:0: seq=0 ttl=64 time=0.754 ms
64 bytes from 2001:db8:3111:80:5:0:400:0: seq=1 ttl=64 time=0.153 ms
64 bytes from 2001:db8:3111:80:5:0:400:0: seq=2 ttl=64 time=0.277 ms
64 bytes from 2001:db8:3111:80:5:0:400:0: seq=3 ttl=64 time=0.269 ms
64 bytes from 2001:db8:3111:80:5:0:400:0: seq=4 ttl=64 time=0.372 ms
--- 2001:db8:3111:80:5:0:400:0 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.153/0.365/0.754 ms
/ # exit
... but loadbalancer service is in "pending" state as pod/svclb-traefik is looping "CrashLoopBackOff"
# kubectl get all -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default pod/busybox 0/1 Unknown 0 24h <none> k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/helm-install-traefik-svbx9 0/1 Completed 0 151m 2001:db8:3111:80:90d5::e812 k3os-cirrus-v2-server-ox0000 <none> <none>
tigera-operator pod/tigera-operator-6659cdcd96-5kgpl 1/1 Running 0 25d 2001:db8:3111:80:5:0:400:0 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/metrics-server-7566d596c8-stx49 1/1 Running 0 25d 2001:db8:3111:80:90d5::e81a k3os-cirrus-v2-server-ox0000 <none> <none>
calico-system pod/calico-typha-679cd7855c-qr8p9 1/1 Running 0 25d 2001:db8:3111:80:5:0:400:0 k3os-cirrus-v2-server-ox0000 <none> <none>
calico-system pod/calico-node-zl78s 1/1 Running 0 25d 2001:db8:3111:80:5:0:400:0 k3os-cirrus-v2-server-ox0000 <none> <none>
calico-system pod/calico-kube-controllers-5687f44fd5-qxp76 1/1 Running 1 25d 2001:db8:3111:80:90d5::e81d k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/local-path-provisioner-6d59f47c7-5fvxv 1/1 Running 1 25d 2001:db8:3111:80:90d5::e819 k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/traefik-758cd5fc85-4p5kz 1/1 Running 0 25d 2001:db8:3111:80:90d5::e81e k3os-cirrus-v2-server-ox0000 <none> <none>
k3os-system pod/system-upgrade-controller-fb57fbc56-qgdtr 1/1 Running 1 24h 2001:db8:3111:80:90d5::e81c k3os-cirrus-v2-server-ox0000 <none> <none>
kube-system pod/svclb-traefik-g4tfl 0/2 CrashLoopBackOff 50 25d 2001:db8:3111:80:90d5::e81b k3os-cirrus-v2-server-ox0000 <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 2001:db8:3111:80:5e45:1ce5:0:1 <none> 443/TCP 25d <none>
kube-system service/metrics-server ClusterIP 2001:db8:3111:80:5e45:1ce5:0:3de9 <none> 443/TCP 25d k8s-app=metrics-server
calico-system service/calico-typha ClusterIP 2001:db8:3111:80:5e45:1ce5:0:2c32 <none> 5473/TCP 25d k8s-app=calico-typha
kube-system service/traefik-prometheus ClusterIP 2001:db8:3111:80:5e45:1ce5:0:4ff0 <none> 9100/TCP 25d app=traefik,release=traefik
kube-system service/traefik LoadBalancer 2001:db8:3111:80:5e45:1ce5:0:42a8 <pending> 80:32269/TCP,443:30831/TCP 25d app=traefik,release=traefik
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
calico-system daemonset.apps/calico-node 1 1 1 1 1 <none> 25d calico-node docker.io/calico/node:v3.15.1 k8s-app=calico-node
kube-system daemonset.apps/svclb-traefik 1 1 0 1 0 <none> 25d lb-port-80,lb-port-443 rancher/klipper-lb:v0.1.2,rancher/klipper-lb:v0.1.2 app=svclb-traefik
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
tigera-operator deployment.apps/tigera-operator 1/1 1 1 25d tigera-operator quay.io/tigera/operator:v1.7.1 name=tigera-operator
kube-system deployment.apps/metrics-server 1/1 1 1 25d metrics-server rancher/metrics-server:v0.3.6 k8s-app=metrics-server
kube-system deployment.apps/local-path-provisioner 1/1 1 1 25d local-path-provisioner rancher/local-path-provisioner:v0.0.11 app=local-path-provisioner
kube-system deployment.apps/traefik 1/1 1 1 25d traefik rancher/library-traefik:1.7.19 app=traefik,release=traefik
calico-system deployment.apps/calico-typha 1/1 1 1 25d calico-typha docker.io/calico/typha:v3.15.1 k8s-app=calico-typha
calico-system deployment.apps/calico-kube-controllers 1/1 1 1 25d calico-kube-controllers docker.io/calico/kube-controllers:v3.15.1 k8s-app=calico-kube-controllers
k3os-system deployment.apps/system-upgrade-controller 1/1 1 1 24h system-upgrade-controller rancher/system-upgrade-controller:v0.5.0 upgrade.cattle.io/controller=system-upgrade-controller
NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
calico-system replicaset.apps/calico-typha-679cd7855c 1 1 1 25d calico-typha docker.io/calico/typha:v3.15.1 k8s-app=calico-typha,pod-template-hash=679cd7855c
tigera-operator replicaset.apps/tigera-operator-6659cdcd96 1 1 1 25d tigera-operator quay.io/tigera/operator:v1.7.1 name=tigera-operator,pod-template-hash=6659cdcd96
kube-system replicaset.apps/metrics-server-7566d596c8 1 1 1 25d metrics-server rancher/metrics-server:v0.3.6 k8s-app=metrics-server,pod-template-hash=7566d596c8
calico-system replicaset.apps/calico-kube-controllers-5687f44fd5 1 1 1 25d calico-kube-controllers docker.io/calico/kube-controllers:v3.15.1 k8s-app=calico-kube-controllers,pod-template-hash=5687f44fd5
kube-system replicaset.apps/local-path-provisioner-6d59f47c7 1 1 1 25d local-path-provisioner rancher/local-path-provisioner:v0.0.11 app=local-path-provisioner,pod-template-hash=6d59f47c7
kube-system replicaset.apps/traefik-758cd5fc85 1 1 1 25d traefik rancher/library-traefik:1.7.19 app=traefik,pod-template-hash=758cd5fc85,release=traefik
k3os-system replicaset.apps/system-upgrade-controller-fb57fbc56 1 1 1 24h system-upgrade-controller rancher/system-upgrade-controller:v0.5.0 pod-template-hash=fb57fbc56,upgrade.cattle.io/controller=system-upgrade-controller
NAMESPACE NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR
kube-system job.batch/helm-install-traefik 1/1 20s 151m helm rancher/klipper-helm:v0.2.5 controller-uid=802203c0-f359-4983-98fe-2948f048b1b6
It seems that svclb-traefik fails to set clean iptables rules for IPv6, if I believe the container logs
# kubectl logs -n kube-system --all-containers pod/svclb-traefik-g4tfl
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 2001:db8:3111:80:5e45:1ce5:0:42a8/32 -p TCP --dport 80 -j DNAT --to 2001:db8:3111:80:5e45:1ce5:0:42a8:80
iptables v1.6.2: Invalid port:port syntax - use dash
Try `iptables -h' or 'iptables --help' for more information.
+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 2001:db8:3111:80:5e45:1ce5:0:42a8/32 -p TCP --dport 443 -j DNAT --to 2001:db8:3111:80:5e45:1ce5:0:42a8:443
iptables v1.6.2: Invalid port:port syntax - use dash
Try `iptables -h' or 'iptables --help' for more information
2001:db8:3111:80:5e45:1ce5:0:42a8:80 and 2001:db8:3111:80:5e45:1ce5:0:42a8:443 are not correct !! Still the same misunderstanding regarding the syntax of IPv6 address when specifying the port in the IPv6 address. Address should be bracket enclosed [2001:db8:3111:80:5e45:1ce5:0:42a8]:443 !! or here maybe --dport explicitly specified; and IPv4 prefix length of "/32" in "-s 2001:db8:3111:80:5e45:1ce5:0:42a8/32" is stangely short for an IPv6 address ?? Shouldn't it be set to "/128" or "/64" ??
Rancher help would be welcome to adjust that iptables directive, as I don't know where it's declared, maybe buried in the rancher/klipper-lb image ? (seems to be in "entry" script in rancher/klipper-lb dockerfile, if that the case : useful URL https://helloacm.com/how-to-valid-ipv6-addresses-using-bash-and-regex/ to bash test IP address type to patch klipper-lb entry script or alternatively add alpine sipcalc package to test IP address type and adjust IPtables directive ?? )
j-landru
on 13 Aug 2020
@j-landru the load balancer is based on code taken from kube-router, which is guaranteed not to function correctly with IPv6 currently. Right now, k3s will not function in an IPv6-only cluster, it requires code changes - there are numerous locations where IPv4 addresses/schemes are hard-coded. For that matter, Kubernetes upstream has outstanding bugs for IPv6, and in particular IPv6-only.
This issue will surely get updated when the current situation changes, but unless you're comfortable contributing code changes all the way up the stack, and can work out what they should be by reading the existing code, I would recommend putting off the goal of IPv6-only Kubernetes via k3s for another day.
My plan (if I can allocate some time to it) is to finish getting dual-stack working as a first step, which should close the gap towards allowing IPv6-only (once upstream irons out their bugs).
pdf
on 13 Aug 2020
Related issues
Moep90
路
3Comments
dduportal
路
4Comments
jgreat
路
3Comments
giezi
路
3Comments
wpwoodjr
路
3Comments
Most helpful comment
Opened #1405 to track dual stack