K3s: Over Internet Setup

[adi90x]

Hello,

I have been trying to setup a working cluster using k3s for the last couple of days.
Right now, all system pod are Running, all services have endpoint, coredns is working.

However, inter host is not, thus pods on host not running coredns are unable to do nslookup google.fr

At the moment, I am trying to run the cluster over the internet : master node is in the cloud, some worker are in other clouds and some are at home ( behind a NAT ) .
All those node are in Ready state and able to run pods, so issue seems more likely to be on flannel setup I guess...
Is there anything I can do to investigate a bit more and make it work ( already been through k8s network debug guide without success ) ?

Regards,

[tbruun]

Hi.
Having the same issue - did you manage to find a solution for this ?

[bugfest]

Hi @adi90x, @tbruun - I've managed to get it working by patching the agent so that it uses the Public IP (behind NAT) of the server: https://github.com/bugfest/k3s/commit/278a972ea7fc785ef6bf8674db94cf4d967a7f66 - It's working for v0.9.1

[johansmitsnl]

I have a similar setup, server in the cloud on public IPv6 and IPv4 address.
Node at home as a worker.

It says that it is online and ready from the get nodes. I even have a pod running on the node and I can shell into it but the traefik can't access the pod over the network. How can I verify if the flannel network is running ok?
I cannot forward any ports on the home network because there are multiple workers on the home network and I have only one ipv4 address on the router.

[johansmitsnl]

I can ping both sides now but when lb want to open the location it fails.

Here is a tcpdump:

Master

IP (tos 0x0, ttl 252, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    10.42.1.15 > 10.42.0.5: ICMP net 10.42.1.15 unreachable, length 36
    IP (tos 0x0, ttl 60, id 51021, offset 0, flags [DF], proto TCP (6), length 60)
    10.42.0.5.36668 > 10.42.1.15.http: [|tcp]

Worker

21:58:02.237814 IP 10.42.0.5.36668 > 10.42.1.15.http: Flags [S], seq 3839720869, win 28200, options [mss 1410,sackOK,TS val 4192680549 ecr 0,nop,wscale 7], length 0
21:58:02.242987 IP 10.42.1.15 > 10.42.0.5: ICMP net 10.42.1.15 unreachable, length 36

Also when I'm in a pod on the worker trying to do internet things I get this timeouts.

$ wget -T 1 https://google.com
Connecting to google.com (216.58.208.110:443)
wget: download timed out

so it looks like a net/iptable issue? like #116