K3s: Encrypted Networking Support

[rmoriz]

k3s targets non/mixed-cloud deployments without having a private LAN, for this use case plain flannel seems to be insecure.

Given that e.g. even Armbian (an ARM SoC focussed distro) ships with WireGuard tooling installed by default, you may want to consider it as an an easy to use, secure, low-overhead, high performance (yes, all buzzwords are true ;) VPN solution. It can also deal with NATs/CGN.

Flannel should be able to use WireGuard somehow.

Goals:

• encrypt traffic
• no public access to port 4789 (WireGuard's public exposure is very limited)

[akoenig]

wireguard support which ships with k3s would be super super helpful. I love the simplicity of k3s and how easy it is to setup a cluster. Having unencrypted traffic between the components of the cluster is actually a no-go for production-kind of setups though. Especially on cloud providers which don't support private networks (even then an encrypted communication would be required).

Do you have something like this on your roadmap? :slightly_smiling_face:

[fire]

Maybe this network plugin is easily usable? https://gravitational.com/blog/announcing_wormhole/

[digger18]

Try this:
https://github.com/squat/kilo

[cjellick]

Work for this underway. In addition to supporting WireGuard, we'll be supporting IPSec-based encrypted networks. We'll be packaging and deploying charon so it works out of the box.

[huapox]

Try this:
https://github.com/squat/kilo

I'm using this, ref:

• squat/kilo#13
• squat/kilo#12

[erikwilson]

Encryption support has been added to v0.10.0-alpha1.

There is an experimental server --flannel-backend flag to control what type of encryption is used. It currently defaults to vxlan (no encryption), but we are planning to default it to ipsec in a future release.

There are two reasons we are planning to default to ipsec instead of wireguard:

• ipsec only needs the charon user space process (provided with k3s), where wireguard needs kernel modules installed or the use of user-space wireguard implementations which may be buggy or have poor performance
• ipsec is faster than wireguard kernel modules from initial testing. This contradicts claims of the wireguard community so probably needs extra investigation, my initial results are here: https://docs.google.com/spreadsheets/d/1CuaLtmGmSEzW1OUaedZJfiPnoPFxCoXFQVTT2X5cAPs/edit?usp=sharing. The performance results of wireguard & UDP seem unrealistic compared to the baseline results and probably needs extra attention.

[huapox]

That's great feature! We can see v0.10.0-alpha1 with size 54.6 MB, which more than 10Mb than v0.9.1(43.5M), I'll try this latter.
Can ipsec be work in nat? (I'm not similar with it)

• Wireguard sure needs the kernel support, I've used wg in debian9, centos7, barge-os that works.

• Sure wireguard use udp for nat hole for connection which may cause unstable package trans.

1.In my env when my nodes both with a public ip (cloud vm with wan eip, but lan eth0) that works fine.
2.But when my node just in nat without wan-ip routes the wireguard's udp port and set the wan ip in kilo, then packages in connection lost or long-time return in period.

[erikwilson]

Yah the size of k3s is going up slowly but surely. charon adds about 2.7M uncompressed and swanctl (which we can probably drop) about 864K. Upgrading buildroot to 2019.2 LTS release and k8s update to 1.16 also helped to increase the binary size.

I am guessing ipsec should work as well as vxlan over NAT, but any testing that could be provided by the community is greatly appreciated.

If wireguard is broken for this use case I feel like we should just remove it as an option. I am also somewhat concerned that wireguard does not require a PSK for setting up the network, that indicates to me it is configuring the network in an insecure way. edit: looks like flannel is storing and sharing certs through the kubernetes node resource, so psk not needed

[davidnuzik]

v0.10.0-rc2

WireGuard and IPSEC support has been validated. With the upcoming v0.10.0 release encryption will be off by default as Erik has mentioned here: https://github.com/rancher/k3s/issues/50#issuecomment-539134116

To enable encryption with WireGuard or IPSEC pass the --flannel-backend flag on the server such as --flannel-backend=ipsec or --flannel-backend=wireguard for example. Properly installing WireGuard on each node is recommended so you have the necessary kernel modules if you wish to use wg. IPSEC does not require any kernel modules.

We tested primarily on Ubuntu 18.04. For WireGuard we installed via the recommended method https://www.wireguard.com/install/ using the wg PPA. In the future we plan to improve our test coverage by testing in other flavors of linux and other environments.

[huapox]

The result with my test to v0.10.0 with docker-compose:

I'm testing in debian9 with wireguard installed. that run with kilo over wireguard is ok.

1.1: --flannel-backend=ipsec, master and node in same Lan boots up ok.

server_1    | 15[IKE] remote host is behind NAT
server_1    | 15[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
server_1    | 15[NET] sending packet: from 172.22.0.2[500] to 172.22.0.4[500] (728 bytes)
server_1    | 08[NET] received packet: from 172.22.0.4[4500] to 172.22.0.2[4500] (256 bytes)
server_1    | 08[ENC] parsed IKE_AUTH request 1 [ IDi AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
server_1    | 08[CFG] looking for peer configs matching 172.22.0.2[%any]...172.22.0.4[172.22.0.4]
server_1    | 08[CFG] selected peer config '172.22.0.2-7.0.1.0/24-7.0.0.0/24-172.22.0.4'
server_1    | 08[IKE] authentication of '172.22.0.4' with pre-shared key successful
server_1    | 08[IKE] peer supports MOBIKE
server_1    | 08[CFG] no IDr configured, fall back on IP address
server_1    | 08[IKE] authentication of '172.22.0.2' (myself) with pre-shared key
server_1    | 08[IKE] IKE_SA 172.22.0.2-7.0.1.0/24-7.0.0.0/24-172.22.0.4[2] established between 172.22.0.2[172.22.0.2]...172.22.0.4[172.22.0.4]
server_1    | 08[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
server_1    | 08[IKE] CHILD_SA 7.0.1.0/24-7.0.0.0/24{1} established with SPIs cf633ec7_i c999f08c_o and TS 7.0.1.0/24 === 7.0.0.0/24
node_1      | 06[IKE] authentication of '172.22.0.4' (myself) with pre-shared key
node_1      | 06[IKE] establishing CHILD_SA 7.0.0.0/24-7.0.1.0/24{1}
node_1      | 06[ENC] generating IKE_AUTH request 1 [ IDi AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
node_1      | 06[NET] sending packet: from 172.22.0.4[4500] to 172.22.0.2[4500] (256 bytes)
node_1      | 07[NET] received packet: from 172.22.0.2[4500] to 172.22.0.4[4500] (224 bytes)
node_1      | 07[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) ]
node_1      | 07[IKE] authentication of '172.22.0.2' with pre-shared key successful
node_1      | 07[IKE] IKE_SA 172.22.0.4-7.0.0.0/24-7.0.1.0/24-172.22.0.2[2] established between 172.22.0.4[172.22.0.4]...172.22.0.2[172.22.0.2]
node_1      | 07[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
node_1      | 07[IKE] CHILD_SA 7.0.0.0/24-7.0.1.0/24{1} established with SPIs c999f08c_i cf633ec7_o and TS 7.0.0.0/24 === 7.0.1.0/24

[root@(⎈ |default:default) ~]$ kc get no -o  wide
NAME     STATUS   ROLES    AGE   VERSION         INTERNAL-IP   EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION   CONTAINER-RUNTIME
node1    Ready    <none>   18h   v1.16.2-k3s.1   172.22.0.4    <none>        Unknown    4.9.0-4-amd64    containerd://1.3.0-k3s.2
server   Ready    master   18h   v1.16.2-k3s.1   172.22.0.2    <none>        Unknown    4.9.0-4-amd64    containerd://1.3.0-k3s.2

1.2: --flannel-backend=ipsec,When node in diff Lan, The LB-IP of master's can not be reached.

root @ debian in /opt/dcp-k3s/v0-10/t_node |09:57:20  
$ dcp -f node2.yml up
Recreating t_node_node_1 ... done
Attaching to t_node_node_1
node_1  | time="2019-10-27T01:57:24.915012409Z" level=info msg="Starting k3s agent v0.10.0 (f9888ca3)"
node_1  | time="2019-10-27T01:57:24.916715615Z" level=info msg="Running load balancer 127.0.0.1:33369 -> [6.6.6.201:6443]"
node_1  | time="2019-10-27T01:57:25.264112941Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
node_1  | time="2019-10-27T01:57:25.264604596Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
node_1  | time="2019-10-27T01:57:25.265333548Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
node_1  | time="2019-10-27T01:57:26.270894058Z" level=info msg="module br_netfilter was already loaded"
node_1  | time="2019-10-27T01:57:26.271032326Z" level=warning msg="failed to write value 1 at /proc/sys/net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
node_1  | time="2019-10-27T01:57:26.296940758Z" level=info msg="Updating load balancer server addresses -> [172.22.0.2:6443 6.6.6.201:6443]"
node_1  | time="2019-10-27T01:57:26.298428602Z" level=info msg="Connecting to proxy" url="wss://172.22.0.2:6443/v1-k3s/connect"
node_1  | time="2019-10-27T01:59:35.814752257Z" level=error msg="Failed to connect to proxy" error="dial tcp 172.22.0.2:6443: connect: connection timed out"

Can I infer that IPSEC just resolv the encrytion in Lan/Nat (Which master can be reached by node's network), but not the connection for node to master when the node startup(When master in Nat env). And I didn't see new tunnel-device in master/node: no flannel.x and no strongswan's tunnel, if something wrong with my env?

root @ debian in /opt/dcp-k3s/v0-10 |10:15:23  
$ dcp exec server sh
/ # ip a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1415 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:05:6a:c5:6b:eb brd ff:ff:ff:ff:ff:ff
    inet 7.0.1.1/24 scope global cni0
       valid_lft forever preferred_lft forever
3: veth5067deaf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1415 qdisc noqueue master cni0 state UP group default 
    link/ether 26:76:97:66:57:b6 brd ff:ff:ff:ff:ff:ff link-netns cni-31e6c753-d276-13dd-6920-91d22113e7c1
4: veth951b7871@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1415 qdisc noqueue master cni0 state UP group default 
    link/ether 82:78:eb:c7:f1:a7 brd ff:ff:ff:ff:ff:ff link-netns cni-3fa4ff39-8c64-2ce4-506a-257da77f922b
70: eth0@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:16:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.22.0.2/16 brd 172.22.255.255 scope global eth0
       valid_lft forever preferred_lft forever

2. --flannel-backend=wireguard, need wg to be executed, but it not in the rancher/k3s:v0.10.0-amd64 container's image. that say we need to run k3s binary in the host with wireguard's installation. we can't use it in DinD mode in the current image.

server_1    | time="2019-10-24T10:50:54.439686673Z" level=fatal msg="flannel exited: failed to run command: wg genkey | tee privatekey | wg pubkey Err: exit status 127 Output: sh: wg: not found\nsh: wg: not found"

node_1      | time="2019-10-24T10:50:54.619537523Z" level=fatal msg="flannel exited: failed to run command: wg genkey | tee privatekey | wg pubkey Err: exit status 127 Output: sh: wg: not found\nsh: wg: not found"

Suggestion for wireguard usage in DinD:

1. Make wg into the rancher/k3s image.

2. Can we just compile wg into k3s binary? when I use kilo it runs just without the wg binary.

[vitobotta]

Hi! Is ipsec still the better choice for performance? Thanks

[vitobotta]

So I installed a cluster with ipsec and specifying the private network interface for Flannel as well. How can I check that the traffic is going through the private network and is encrypted? Thanks a lot in advance 🙂

[rmoriz]

Is WireGuard support still not released?

#/usr/local/bin/k3s --version
k3s version v1.0.0 (18bd921c)

# /usr/local/bin/k3s server --help | grep flannel
   --flannel-backend value                    (networking) One of 'none', 'vxlan', 'ipsec', or 'flannel' (default: "vxlan")
   --flannel-iface value                      (agent/networking) Override default flannel interface
   --flannel-conf value                       (agent/networking) Override default flannel config file
   --no-flannel                               (deprecated) use --flannel-backend=none

[erikwilson]

Wireguard is available, there is a ~type~ typo there. Instead of flannel it should say wireguard. :)

[alexellis]

Has anyone been able to get either IPSec or Wireguard to work through NAT traversal, i.e. a private arm SoC behind an ISP router “at home” and a “server” with a public IP on a cloud host.

[adi90x]

Has anyone been able to get either IPSec or Wireguard to work through NAT traversal, i.e. a private arm SoC behind an ISP router “at home” and a “server” with a public IP on a cloud host.

I have the exact same setup, did you find a working solution ?

[alexppg]

I do have it with that setup, it works out of the box if the master is in a remote server.
Also, It could work WHEN the master is at home if you do have a static IP, thought.

[adi90x]

I do have it with that setup, it works out of the box if the master is in a remote server.
Also, It could work WHEN the master is at home if you do have a static IP, thought.

Hello,
So just using k3s with the wireguard flag make it work ? All node are able to talk to each other ? Even those behind NAT at home ?

[rmoriz]

@adi90x it should work. Starting with v1.17.0 there is a persistent-keepalive every 25s to keep the tunnel open https://github.com/rancher/k3s/pull/1223/files

[alexppg]

Yes @adi90x. I only tried with ipsec, though.

[adi90x]

Haven't been able to make it work as expected... Will retry this week-end !

Le ven. 14 févr. 2020 à 19:16, Àlex Pérez [e-mail] a
écrit :

Yes @adi90x https://github.com/adi90x. I only tried with ipsec, though.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/k3s/issues/50?email_source=notifications&email_token=ACVGU5CUCPIR2STR3G4K7RLRC3NW7A5CNFSM4G2HDGB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELZ53PY#issuecomment-586407359,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACVGU5COROAJEWWU43G42ODRC3NW7ANCNFSM4G2HDGBQ
.

[davidnuzik]

WireGuard should work fine as long as it's set up on each node before K3s install. For example you probably need the kernel modules. Reference https://www.wireguard.com/install/

[ambis]

Does the wireguard option just use the preconfigured wg0 adapter? Since you can have multiple wg adapters (wg0, wg1 etc).

So I'm assuming, that if my nodes can see each other through wg0 interface and communicate through it, --flannel-backend=wireguard should just work out of the box?

Edit: Also, lets say that I had wg1 and I wanted to use that, would that be in any way possible?

[kamikaze]

just specifying --flannel-iface=wg0 works w/o any backend specification... so what does that --flannel-backend=wireguard then? Very confusing documentation

[alexppg]

I'll correct myself, the ipsec backend does not support NAT. I thought it work in my case, but id didn't. Wireguard, though, it does. Without anything special, only having the kernel modules.
It may require restarting the boxes, in my case there were a mix of problems with the routes in both workers and master.

[alexellis]

You can usually establish the network and schedule pods, but I found that inter-pod communication did not work, especially behind NAT.

I.e. an API that talks to a DB with both running on separate nodes.

[bduisenov]

@alexellis I'm experiencing the same issue, were you able to find a solution to that?

[unixfox]

@bduisenov you may try kilo as an alternative: https://github.com/squat/kilo

[fire]

I think cillium also has an encrypted networking plugin. Not sure.

https://cilium.io/

If anyone wants to write a guide for k3s, it'll be appreciated.

[alexppg]

@bduisenov the wireguard backend on flannel works flawlessly. I've been running it for a couple of months and it's awesome.
To make it work you only have to install wireguard and it will install the required kernel's modules. Once this is done, just start k3s with flannel and the backend as wireguard.
https://rancher.com/docs/k3s/latest/en/installation/network-options/#flannel-options

[matti]

What about when the public machine peer is reported as the internal ip, see https://github.com/k3s-io/k3s/issues/2689 - where can I override the wireguard peer ip?

[unixfox]

@matti use this annotation on the node: flannel.alpha.coreos.com/public-ip-overwrite=yourip

[matti]

@unixfox thanks, just found this also: https://github.com/alekc-go/flannel-fixer