Flannel: ipsec backend not working

[mwmix]

Expected Behavior

When using the ipsec backend the flannel.1 interface should eventually be created.

Current Behavior

I start flannel and after what looks like a good startup sequence I check ifconfig and find no interface. However, I can see ip xfrm policy loaded with all the correct information.

I can see no apparent errors coming from flannel. I can see that the psk authentication is successful.

Possible Solution

No idea sorry :(

Steps to Reproduce (for bugs)

1. Download and compile latest version of master as of 3/20/2018
2. Add following to /coreos.com/network/config in etcd:
   {
   "Network": "172.16.0.0/16",
   "Backend": {
   "Type": "ipsec",
   "PSK": "a-valid-psk-using-the-docs-recommended-dd-method"
   }
   }
   or even:
   {
   "Network": "172.16.0.0/16",
   "Backend": {
   "Type": "ipsec",
   "PSK": "a-valid-psk-using-the-docs-recommended-dd-method",
   "UDPEncap": true
   }
   }
3. 
   Execute "flanneld --etcd-endpoints --etcd-cafile --etcd-certfile --etcd-keyfile --etcd-prefix=/coreos.com/network
   or
   Execute "flanneld --etcd-endpoints --etcd-cafile --etcd-certfile --etcd-keyfile --etcd-prefix=/coreos.com/network --public-ip

Context

We are trying to setup a cross region encrypted overlay network across an openstack environment.

I have been able to get a vxlan working without issue

Your Environment

We are running in an openstack environment.

• Flannel version: latest copy of master
• Backend used (e.g. vxlan or udp): ipsec
• Etcd version: 3.1.6
• Kubernetes version (if used): n/a
• Operating System and version: centos 7 kernel 3.10.0-693.21.1.el7.x86_64
• Link to your project (optional): n/a

[mwmix]

So I got this to work. Basically, I had to do the following:

for each server in overlay network add their overlay ip to eth0:
servera 172.16.1.0
serverb 172.16.2.0

on servera:
ip addr add 172.16.1.0 dev eth0

on serverb:
ip addr add 172.16.2.0 dev eth0

I was then able to do:
on server a:
ping -I 172.16.1.0 172.16.2.0 and see a proper echo reply.

Is there any reason why flannel, strongswan or charon are not already doing this for us?

Also I think it would be a good idea to mention that in order for "ip xfrm state" to work you need to edit:
/etc/strongswan/strongswan.d/charon/kernel-libipsec.conf changing "load=yes" to "load=no". This apparently then lets kernel-netlink handle things at the kernel level as opposed to userland.

[mwmix]

I also found a use case that apparently flannel can't handle right now at least regarding ipsec. In openstack servers are assigned ips that aren't routable across regions or projects. You assign a server a floating ip to get around this which is a publicly accessible ip. However, when you do this that server you assigned the floating ip to has no idea about that floating ip. It just knows about it's private ip address and openstack handles the natting of the traffic. So when you want to do cross region tunnels you run into an issue. The ipsec backend right now generates a policy that looks for a src address of either what is assigned to eth0 or what you specify with --public-ip . So if we tell flannel to use the floating ip via the --public-ip option then the ipsec backend generates policies that look for a src that matches the floating ip. This doesn't work in our openstack scenario since the local server has no concept of that floating ip and so no traffic will ever be sent out with a src ip == the server's floating ip. What we need in this case is a --private-ip option where we can have the ipsec backend use the private ip for the src from the server and then use the --public-ip of the other servers for the given destinations.

Hopefully the above makes sense. I am more than willing to elaborate on this issue. And I have confirmed strongswan with charon will handle this scenario if configured properly.

[mwmix]

One final useful comment of note is perhaps considering adding a link in the troubleshooting the ipsec backend documentation to point users at the strongswan HelpRequests wiki page. Most notable it details a really good logging configuration that helps greatly in diagnosing most issues. https://wiki.strongswan.org/projects/strongswan/wiki/HelpRequests

[bchannak]

@mwmix I ran into a issue similar to your and filed this issue: https://github.com/coreos/flannel/issues/1027

Is flannel.1 interface necessary when using IPSec backend?

[ncabatoff]

I've run into this issue too. I too don't have a flannel.1 interface, but having studied the code and compared it to the vxlan backend code, it's clear that's deliberate. So I'm treating this issue as a generic "flannel ipsec doesn't work out of the box with Kubernetes" bug report, as per its title, which matches my experience.

In my case, everything comes up just fine and reports as healthy, but I can't connect to any services I deploy. Pod-to-pod networking works just fine, but node-to-pod networking does not.

I didn't find adding overlay IPs (as proposed above) to help. What worked for me was to add a route on masters:

route add -net 10.244.0.0 netmask 255.255.0.0 cni0

The resulting route table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.0.2     0.0.0.0         UG    0      0        0 enp0s3
10.0.0.0        *               255.255.255.0   U     0      0        0 enp0s8
10.244.0.0      *               255.255.255.0   U     0      0        0 cni0
10.244.0.0      *               255.255.0.0     U     0      0        0 cni0
172.17.0.0      *               255.255.0.0     U     0      0        0 docker0
192.168.0.0     *               255.255.0.0     U     0      0        0 enp0s3

Setup:

• created a cluster via kubeadm init with the --pod-network-cidr=10.244.0.0/16 argument as per kubernetes docs for flannel
• modified the kube-flannel.yaml manifest:
  
  
  
  • added --iface=enp0s8 to the kube-flannel container arguments
  
  
  • changed image to a locally built image since the quay.io image doesn't seem to have ipsec included
  
  
  • changed the kube-flannel-cfg configmap (PSK generated using dd as descibed in backends):
  
  

  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "ipsec",
        "PSK": "ee9b0c24c5d015f4ed23e84fb1473314a8baa8bee65fcefd659981d59423c6ee718eee681f579cccf18fd5d7f759f736"
      }
    }

[pms1969]

I've run into this problem just now. The route fix specified by @ncabatoff did the trick, but no matter what I did with swanctl to reload setting to with the charon.conf: install_routes=yes line uncommented, didn't seem to work. Am I missing something here?

Either way, how do I make sure that route is added? an init container?

[ghouscht]

Same issue here... Flannel with vxlan backend is working as expected but as soon as we switch to ipsec, host -> pod communication is broken. When using ipsec backend no flannel interface nor the corresponding routes are created on the host.

Is there any work being done to resolve this issue?

[alvaroaleman]

Either way, how do I make sure that route is added? an init container?

We use a startup script DS for this:

+# Required for node->pod traffic to work
+# See https://github.com/coreos/flannel/issues/966#issuecomment-425743884
+kind: DaemonSet
+apiVersion: extensions/v1beta1
+metadata:
+  name: add-missing-flannel-route
+  namespace: kube-system
+  labels:
+    type: startup-script
+    app: add-missing-flannel-route
+spec:
+  selector:
+    matchLabels:
+      app: add-missing-flannel-route
+  template:
+    metadata:
+      labels:
+        app: add-missing-flannel-route
+    spec:
+      hostPID: true
+      tolerations:
+      - operator: Exists
+      containers:
+        - name: startup-script
+          image: quay.io/kubermatic/startup-script:v0.1.0
+          imagePullPolicy: Always
+          securityContext:
+            privileged: true
+          env:
+          - name: STARTUP_SCRIPT
+            value: |
+              #!/bin/bash
+              set -o errexit
+              set -o pipefail
+              set -o nounset
+              if ! ip r s|grep 10.244.0.0/16 -q|grep cni0 -q; then
+                route add -net 10.244.0.0 netmask 255.255.0.0 cni0
+              fi

What I find a bit confusing is: Why is there a route for 10.244.0.0/24 via cni0? It looks like someone just forgot to put the right netmask on it?