Cilium: transparent encryption: setting nodEncryption=true results in nodes being unreachable

[kkourt]

Following the encryption guide with:

$ helm install cilium cilium/cilium --version 1.9.0-rc2 --namespace kube-system --set encryption.enabled=true --set encryption.nodeEncryption=true

results in nodes being unreachable.

[ti-mo]

:wave: @kkourt did you do some troubleshooting already? What does 'nodes unreachable' mean in this context? host->host traffic or node->node traffic?

@pchaigno Maybe we should add a release blocker label here? (I can't..)

[kkourt]

did you do some troubleshooting already?

Not really, just the basic things (logs, etc.) but didn't notice anything relevant.

What does 'nodes unreachable' mean in this context? host->host traffic or node->node traffic?

Not sure what the difference between host->host and node->node is...

Some output from my original report that might help clarify things:

kkourt@t1:~$ ip r
default via 10.172.0.1 dev ens4 proto dhcp src 10.172.15.198 metric 100 
10.0.0.0/24 via 10.0.0.26 dev cilium_host src 10.0.0.26 mtu 1333 
10.0.0.26 dev cilium_host scope link 
10.0.1.0/24 via 10.0.0.26 dev cilium_host src 10.0.0.26 mtu 1333 
10.172.0.1 dev ens4 proto dhcp scope link src 10.172.15.198 metric 100 
10.172.15.199 dev cilium_host proto eigrp mtu 1333 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
kkourt@t1:~$ ip route get 10.172.15.199
10.172.15.199 dev cilium_host src 10.172.15.198 uid 1001 
    cache mtu 1333 
kkourt@t1:~$ ping 10.172.15.199
PING 10.172.15.199 (10.172.15.199) 56(84) bytes of data.
^C
--- 10.172.15.199 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1036ms

[pchaigno]

@kkourt Could you check if you have the same issue with the latest 1.8?

[jrfastab]

@kkourt Also any hints on the environment would be helpful? Is this GKE, AWS, packetnet, custom env. I'll take a look, but I did a GKE setup like this not so long ago.

[kkourt]

Seems to also happen with latest 1.8. (to clarify, this is with vxlan.)

[kkourt]

@kkourt Also any hints on the environment would be helpful? Is this GKE,

This is GCE, i.e., google VMs.

[ti-mo]

Not sure what the difference between host->host and node->node is...

Woops, of course I meant host->host and pod->pod. :sweat_smile:

[joestringer]

Marked as release-blocker to evaluate whether this is a regression and at least figure out the potential scope for the breakage (Is the feature now completely broken? Or broken in certain circumstances? Or the documentation misses an important configuration item?).

[sduranc]

I've observed this behaviour also running OKD v4.5 on top of GCP. With encryption.enabled=true it was fine but after trying encryption.nodeEncryption=true I lost connection to the cluster. Trying a simple kubectl get pods showed one of the following:

1. Error from server: etcdserver: request timed out
2. The connection to the server api.xxx.yyy.zzz:6443 was refused - did you specify the right host or port?

[jrfastab]

Thanks for the reports. EncryptNode=true has not been enabled with vxlan use cases yet. Its only been enabled in the routing case. I'll submit a fix to the docs to make this clear, sorry for the confusion. Going forward we can mark this as a feature request. I believe it should be not-to-hard(tm) to get it working, but I've not had a chance to get to it and its not been top of list for me. If someone wants to pick it up I'm happy to do code-review, give some guidance on what needs to be done. Either reach out here or on slack. Thanks.

@sduranc In your example above was that with vxlan enabled?

[jrfastab]

@joestringer I'm tempted to just make it a note in the docs and let the agent try to set it up. iirc there are some scenarios where it does work and its been in this state for multiple releases so if anyone has a working setup we don't want to hit them with a 'this doesn't work' message.

[jrfastab]

Proposed document update,

--- a/Documentation/gettingstarted/encryption.rst
+++ b/Documentation/gettingstarted/encryption.rst
@@ -107,6 +107,11 @@ In order to enable node-to-node encryption, add:
     --set encryption.enabled=true \
     --set encryption.nodeEncryption=true

+.. note::
+
+    Node to node encryption feature is tested and supported with direct routing
+    modes. Using with tunnel modes is not currently tested or supported.
+

https://github.com/cilium/cilium/pull/13800

[sduranc]

Thanks for the reports. EncryptNode=true has not been enabled with vxlan use cases yet. Its only been enabled in the routing case. I'll submit a fix to the docs to make this clear, sorry for the confusion. Going forward we can mark this as a feature request. I believe it should be not-to-hard(tm) to get it working, but I've not had a chance to get to it and its not been top of list for me. If someone wants to pick it up I'm happy to do code-review, give some guidance on what needs to be done. Either reach out here or on slack. Thanks.

@sduranc In your example above was that with vxlan enabled?

Yes @jrfastab my example was with vxlan

[joestringer]

The documentation for this has been updated to be more clear about what's supported for now. We would welcome further investigation into enabling this use case if someone is interested in working on it, but we will not block the v1.9.0 release upon this feature.

[acholt]

I've been trying to get nodeEncryption working with Native-Routing by following the guides at the bottom.

The nodes became unavailable, same as @kkourt described. I followed the node logs as I activated nodeEncryption, the last few messages before the node become unavailable was:

cni.go:202] Error validating CNI config list {"cniVersion":"0.3.1","name":"cilium","plugins":[{"cniVersion":"0.3.1","enable-debug":false,"name":"cilium","type":"cilium-cni"}]}: [failed to find plugin "cilium-cni" in path [/opt/cni/bin]]

Cluster details:

• Amazon EKS v1.18
• Cilium v1.9.0
• Kernel v5.4.74-36.135.amzn2.x86_64

Helm value for Cilium Version 1.9.0:

$ helm get values cilium -n kube-system
USER-SUPPLIED VALUES:
egressMasqueradeInterfaces: eth+
encryption:
  enabled: true
  nodeEncryption: true
endpointRoutes:
  enabled: true
eni: true
hubble:
  enabled: true
  listenAddress: :4244
  metrics:
    enabled:
    - dns
    - drop
    - tcp
    - flow
    - port-distribution
    - icmp
    - http
  relay:
    enabled: true
  ui:
    enabled: true
ipam:
  mode: eni
nodeinit:
  enabled: true
operator:
  prometheus:
    enabled: true
  replicas: 3
prometheus:
  enabled: true
tunnel: disabled

Cilium Status:

$ ./k8s-cilium-exec.sh cilium status
KVStore:                                      Ok   Disabled
Kubernetes:                                   Ok   1.18+ (v1.18.8-eks-7c9bda) [linux/amd64]
Kubernetes APIs:                              ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:                         Probe
Cilium:                                       Ok      OK
NodeMonitor:                                  Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:                         Ok
IPAM:                                         IPv4: 6/14 allocated,
BandwidthManager:                             Disabled
Host Routing:                                 Legacy
Masquerading:                                 IPTables
Controller Status:                            37/37 healthy
Proxy Status:                                 OK, ip 10.16.4.123, 0 redirects active on ports 10000-20000
Hubble:                                       Ok              Current/Max Flows: 4096/4096 (100.00%), Flows/s: 10.27   Metrics: Ok
Cluster health:                               1/3 reachable   (2020-11-19T16:56:56Z)
  Name                                        IP              Node          Endpoints
  ip-10-16-3-86.eu-west-2.compute.internal    10.16.3.86      reachable     unreachable
  ip-10-16-5-176.eu-west-2.compute.internal   10.16.5.176     unreachable   unreachable

[1] https://docs.cilium.io/en/v1.9/concepts/networking/routing/#configuration
[2] https://docs.cilium.io/en/v1.9/concepts/networking/ipam/eni/#ipam-eni
[3] https://docs.cilium.io/en/v1.9/gettingstarted/encryption

[jrfastab]

@acholt there appears to be an EKS specific issues with node-encrypt at the moment. How many NIC interfaces exist on this setup? Couple ideas, perhaps the auto-discovery (the thing that picks the egress interface) is choosing the wrong interface. Another idea would be to use Cilium masquerading instead of iptables.

[acholt]

@jrfastab - Thanks for the response.

There are between 2 and 3 eth interfaces:

• I've tried forcing it on a specific interface but that didn't help
• Also tried disabled masquerading but that didn't help.

When you say Cilium masquerading are you referring to bpf.masquerade=true?

[jrfastab]

OK thanks. It might be some IPtables issue then, not sure.

Correct, bpf.masquerade=true is what I was thinking.

Looks like I might need to get an EKS instance to debug. I'll try to get to this early next week. Its a short week here for me.

[acholt]

@jrfastab I tried your second suggestion by enabling bpf.masquerade plus I also enabled ipMasqAgent.enabled.

Unfortunately to get bpf.masquerade to work you need to set encryption.enabled = false as "_IPSec cannot be used with BPF NodePort_" as the logs states and reverts it back to iptables.

level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=warning msg="IPSec cannot be used with BPF NodePort. Disabling BPF NodePort feature." subsys=daemon
level=warning msg="Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host." subsys=daemon
level=info msg="Restored services from maps" failed=0 restored=10 subsys=service
level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
level=info msg="Reading old endpoints..." subsys=daemon
level=info msg="Reusing previous DNS proxy port: 36125" subsys=daemon
level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
level=info msg="Retrieved node information from kubernetes node" nodeName=ip-10-16-5-189.eu-west-2.compute.internal subsys=k8s
level=info msg="Received own node information from API server" ipAddr.ipv4=10.16.5.189 ipAddr.ipv6="<nil>" k8sNodeIP=10.16.5.189 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:m5.xlarge beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:eu-west-2 failure-domain.beta.kubernetes.io/zone:eu-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-10-16-5-189.eu-west-2.compute.internal kubernetes.io/os:linux node.kubernetes.io/instance-type:m5.xlarge topology.kubernetes.io/region:eu-west-2 topology.kubernetes.io/zone:eu-west-2c workload:trusted]" nodeName=ip-10-16-5-189.eu-west-2.compute.internal subsys=k8s v4Prefix=10.189.0.0/16 v6Prefix="<nil>"
level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
level=info msg="Enabling k8s event listener" subsys=k8s-watcher
level=warning msg="BPF masquerade requires NodePort (--enable-node-port=\"true\"). Falling back to iptables-based masquerading." subsys=daemon
level=fatal msg="BPF ip-masq-agent requires --masquerade=\"true\" and --enable-bpf-masquerade=\"true\"" subsys=daemon

When you have the time, let me know what result you get from your testing.