Cilium: Deploying Cilium quick-install YAML in Hetzner cloud leads to connectivity outage

[joestringer]

Report from Alexander Becker on Slack:

I'm trying to use cilium with the hetzner native cloud networking.
The setup which is explained here is related to an old cilium version.
Using the old version the cilium deployments keep crashing/restarting, but the routes are created correctly in the hetzner cloud console.
Using the latest version with this guide and the native networking concept, my node goes unreachable. SSH still works, but I can't access kubectl anymore (using a loadbalancer which points to the internal IP of the node).

Symptoms include:

• Cilium doesn't become ready, even after 5 minutes:
  root@test-cluster-master-01:~# k --insecure-skip-tls-verify=true get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-2j42b 0/1 Running 0 5m2s ...
• Cilium readiness probes fail:
  root@test-cluster-master-01:~# k --insecure-skip-tls-verify=true describe pod -n kube-system cilium-tkvvl ... Normal Created 7m45s kubelet Created container cilium-agent Normal Started 7m45s kubelet Started container cilium-agent Warning Unhealthy 7m10s (x2 over 7m40s) kubelet Readiness probe failed: Get "http://127.0.0.1:9876/healthz": dial tcp 127.0.0.1:9876: connect: connection refused

[joestringer]

One issue we identified was that the quick-install YAML specifies the following cluster-pool IPAM CIDR range:

cluster-pool-ipv4-cidr: "10.0.0.0/8"

However, this commonly overlaps with the CIDR range used for internal IPs from cloud servers in Hetzner. So users would need to configure a distinct CIDR range to ensure that Cilium doesn't try to allocate IPs that conflict with server IPs.

The Hetzner documentation on networks specifies which address ranges can be used, ideally users should use a different range for the underlying network vs. the range used for k8s pod ip address management.

[joestringer]

Helpful debugging tip: If you lose access via kubectl, it may still be possible to SSH into a node and point kubectl at 127.0.0.1:6443 to run kubectl commands:

root@test-cluster-master-01:~# k --insecure-skip-tls-verify=true get pods -A

[joestringer]

I believe @mmack has had some success on Hetzner systems in the past, I wonder if you have a particular guide or set of steps that you follow to successfully deploy Cilium on Hetzner cloud?

[ByteAlex]

So I've been doing a few more tests on this issue, and it seems like I was just having 10.20.0.0/8 in the configmap, using 10.20.0.0/16 it does deploy w/o the cluster going unavailable.

But yet I have another issue, before I was able to run the connectivity tests I'm seeing this on coredns:

  Warning  FailedCreatePodSandBox  46m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c510ac0dbb9010de933f52920d7088f13cbd4b345256ee7041d2336943969334" network for pod "coredns-f9fd979d6-c2cm4": networkPlugin cni failed to set up pod "coredns-f9fd979d6-c2cm4_kube-system" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure  range is full
  Normal   SandboxChanged          11m (x1966 over 46m)   kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  106s (x2516 over 46m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fd8ddf72ef4889b9f1b4a8ff2376dd6ac9e8fd5f6506f83f133eacf208ec0e2c" network for pod "coredns-f9fd979d6-c2cm4": networkPlugin cni failed to set up pod "coredns-f9fd979d6-c2cm4_kube-system" network: unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure  range is full

The cilium ds logs look like this:
level=info msg="Delete endpoint request" id="container-id:06190dc8fa40ec58f8e910f66786d14ae42676a33611b03b2e84bc21361300e7" subsys=daemon

The operator logs are this:

level=info msg="Cilium Operator 1.8.5 fc1a8ac7c 2020-10-28T14:44:12-07:00 go version go1.14.10 linux/amd64" subsys=cilium-operator-generic
level=info msg="Starting apiserver on address 127.0.0.1:9234" subsys=cilium-operator-generic
level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="attempting to acquire leader lease  kube-system/cilium-operator-resource-lock..." subsys=klog
level=info msg="successfully acquired lease kube-system/cilium-operator-resource-lock" subsys=klog
level=info msg="Leading the operator HA deployment" subsys=cilium-operator-generic
level=info msg="Initializing IPAM" mode=cluster-pool subsys=cilium-operator-generic
level=info msg="Starting ClusterPool IP allocator" ipv4CIDRs="[10.20.0.0/16]" ipv6CIDRs="[]" subsys=ipam-allocator-clusterpool
level=info msg="Starting to synchronize CiliumNode custom resources..." subsys=cilium-operator-generic
level=info msg="Waiting for CRD ciliumnodes.cilium.io to be available" subsys=cilium-operator-generic
level=error msg=k8sError error="github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:82: Failed to list *v2.CiliumEndpoint: the server could not find the requested resource (get ciliumendpoints.cilium.io)" subsys=k8s
level=error msg=k8sError error="github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:82: Failed to list *v2.CiliumEndpoint: the server could not find the requested resource (get ciliumendpoints.cilium.io)" subsys=k8s
level=info msg="CRD ciliumnodes.cilium.io found" subsys=cilium-operator-generic
level=error msg=k8sError error="github.com/cilium/cilium/operator/cilium_node.go:87: Failed to list *v2.CiliumNode: the server could not find the requested resource (get ciliumnodes.cilium.io)" subsys=k8s
level=error msg=k8sError error="github.com/cilium/cilium/operator/watchers/cilium_endpoint.go:82: Failed to list *v2.CiliumEndpoint: the server could not find the requested resource (get ciliumendpoints.cilium.io)" subsys=k8s
level=info msg="Waiting for CRD ciliumidentities.cilium.io to be available" subsys=cilium-operator-generic
level=info msg="CRD ciliumidentities.cilium.io found" subsys=cilium-operator-generic
level=info msg="Starting CRD identity garbage collector with 15m0s interval..." subsys=cilium-operator-generic
level=info msg="Starting to garbage collect stale CiliumEndpoint custom resources..." subsys=cilium-operator-generic
level=info msg="Starting CNP derivative handler..." subsys=cilium-operator-generic
level=info msg="Waiting for CRD ciliumnetworkpolicies.cilium.io to be available" subsys=cilium-operator-generic
level=info msg="CRD ciliumnetworkpolicies.cilium.io found" subsys=cilium-operator-generic
level=info msg="Starting CCNP derivative handler..." subsys=cilium-operator-generic
level=info msg="Initialization complete" subsys=cilium-operator-generic

[joestringer]

@ByteAlex the postIpamFailure range is full is likely to be #9793, caused by having a route in the node which overlaps with the IPAM range. The full writeup is on that issue but I suspect that setting blacklist-conflicting-routes: false in the configmap should be sufficient to resolve this issue on v1.8 or earlier. v1.9 is not expected to have the same issue.

[ByteAlex]

After checking in https://github.com/hetznercloud/csi-driver/issues/160 I noticed that blacklist-conflicting-routes: false as you just suggested.

That works to get an IP assigned to the pod, but it still doesn't seem to have any networking.

kubectl logs

E1109 19:18:27.494404       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I1109 19:18:27.576905       1 trace.go:116] Trace[436340495]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 19:17:57.574271636 +0000 UTC m=+63.812283510) (total time: 30.002601288s):
Trace[436340495]: [30.002601288s] [30.002601288s] END
E1109 19:18:27.576939       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout

[joestringer]

@ByteAlex that looks like the kubernetes API server that cannot be reached. Which component is responsible for implementing kubernetes services in your cluster? The Cilium quick-install.yaml assumes that you have a standard k8s cluster with components like kube-proxy running to provide that functionality. If that is not the case, Cilium also has an implementation but you will need to configure it.

[mmack]

@joestringer I have a running cilium config for Hetzner Bare-Metal Servers. My initial plan was spanning a wireguard network over metal and cloud nodes but that turned out not to be supported atm. So no, i don't have a running hcloud config.

Which brings me to an idea, maybe cilium should host a extra repository with different configs for different hosting providers with a short explanation of features working / not working. @tgraf thoughts?

[ByteAlex]

@joestringer I've been using this

kubeadm init --ignore-preflight-errors=NumCPU --apiserver-cert-extra-sans $API_SERVER_CERT_EXTRA_SANS --control-plane-endpoint "$CONTROL_PLANE_LB" \
  --upload-certs --kubernetes-version=$KUBE_VERSION --pod-network-cidr=$POD_NETWORK_CIDR

plus hcloud CCM w/hcloud network

kubectl apply -f https://raw.githubusercontent.com/hetznercloud/hcloud-cloud-controller-manager/master/deploy/ccm-networks.yaml

After that I install cilium

Pods:

root@test-cluster-master-01:~# k get pods -A
NAMESPACE     NAME                                              READY   STATUS    RESTARTS   AGE
kube-system   cilium-8txw5                                      1/1     Running   0          82m
kube-system   cilium-d8n5r                                      1/1     Running   0          82m
kube-system   cilium-operator-648569fbb8-5m72n                  1/1     Running   0          81m
kube-system   cilium-operator-648569fbb8-ls84h                  1/1     Running   0          81m
kube-system   cilium-p8zq7                                      1/1     Running   0          82m
kube-system   coredns-f9fd979d6-c2cm4                           0/1     Running   0          138m
kube-system   coredns-f9fd979d6-qbx8x                           0/1     Running   0          138m
kube-system   etcd-test-cluster-master-01                       1/1     Running   0          149m
kube-system   hcloud-cloud-controller-manager-cb9c6698d-pbxnt   1/1     Running   0          148m
kube-system   kube-apiserver-test-cluster-master-01             1/1     Running   0          149m
kube-system   kube-controller-manager-test-cluster-master-01    1/1     Running   0          149m
kube-system   kube-proxy-cjqwb                                  1/1     Running   0          149m
kube-system   kube-proxy-kd6p5                                  1/1     Running   0          144m
kube-system   kube-proxy-vrgsr                                  1/1     Running   0          144m
kube-system   kube-scheduler-test-cluster-master-01             1/1     Running   0          149m

[joestringer]

@ByteAlex which pods have these connectivity issues? Did you restart them after Cilium was successfully running?

[ByteAlex]

@joestringer coredns is not getting ready, I restarted them once, let me try to restart them again

[ByteAlex]

Restarted, but same thing:

^Croot@test-cluster-master-01:~# k logs -n kube-system coredns-f9fd979d6-47sz4
[INFO] plugin/ready: Still waiting on: "kubernetes"
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I1109 20:42:36.192523       1 trace.go:116] Trace[911902081]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 20:42:06.191602802 +0000 UTC m=+0.030458951) (total time: 30.000769772s):
Trace[911902081]: [30.000769772s] [30.000769772s] END
E1109 20:42:36.192585       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I1109 20:42:36.192907       1 trace.go:116] Trace[140954425]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 20:42:06.1925064 +0000 UTC m=+0.031362540) (total time: 30.000368502s):
Trace[140954425]: [30.000368502s] [30.000368502s] END
E1109 20:42:36.192918       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I1109 20:42:36.193138       1 trace.go:116] Trace[336122540]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 20:42:06.192834183 +0000 UTC m=+0.031690322) (total time: 30.000278175s):
Trace[336122540]: [30.000278175s] [30.000278175s] END
E1109 20:42:36.193150       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"

Might be interesting:
kube-api-server has an external IP in -owide

[mmack]

My bet is that service ips are not working and therefore coredns‘s not able to reach the k8s api.

Von meinem iPhone gesendet

Am 09.11.2020 um 21:43 schrieb Alex [e-mail]:


Restarted, but same thing:

^Croot@test-cluster-master-01:~# k logs -n kube-system coredns-f9fd979d6-47sz4
[INFO] plugin/ready: Still waiting on: "kubernetes"
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I1109 20:42:36.192523 1 trace.go:116] Trace[911902081]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 20:42:06.191602802 +0000 UTC m=+0.030458951) (total time: 30.000769772s):
Trace[911902081]: [30.000769772s] [30.000769772s] END
E1109 20:42:36.192585 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I1109 20:42:36.192907 1 trace.go:116] Trace[140954425]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 20:42:06.1925064 +0000 UTC m=+0.031362540) (total time: 30.000368502s):
Trace[140954425]: [30.000368502s] [30.000368502s] END
E1109 20:42:36.192918 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I1109 20:42:36.193138 1 trace.go:116] Trace[336122540]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125 (started: 2020-11-09 20:42:06.192834183 +0000 UTC m=+0.031690322) (total time: 30.000278175s):
Trace[336122540]: [30.000278175s] [30.000278175s] END
E1109 20:42:36.193150 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[ByteAlex]

So coredns is trying to reach 10.96.0.1, I configured the following

  masquerade: "true"
  enable-bpf-masquerade: "true"
  enable-xt-socket-fallback: "true"
  install-iptables-rules: "true"
  auto-direct-node-routes: "false"
  kube-proxy-replacement:  "probe"
  enable-health-check-nodeport: "true"
  node-port-bind-protection: "true"
  enable-auto-protect-node-port-range: "true"
  enable-session-affinity: "true"
  enable-endpoint-health-checking: "true"
  enable-well-known-identities: "false"
  enable-remote-node-identity: "true"
  enable-api-rate-limit: "false"
  operator-api-serve-addr: "127.0.0.1:9234"
  ipam: "cluster-pool"
  cluster-pool-ipv4-cidr: "10.20.0.0/16"
  cluster-pool-ipv4-mask-size: "24"
  disable-cnp-status-updates: "true"

  blacklist-conflicting-routes: "false"
  enable-endpoint-routes: "true"
  native-routing-cidr: "10.10.0.0/16"

I wonder how it pulled the 10.96.0.1

[ByteAlex]

@mmack Would you mind sharing your kubeadm init command & configmap for cilium as far as it doesn't contain any public IPs?

[joestringer]

@ByteAlex is the 10.96.0.1 the service IP of the kubernetes service in the default namespace?

[ByteAlex]

root@test-cluster-master-01:~# k get svc -A
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  3h4m
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   3h3m

Oh indeed! Idk why I was looking at pods -owide

[joestringer]

@ByteAlex Can you use cilium monitor or hubble observe to filter traffic from the pod having trouble, and see if the traffic is indeed exiting the pod successfully?

[ByteAlex]

Where can I find those commands?
Nvmd, I can see them by attaching to the cilium pod on the correct node.

root@test-cluster-master-01:/home/cilium# cilium monitor
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> stack flow 0x90d56154 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.0.95:4240 -> 138.201.94.89:48992 tcp ACK
-> endpoint 1188 flow 0x4dd2a957 identity 1->4 state established ifindex 0 orig-ip 138.201.94.89: 138.201.94.89:48992 -> 10.20.0.95:4240 tcp ACK
-> stack flow 0x90d56154 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.0.95:4240 -> 138.201.94.89:48992 tcp ACK
-> endpoint 1188 flow 0x4dd2a957 identity 1->4 state established ifindex 0 orig-ip 138.201.94.89: 138.201.94.89:48992 -> 10.20.0.95:4240 tcp ACK
-> stack flow 0x90d56154 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.0.95:4240 -> 138.201.94.89:48992 tcp ACK
-> endpoint 1188 flow 0x4dd2a957 identity 1->4 state established ifindex 0 orig-ip 138.201.94.89: 138.201.94.89:48992 -> 10.20.0.95:4240 tcp ACK
>> IPCache entry upserted: {"cidr":"138.201.94.89/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.0.0.2/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.20.0.251/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"0.0.0.0/0","id":2,"old-id":2,"encrypt-key":0}
-> endpoint 1188 flow 0x4dd2a957 identity 1->4 state established ifindex 0 orig-ip 138.201.94.89: 138.201.94.89:48992 -> 10.20.0.95:4240 tcp ACK
-> stack flow 0x90d56154 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.0.95:4240 -> 138.201.94.89:48992 tcp ACK
-> endpoint 1188 flow 0x0 identity 1->4 state new ifindex 0 orig-ip 138.201.94.89: 138.201.94.89 -> 10.20.0.95 EchoRequest
-> stack flow 0x0 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.0.95 -> 138.201.94.89 EchoReply

I don't mind sharing the public ips here, since its just a test cluster

[joestringer]

@ByteAlex can you use https://docs.cilium.io/en/latest/operations/troubleshooting/#observing-flows-with-hubble to narrow down specifically to the pods that are having connectivity trouble? Or use -t drop to search specifically for dropped traffic?

[ByteAlex]

The only drop I saw in cilium monitor so far is:
xx drop (Unsupported protocol for NAT masquerade) flow 0x0 to endpoint 0, identity 0->0: 138.201.94.89 -> 80.94.93.22 DestinationUnreachable(Port)
But this seems unrelated to coredns, - I restarted a coreDNS instance, but no entry was written to the log.

138.201.94.89 is the public ip of my master node, 80.94.93.22 is some ip from the netherlands (possibly some scraper)

The worker node having the coredns pod also doesn't show any errors in the cilium monitor.

Big log:

root@test-cluster-worker-01:/home/cilium# cilium monitor
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> stack flow 0x226ca4fe identity 5121->6 state new ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37228 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0x65e5ac3b identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51642 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0xa268ce49 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51642 tcp SYN, ACK
-> endpoint 2806 flow 0x65e5ac3b identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51642 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x65e5ac3b identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51642 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0xa268ce49 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51642 tcp ACK
-> endpoint 2806 flow 0x65e5ac3b identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51642 -> 10.20.1.1:8181 tcp ACK, FIN
-> stack flow 0xa268ce49 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51642 tcp ACK, FIN
-> endpoint 2806 flow 0x65e5ac3b identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51642 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0xecf20636 identity 5121->6 state new ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37238 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0xd99829c3 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44062 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x1d95cebd identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44062 tcp SYN, ACK
-> endpoint 2806 flow 0xd99829c3 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44062 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xd99829c3 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44062 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x1d95cebd identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44062 tcp ACK
-> stack flow 0x1d95cebd identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44062 tcp ACK, FIN
-> endpoint 2806 flow 0xd99829c3 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44062 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> stack flow 0x549028fa identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37228 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0x690177a5 identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37238 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0xcad6aa31 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51676 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x30223daf identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51676 tcp SYN, ACK
-> endpoint 2806 flow 0xcad6aa31 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51676 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xcad6aa31 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51676 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xcad6aa31 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51676 -> 10.20.1.1:8181 tcp ACK, FIN
-> stack flow 0x30223daf identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51676 tcp ACK
-> stack flow 0x30223daf identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51676 tcp ACK, FIN
-> endpoint 2806 flow 0xa0f03c59 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44094 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x3111b084 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44094 tcp SYN, ACK
-> endpoint 2806 flow 0xa0f03c59 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44094 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x3111b084 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44094 tcp ACK
-> stack flow 0x3111b084 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44094 tcp ACK, FIN
-> endpoint 2806 flow 0xa0f03c59 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44094 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xa0f03c59 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44094 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0x1c2e1469 identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37228 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xb40335e2 identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37238 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> endpoint 2806 flow 0xfb8d0e48 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51704 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x1c285dcd identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51704 tcp SYN, ACK
-> endpoint 2806 flow 0xfb8d0e48 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51704 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xfb8d0e48 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51704 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x1c285dcd identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51704 tcp ACK
-> stack flow 0x1c285dcd identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51704 tcp ACK, FIN
-> endpoint 2806 flow 0xfb8d0e48 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51704 -> 10.20.1.1:8181 tcp ACK, FIN
-> endpoint 2806 flow 0xb2db4093 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44122 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0xc0da366b identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44122 tcp SYN, ACK
-> endpoint 2806 flow 0xb2db4093 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44122 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xb2db4093 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44122 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0xc0da366b identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44122 tcp ACK
-> stack flow 0xc0da366b identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44122 tcp ACK, FIN
-> endpoint 2806 flow 0xb2db4093 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44122 -> 10.20.1.1:8080 tcp ACK, FIN
-> endpoint 2806 flow 0x7c35cf8 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51730 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x877a1fd7 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51730 tcp SYN, ACK
-> endpoint 2806 flow 0x7c35cf8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51730 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x7c35cf8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51730 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x877a1fd7 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51730 tcp ACK
-> stack flow 0x877a1fd7 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51730 tcp ACK, FIN
-> endpoint 2806 flow 0x7c35cf8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51730 -> 10.20.1.1:8181 tcp ACK, FIN
>> IPCache entry upserted: {"cidr":"49.12.44.179/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.0.0.3/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.20.1.224/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"0.0.0.0/0","id":2,"old-id":2,"encrypt-key":0}
-> endpoint 2806 flow 0xd3e5a035 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44148 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x10b26af1 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44148 tcp SYN, ACK
-> endpoint 2806 flow 0xd3e5a035 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44148 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xd3e5a035 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44148 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x10b26af1 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44148 tcp ACK
-> stack flow 0x10b26af1 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44148 tcp ACK, FIN
-> endpoint 2806 flow 0xd3e5a035 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44148 -> 10.20.1.1:8080 tcp ACK, FIN
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0x0 identity 1->4 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179 -> 10.20.1.254 EchoRequest
-> stack flow 0x0 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254 -> 49.12.44.179 EchoReply
-> endpoint 2806 flow 0x3a8ec2a8 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51760 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x38fb9415 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51760 tcp SYN, ACK
-> endpoint 2806 flow 0x3a8ec2a8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51760 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x3a8ec2a8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51760 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x3a8ec2a8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51760 -> 10.20.1.1:8181 tcp ACK, FIN
-> stack flow 0x38fb9415 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51760 tcp ACK
-> stack flow 0x38fb9415 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51760 tcp ACK, FIN
-> endpoint 2806 flow 0x1cd47855 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44178 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0xe0baaa2 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44178 tcp SYN, ACK
-> endpoint 2806 flow 0x1cd47855 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44178 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0x1cd47855 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44178 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0xe0baaa2 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44178 tcp ACK
-> stack flow 0xe0baaa2 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44178 tcp ACK, FIN
-> endpoint 2806 flow 0x1cd47855 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44178 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> endpoint 2806 flow 0x42706cb identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51788 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x1898506a identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51788 tcp SYN, ACK
-> endpoint 2806 flow 0x42706cb identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51788 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x42706cb identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51788 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x1898506a identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51788 tcp ACK
-> stack flow 0x1898506a identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51788 tcp ACK, FIN
-> endpoint 2806 flow 0x42706cb identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51788 -> 10.20.1.1:8181 tcp ACK, FIN
-> stack flow 0xf430e97 identity 5121->6 state new ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37384 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0x77ad158e identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44208 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0xb542e60c identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44208 tcp SYN, ACK
-> endpoint 2806 flow 0x77ad158e identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44208 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0x77ad158e identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44208 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0xb542e60c identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44208 tcp ACK
-> stack flow 0xb542e60c identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44208 tcp ACK, FIN
-> endpoint 2806 flow 0x77ad158e identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44208 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0x36a3777b identity 5121->6 state new ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37404 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0x7f5ae219 identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37384 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0x9e974faf identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51818 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x5c8a6bff identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51818 tcp SYN, ACK
-> endpoint 2806 flow 0x9e974faf identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51818 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x9e974faf identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51818 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x5c8a6bff identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51818 tcp ACK
-> stack flow 0x5c8a6bff identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51818 tcp ACK, FIN
-> endpoint 2806 flow 0x9e974faf identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51818 -> 10.20.1.1:8181 tcp ACK, FIN
-> endpoint 2806 flow 0x5efeb8c0 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44236 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0xa42d711 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44236 tcp SYN, ACK
-> endpoint 2806 flow 0x5efeb8c0 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44236 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0x5efeb8c0 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44236 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0xa42d711 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44236 tcp ACK
-> stack flow 0xa42d711 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44236 tcp ACK, FIN
-> endpoint 2806 flow 0x5efeb8c0 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44236 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> stack flow 0x73c889ed identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37404 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xfd50778d identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37384 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0x23af3bd8 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51852 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x8528dd9b identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51852 tcp SYN, ACK
-> endpoint 2806 flow 0x23af3bd8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51852 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x23af3bd8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51852 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x23af3bd8 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51852 -> 10.20.1.1:8181 tcp ACK, FIN
-> stack flow 0x8528dd9b identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51852 tcp ACK
-> stack flow 0x8528dd9b identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51852 tcp ACK, FIN
-> endpoint 2806 flow 0x6b695173 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44270 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x776d24f3 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44270 tcp SYN, ACK
-> endpoint 2806 flow 0x6b695173 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44270 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0x6b695173 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44270 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x776d24f3 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44270 tcp ACK
-> stack flow 0x776d24f3 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44270 tcp ACK, FIN
-> endpoint 2806 flow 0x6b695173 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44270 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0x13317daa identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37404 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> endpoint 2806 flow 0xe19e1aa6 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51880 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0xbcc22ab6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51880 tcp SYN, ACK
-> endpoint 2806 flow 0xe19e1aa6 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51880 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xe19e1aa6 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51880 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0xbcc22ab6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51880 tcp ACK
-> stack flow 0xbcc22ab6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51880 tcp ACK, FIN
-> endpoint 2806 flow 0xe19e1aa6 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51880 -> 10.20.1.1:8181 tcp ACK, FIN
-> endpoint 2806 flow 0xf23c9591 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44298 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x3f3cf697 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44298 tcp SYN, ACK
-> endpoint 2806 flow 0xf23c9591 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44298 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xf23c9591 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44298 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x3f3cf697 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44298 tcp ACK
-> endpoint 2806 flow 0xf23c9591 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44298 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0x3f3cf697 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44298 tcp ACK, FIN
-> stack flow 0x54d454e3 identity 5121->6 state new ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37496 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0xb4137020 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51908 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x308eda7e identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51908 tcp SYN, ACK
-> endpoint 2806 flow 0xb4137020 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51908 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xb4137020 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51908 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x308eda7e identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51908 tcp ACK
-> stack flow 0x308eda7e identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51908 tcp ACK, FIN
-> endpoint 2806 flow 0xb4137020 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51908 -> 10.20.1.1:8181 tcp ACK, FIN
>> IPCache entry upserted: {"cidr":"49.12.44.179/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.0.0.3/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"10.20.1.224/32","id":1,"old-id":1,"encrypt-key":0}
>> IPCache entry upserted: {"cidr":"0.0.0.0/0","id":2,"old-id":2,"encrypt-key":0}
-> endpoint 2806 flow 0xd22e3257 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44326 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x9583b92f identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44326 tcp SYN, ACK
-> endpoint 2806 flow 0xd22e3257 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44326 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xd22e3257 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44326 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x9583b92f identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44326 tcp ACK
-> stack flow 0x9583b92f identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44326 tcp ACK, FIN
-> endpoint 2806 flow 0xd22e3257 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44326 -> 10.20.1.1:8080 tcp ACK, FIN
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0x0 identity 1->4 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179 -> 10.20.1.254 EchoRequest
-> stack flow 0x0 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254 -> 49.12.44.179 EchoReply
-> stack flow 0x1738e9b3 identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37496 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0x3495309c identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51938 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x4bd13df6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51938 tcp SYN, ACK
-> endpoint 2806 flow 0x3495309c identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51938 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x3495309c identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51938 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x4bd13df6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51938 tcp ACK
-> stack flow 0x4bd13df6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51938 tcp ACK, FIN
-> endpoint 2806 flow 0x3495309c identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51938 -> 10.20.1.1:8181 tcp ACK, FIN
-> endpoint 2806 flow 0x50de3143 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44356 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x1a413bf6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44356 tcp SYN, ACK
-> endpoint 2806 flow 0x50de3143 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44356 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0x50de3143 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44356 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x1a413bf6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44356 tcp ACK
-> stack flow 0x1a413bf6 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44356 tcp ACK, FIN
-> endpoint 2806 flow 0x50de3143 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44356 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0x21eea7e9 identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37496 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> endpoint 2806 flow 0xef79f997 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51966 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0xf49beaba identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51966 tcp SYN, ACK
-> endpoint 2806 flow 0xef79f997 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51966 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xef79f997 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51966 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0xf49beaba identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51966 tcp ACK
-> stack flow 0xf49beaba identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51966 tcp ACK, FIN
-> endpoint 2806 flow 0xef79f997 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51966 -> 10.20.1.1:8181 tcp ACK, FIN
-> endpoint 2806 flow 0x63bc5e40 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44384 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0xfbd83e0d identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44384 tcp SYN, ACK
-> endpoint 2806 flow 0x63bc5e40 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44384 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0x63bc5e40 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44384 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0xfbd83e0d identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44384 tcp ACK
-> stack flow 0xfbd83e0d identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44384 tcp ACK, FIN
-> endpoint 2806 flow 0x63bc5e40 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44384 -> 10.20.1.1:8080 tcp ACK, FIN
-> endpoint 2806 flow 0x661dcf13 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51992 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x8693c308 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51992 tcp SYN, ACK
-> endpoint 2806 flow 0x661dcf13 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51992 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0x661dcf13 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51992 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x8693c308 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51992 tcp ACK
-> endpoint 2806 flow 0x661dcf13 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51992 -> 10.20.1.1:8181 tcp ACK, FIN
-> stack flow 0x8693c308 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:51992 tcp ACK, FIN
-> endpoint 2806 flow 0x661dcf13 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:51992 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xa173a520 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44410 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0x77ff5370 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44410 tcp SYN, ACK
-> endpoint 2806 flow 0xa173a520 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44410 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xa173a520 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44410 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0x77ff5370 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44410 tcp ACK
-> stack flow 0x77ff5370 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44410 tcp ACK, FIN
-> endpoint 2806 flow 0xa173a520 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44410 -> 10.20.1.1:8080 tcp ACK, FIN
-> stack flow 0xa435ba82 identity 4->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.254:4240 -> 49.12.44.179:49470 tcp ACK
-> endpoint 114 flow 0xcd61e79e identity 1->4 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:49470 -> 10.20.1.254:4240 tcp ACK
-> stack flow 0x1ae2883f identity 5121->6 state new ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37608 -> 138.201.94.89:6443 tcp SYN
-> endpoint 2806 flow 0xf3dcec1c identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:52024 -> 10.20.1.1:8181 tcp SYN
-> stack flow 0x24fc9506 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:52024 tcp SYN, ACK
-> endpoint 2806 flow 0xf3dcec1c identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:52024 -> 10.20.1.1:8181 tcp ACK
-> endpoint 2806 flow 0xf3dcec1c identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:52024 -> 10.20.1.1:8181 tcp ACK
-> stack flow 0x24fc9506 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:52024 tcp ACK
-> stack flow 0x24fc9506 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8181 -> 49.12.44.179:52024 tcp ACK, FIN
-> endpoint 2806 flow 0xf3dcec1c identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:52024 -> 10.20.1.1:8181 tcp ACK, FIN
-> endpoint 2806 flow 0xe249ff29 identity 1->5121 state new ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44442 -> 10.20.1.1:8080 tcp SYN
-> stack flow 0xda2ffa95 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44442 tcp SYN, ACK
-> endpoint 2806 flow 0xe249ff29 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44442 -> 10.20.1.1:8080 tcp ACK
-> endpoint 2806 flow 0xe249ff29 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44442 -> 10.20.1.1:8080 tcp ACK
-> stack flow 0xda2ffa95 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44442 tcp ACK
-> stack flow 0xda2ffa95 identity 5121->1 state reply ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:8080 -> 49.12.44.179:44442 tcp ACK, FIN
-> endpoint 2806 flow 0xe249ff29 identity 1->5121 state established ifindex 0 orig-ip 49.12.44.179: 49.12.44.179:44442 -> 10.20.1.1:8080 tcp ACK, FIN

But given here's so many output I wonder why the master-01 has so low output on the monitor.

[joestringer]

But given here's so many output I wonder why the master-01 has so low output on the monitor.

Looking at the output, it seems like this is mostly packets flowing between a local pod and the local node, but with just a few failed attempts to connect to remote nodes (the master, perhaps?) like this:

-> stack flow 0x73c889ed identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37404 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xfd50778d identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37384 -> 138.201.94.89:6443 tcp SYN

[ByteAlex]

But given here's so many output I wonder why the master-01 has so low output on the monitor.

Looking at the output, it seems like this is mostly packets flowing between a local pod and the local node, but with just a few failed attempts to connect to remote nodes (the master, perhaps?) like this:

-> stack flow 0x73c889ed identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37404 -> 138.201.94.89:6443 tcp SYN
-> stack flow 0xfd50778d identity 5121->6 state established ifindex 0 orig-ip 0.0.0.0: 10.20.1.1:37384 -> 138.201.94.89:6443 tcp SYN

138.201.94.89:6443 is indeed my master ip. What does that mean?

[joestringer]

What does that mean?

Well, searching for port 37404 in all the output only shows TCP SYN packets being sent. Notably, there is no SYN-ACK response, which means that either the destination didn't receive the SYN, or the destination replied with a SYN-ACK and the network didn't deliver the reply to the current node.

Next steps I would suggest would be to try to narrow down which case occurred, you could for instance use tcpdump on the devices of both nodes to see whether this pod is able to send packets to the destination or not.

[ByteAlex]

Okay, I used tcpdump to write log files on all three hosts on 2 interfaces: eth0 (public) & enp7s0 (private).
I'm not an expert with networking but as far as I know these are the interfaces through which all traffic must be bridged for routing outside of the machine.

Public ip dump "src worker-01 and port 6443 or dst worker-01 and srcport 6443"
(ip.src == 49.12.44.179 && tcp.port == 6443) || (ip.dst == 49.12.44.179 && tcp.srcport == 6443)

Using the internal ips dump with the "missing ack"
"src = 10.20.1.1 and port = 37404"
-> EMPTY

[ByteAlex]

I just updated to the latest cilium version, but it didn't fix my issue.
I found out something else though.

I am able to curl that server from both nodes (since it uses native networking), but from inside kubernetes it can't reach this ip.

[ByteAlex]

TL;DR
The cause for the connectivity outage was a misconfiguration of native-routing-cidr and cluster-pool-cidr.
The following issues should not be handled here. This could be closed I guess @joestringer

[joestringer]

@ByteAlex thanks for the update! Would you be able to briefly describe what the misconfiguration looked like? How did you know in the end that it was misconfigured?

I'll close this out since it's resolved, but it may help future users to have additional details posted below in case they hit it in future.

[ByteAlex]

The issue was me misunderstanding the scopes of the configuration.
native-routing-cidr is the whole CIDR which can be used for native routing, e.g. my hetzner vlan is 10.0.0.0/8, so that's the native routing cidr.
For the cluster-pool it should be the same as the kubernetes pod range (from my understanding!), which is by default 10.224.0.0/16