Amazon-vpc-cni-k8s: curl do not pass through when both source node and target node are same

[ravicm]

Steps to reproduce the issue:

1. Create an eks cluster (Killer-eks in our case) with secondary IP support.
2. Create an autoscaling group of one node and attach to the cluster.
3. Create a network load balancer, listeners, target groups.
4. Create a hosted zone in Route 53. Add a recordset of type cname mapped to the loadbalancer.
5. Deploy nginx ingress controller in the cluster of nodeport type.
6. All the above are connected as NLB -> listeners -> Target groups -> autoscaling group nodes
7. Deploy a simple nginx deployment(my-nginx) in the cluster, map it to the nginx service(my-nginx) and to the ingress (my-nginx-ingress).
8. Deploy another simple nginx pod (my-nginx1) in the cluster.
9. Enter into one of the pods say my-nginx1
10. curl the ingress endpoints. curl fails

Slightly different environment is set up in my case and the test cases are demonstrated in the attached file in detail.
eks-nlb-issue-aws-support.txt

[paurosello]

We are facing the same issue, did you find a solution?

[jonathan-mothership]

I'm facing the same issue!
CNI v1.6.0
We have a different ingress controller (Traefik) and are seeing the same issue. In a 3-node ASG, a pod on node A that makes a request to an NLB where that request is served by same backend node running the requesting pod, the connection times out. We have two listeners and this happens on both the TLS and TCP listeners.

My suspicion is that this is because the NLB preserves the source IP of the request when connecting to the backend and the CNI iptables rules are getting tripped up with that connection.

[bhaveshph]

I am seeing this issue too.

But when I call from outside cluster node, it works fine
I guess, if I am on same eks cluster, I would use k8s-svc and not NLB; I guess I am fine for now; but interested to see what how it is happening.

[achevuru]

@ravicm @jonathan-mothership

Expected behavior with NLB when both Source and Target Nodes are same. Internal NLB do not support loopback or hair pinning. Since NLB in instance mode preserves the source ip of the packet, the response (say SYN ACK) will directly go to the local client pod. So, SYN will have client PodIP as SIP and NLB IP as DIP where as SYN ACK will have server PodIP as SIP(and not NLB IP) and client PodIP as DIP when both the pods are on the same node, so client pod will terminate/reset the TCP session.

Refer to “Connections time out for requests from a target to its load balancer” section in the below guide. It also lists out possible options. NLB in IP mode should help out in this scenario.

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html.

[jayanthvn]

Hi,

Since this is an expected behavior will be closing this issue for now.

Thank you!