Amazon-vpc-cni-k8s: Routing Outside VPC - AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS not helping

Created on 14 Jul 2020 · 4Comments · Source: aws/amazon-vpc-cni-k8s

I have set up a private cluster in AWS using kops with amazonvpc networking. I'm using:

amazon-k8s-cni:v1.6.1
kops: Version 1.17.1
kubectl client version: v1.16.6-beta.0
kubectl server version: v1.17.8
iptables: v1.6.0

Currently I can connect to kubernetes services from other instances in the same VPC as the worker node by using the service's cluster ip and port. However I have some instances in a different VPC that also need access to the pods in the kubernetes cluster. These instances have access to the kubernetes subnet via a peering connection, and can ssh and access all ports on the kubernetes ec2 instances, to the point any kubernetes services using nodeport can be accessed. My issue is I still cannot access the kubernetes services using their cluster IP/ports, even when configuring AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS to the CIDR of the subnet I am connecting from.

The kubernetes VPC has CIDR 172.20.0.0/16, I am trying to access the services from CIDR 172.31.0.0/16, thus AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS=172.31.0.0/16.

I have pasted the iptables rules on the worker node below (this cluster only has one worker):

admin@ip-172-20-54-186:~$ sudo iptables --table nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere            !ip-127-0-0-0.eu-central-1.compute.internal/8  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
MASQUERADE  all  --  ip-172-17-0-0.eu-central-1.compute.internal/16  anywhere            
AWS-SNAT-CHAIN-0  all  --  anywhere             anywhere             /* AWS SNAT CHAIN */

Chain AWS-SNAT-CHAIN-0 (1 references)
target     prot opt source               destination         
AWS-SNAT-CHAIN-1  all  --  anywhere            !ip-172-20-0-0.eu-central-1.compute.internal/16  /* AWS SNAT CHAIN */

Chain AWS-SNAT-CHAIN-1 (1 references)
target     prot opt source               destination         
AWS-SNAT-CHAIN-2  all  --  anywhere            !ip-172-31-0-0.eu-central-1.compute.internal/16  /* AWS SNAT CHAIN EXCLUSION */

Chain AWS-SNAT-CHAIN-2 (1 references)
target     prot opt source               destination         
SNAT       all  --  anywhere             anywhere             /* AWS, SNAT */ ADDRTYPE match dst-type !LOCAL to:172.20.54.186 random

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-MARK-DROP (0 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x8000

Chain KUBE-MARK-MASQ (12 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* default/alert-dashboard-prototype: */ tcp dpt:30006
KUBE-SVC-YT6IJS2ZURHRWZVV  tcp  --  anywhere             anywhere             /* default/alert-dashboard-prototype: */ tcp dpt:30006
KUBE-MARK-MASQ  tcp  --  anywhere             anywhere             /* default/mlflow: */ tcp dpt:30005
KUBE-SVC-SUR26NGXXMWY7ANJ  tcp  --  anywhere             anywhere             /* default/mlflow: */ tcp dpt:30005

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere             /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-SEP-5P2GS3OZGNQIGBYV (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-56-184.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.56.184:8080

Chain KUBE-SEP-C7AQ7QKHZJIERVXY (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-39-235.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.39.235:8080

Chain KUBE-SEP-IVYNVMRUZ2JYPJXN (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-55-197.eu-central-1.compute.internal  anywhere            
DNAT       udp  --  anywhere             anywhere             udp to:172.20.55.197:53

Chain KUBE-SEP-RP3TCMRWLCNPKQJB (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-39-54.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.39.54:1433

Chain KUBE-SEP-TDYJCYUZN5Z45XI6 (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-43-110.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.43.110:443

Chain KUBE-SEP-TU2VU5AJQZAPKFBO (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-37-1.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.37.1:5000

Chain KUBE-SEP-VY2QQ5LYHVRSNPKA (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-54-241.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.54.241:53

Chain KUBE-SEP-Y2ZDC26P4CGSCEFP (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-55-197.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.55.197:53

Chain KUBE-SEP-YQFBF2KR5ODFKP7A (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-54-241.eu-central-1.compute.internal  anywhere            
DNAT       udp  --  anywhere             anywhere             udp to:172.20.54.241:53

Chain KUBE-SEP-ZKNJSETAPXK2WCFI (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  ip-172-20-40-44.eu-central-1.compute.internal  anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp to:172.20.40.44:8050

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-SVC-YT6IJS2ZURHRWZVV  tcp  --  anywhere             ip-172-20-8-115.eu-central-1.compute.internal  /* default/alert-dashboard-prototype: cluster IP */ tcp dpt:http-alt
KUBE-SVC-FUIYR2VHB4CCYAP6  tcp  --  anywhere             ip-172-20-20-33.eu-central-1.compute.internal  /* default/sql-old: cluster IP */ tcp dpt:ms-sql-s
KUBE-SVC-SUR26NGXXMWY7ANJ  tcp  --  anywhere             ip-172-20-6-144.eu-central-1.compute.internal  /* default/mlflow: cluster IP */ tcp dpt:5000
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  anywhere             ip-172-20-0-10.eu-central-1.compute.internal  /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  anywhere             ip-172-20-0-10.eu-central-1.compute.internal  /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-LS2CJ4TUIKQ6TN65  tcp  --  anywhere             ip-172-20-2-168.eu-central-1.compute.internal  /* default/px-pypi: cluster IP */ tcp dpt:http-alt
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere             ip-172-20-0-1.eu-central-1.compute.internal  /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-V7FUGYMVWIORDW44  tcp  --  anywhere             ip-172-20-9-58.eu-central-1.compute.internal  /* default/px-dash: cluster IP */ tcp dpt:8050
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
target     prot opt source               destination         
KUBE-SEP-VY2QQ5LYHVRSNPKA  all  --  anywhere             anywhere             statistic mode random probability 0.50000000000
KUBE-SEP-Y2ZDC26P4CGSCEFP  all  --  anywhere             anywhere            

Chain KUBE-SVC-FUIYR2VHB4CCYAP6 (1 references)
target     prot opt source               destination         
KUBE-SEP-RP3TCMRWLCNPKQJB  all  --  anywhere             anywhere            

Chain KUBE-SVC-LS2CJ4TUIKQ6TN65 (1 references)
target     prot opt source               destination         
KUBE-SEP-C7AQ7QKHZJIERVXY  all  --  anywhere             anywhere            

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target     prot opt source               destination         
KUBE-SEP-TDYJCYUZN5Z45XI6  all  --  anywhere             anywhere            

Chain KUBE-SVC-SUR26NGXXMWY7ANJ (2 references)
target     prot opt source               destination         
KUBE-SEP-TU2VU5AJQZAPKFBO  all  --  anywhere             anywhere            

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
target     prot opt source               destination         
KUBE-SEP-YQFBF2KR5ODFKP7A  all  --  anywhere             anywhere             statistic mode random probability 0.50000000000
KUBE-SEP-IVYNVMRUZ2JYPJXN  all  --  anywhere             anywhere            

Chain KUBE-SVC-V7FUGYMVWIORDW44 (1 references)
target     prot opt source               destination         
KUBE-SEP-ZKNJSETAPXK2WCFI  all  --  anywhere             anywhere            

Chain KUBE-SVC-YT6IJS2ZURHRWZVV (2 references)
target     prot opt source               destination         
KUBE-SEP-5P2GS3OZGNQIGBYV  all  --  anywhere             anywhere

I have also tried setting AWS_VPC_K8S_CNI_EXTERNALSNAT=true but this did not solve anything. I'm at a loss how to proceed as have limited experience with configuring routing tables like this. Any help is greatly appreciated!

question

Source

shonik09

All 4 comments

@shaungupta

AWS_VPC_K8S_CNI_EXTERNALSNAT and AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS is for egress calls.

AWS_VPC_K8S_CNI_EXTERNALSNAT -> true means SNAT disable; false means SNAT enable

AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS: 1.2.3.4 -> means SNAT disable for dest. 1.2.3.4

You are calling from non-eks vpc to eks-vpc. So more of ingress call.

What you should try, calling alb-ingress from non-eks vpc. (alb-ingress will load balance traffic to k8s svc or podIP depending you setup of alb-target-type)

Hope this helps.

Thanks,
Bhavesh

bhaveshph on 17 Jul 2020

❤1

@bhaveshph OK so effectively I need to deploy the alb-ingress load balancer that works on the pod level? Do I need to just deploy a single load balancer, or multiple per pod? Is this all I need - https://aws.amazon.com/blogs/opensource/kubernetes-ingress-aws-alb-ingress-controller/?

Sorry I'm pretty new to this so any direction you can give me is appreciated...

shonik09 on 10 Aug 2020

@shaungupta
Sorry for delay in response, git did not notify me when it was addressed to me.

You may be new to ingress-controller, but should have good working knowledge of aws-alb.

You have correct link there, that is all you need, aws-alb-ingress-controller.

Just like load balancer, alb-ingress will load balance traffic to you pods (in case of ClusterIP) or send traffic to service for which load balance to pods (in case of NodePort). Pay real close attention to diagram in link you shared, it tells whole story.

Good luck.

Thanks,
Bhavesh

bhaveshph on 20 Aug 2020

👍1

@shaungupta Like Bhavesh mentioned, the service IP only works "inside" the cluster since it depends on kube-proxy setting up the rules on each node. To pass traffic from outside the cluster to the services, you need to configure an ingress, and ALB is probably your best bet.