Containers-roadmap: EKS:Feature:NLB/ELB Ingress

[dayyehm]

Tell us about your request
What do you want us to build?
Supporting advanced features to allow crearting ELB/NLB with specific subnets ,specific EIPs via K8s service kind, or via Ingress controllers.

Which service(s) is this request for?
EKS (Ingress NLB , ELB)
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Creating NLB with static IPs in specific public subnets , (2 or 3 subnets)
Seems that k8s version should 1.16 or 1.17 for this to work as some customer using different release builds :
https://github.com/kubernetes/kubernetes/issues/63959
Is it possible to add this to the current 1.13 builds ? or if its possible to be considered for the next 1.14/1.15 builds

Appreciated.

Are you currently working around this issue?
How are you currently solving this problem?
No solution currently we chose to use ALB

Additional context
Anything else we should know?

Testing results :

based on the issue : https://github.com/kubernetes/kubernetes/issues/63959
some people managed to get this working with their k8s but not on EKS.

---

https://kubernetes.github.io/ingress-nginx/deploy/#network-load-balancer-nlb
Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB

================================
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc-07e3afcd4b7b5d644,eipalloc-0d9cb0154be5ab55d"
service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-061f4a497621a7179,subnet-001c2e5df9cc93960"
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
name: http
port: 80
targetPort: http
name: https
port: 443

targetPort: https

Thanks

[whereisaaron]

We currently work around this by creating NLB/ELBs outside EKS and then associating the target groups with the deployment of the EKS clusters. More trouble, but on the upside it is easier to move NLB/ELBs between clusters, or span multiple clusters with one NLB/ELB for redundancy or migration.

[schahal]

We currently work around this by creating NLB/ELBs outside EKS and then associating the target groups with the deployment of the EKS clusters

A significant downside of this approach is that if the ingress-controller has changes (e.g., it must update the node ports of the targets), it won't be able to succeed because of:

Error creating load balancer listener: "TargetGroupAssociationLimit: Target group 'arn:aws:elasticloadbalancing:<regio>:<accojnt>:targetgroup/k8s-tg-xxx/yyy' cannot be associated with more than one load balancer

We can work around even this by temporarily creating a clone of the TG, attaching to the outsided NLB, etc, etc, reattaching. Suck, as there's still downtime (even for couple seconds).

I sure hope this change https://github.com/kubernetes/kubernetes/blob/release-1.16/staging/src/k8s.io/legacy-cloud-providers/aws/aws_loadbalancer.go#L140 can be backported into Kubernetes (and therefore EKS) much earlier.

[whereisaaron]

@schahal oh we don't let k8s manage the target groups. We manage those too, and set the ingress services to fixed NodePort numbers.

[schahal]

Interesting @whereisaaron! Fixed NodePorts would work, especially if nodes stay static. But we autoscale up and down the actual Nodes themselves, so not letting k8s manage the target groups (to add/remove the new instances) incurs a lot of overhead.

[whereisaaron]

@schahal, no overhead, the target groups are associated with the ASG for each node pool, so as the clusters scale up/down the new nodes automatically have all the required target groups.

[mtparet]

Is it fixed ?

A new annotation: service.beta.kubernetes.io/aws-load-balancer-eip-allocations is available on service type LoadBalancer to assign an elastic IP address to Network Load Balancers. For more information, see the Support EIP Allocations with AWS NLB GitHub issue.

https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

[astrived]

@dayyehm given that we already released EKS 1.16 is there any drivers to keep this open ? See this https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html "A new annotation: service.beta.kubernetes.io/aws-load-balancer-eip-allocations is available on service type LoadBalancer to assign an elastic IP address to Network Load Balancers. For more information, see the Support EIP Allocations with AWS NLB GitHub issue. "

[astrived]

I am marking this fixed with EKS 1.16.