agones-ping-udp-service brings up incompatible load balancer in EKS

[structurefall]

What happened:

When installing Agones in an EKS cluster, the agones-ping-udp-service fails to create with the following error:

Warning CreatingLoadBalancerFailed 1s (x4 over 37s) service-controller Error creating load balancer (will retry): failed to ensure load balancer for service agones-system/agones-ping-udp-service: Only TCP LoadBalancer is supported for AWS ELB

What you expected to happen:
The service should have created successfully.

How to reproduce it (as minimally and precisely as possible):
Build an EKS cluster, follow installation instructions for Agones.

Anything else we need to know?:
Yes, we already know the fix! The service needs the following annotation:
annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb

Environment:

• Agones version: 1.0.0
• Kubernetes version (use kubectl version): 1.12
• Cloud provider or hardware configuration: AWS/EKS
• Install method (yaml/helm): YAML, but I checked the Helm chart and it would have the same issue
• Troubleshooting guide log(s): n/a
• Others:

[markmandel]

I figure we could set this annotation by default. I don't think it's going to affect other cloud platform's load balancers?

[aLekSer]

I can check if it would work.
/assign alekser

[aLekSer]

I tried the fix which you suggested and it has not worked for the Kubernetes version: 1.12.
https://github.com/kubernetes/kubernetes/issues/79523
I think we should wait till 1.17 for bug triage as per comments.

[aLekSer]

kubectl describe agones-ping-udp-service output:

Name:         agones-ping-udp-service
Namespace:    agones-system
Labels:       app=agones
              chart=agones-1.1.0
              component=ping
              heritage=Tiller
              release=agones-manual
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-type":"nlb"},"labels":{"app":...
              service.beta.kubernetes.io/aws-load-balancer-type: nlb
API Version:  v1
Kind:         Service
Metadata:
  Creation Timestamp:  2019-11-01T11:49:47Z
  Resource Version:    4196
  Self Link:           /api/v1/namespaces/agones-system/services/agones-ping-udp-service
  UID:                 b44a207f-fc9d-11e9-b526-0ad256200a94
Spec:
  Cluster IP:               10.100.25.80
  External Traffic Policy:  Cluster
  Ports:
    Name:         udp
    Node Port:    30518
    Port:         50000
    Protocol:     UDP
    Target Port:  8080
  Selector:
    agones.dev/role:  ping
  Session Affinity:   None
  Type:               LoadBalancer
Status:
  Load Balancer:
Events:
  Type     Reason                      Age                From                Message
  ----     ------                      ----               ----                -------
  Normal   EnsuringLoadBalancer        51s (x2 over 82s)  service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  51s (x2 over 82s)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service agones-system/agones-ping-udp-service: Only TCP LoadBalancer is supported for AWS ELB

[structurefall]

@aLekSer based on your output, it looks like the service is still trying to build without the annotation. You need to completely destroy the service and rebuild it for that to work. When I tested this it was in fact on 1.12.

You're getting the same error message that indicates that the load balancer was built sans annotation: "Only TCP LoadBalancer is supported for AWS ELB." There's a brief AWS doc on the annotation here.

[aLekSer]

I will soon recheck this if it still actual.

[aLekSer]

Tested this on the most recent Terraform EKS config with udp_expose = "true" in examples/terraform-submodules/eks/module.tf:

  Type     Reason                      Age                    From                Message
  ----     ------                      ----                   ----                -------
  Normal   EnsuringLoadBalancer        4m31s (x7 over 9m46s)  service-controller  Ensuring load balancer                                                                                                 
  Warning  CreatingLoadBalancerFailed  4m31s (x7 over 9m46s)  service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service agones-system/agones-ping-udp-service: Only TCP LoadBalancer is supported for AWS ELB
BY-IT00060:eks alexander.apalikov$ kubectl get services --namespace agones-system
NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)            AGE                                             
agones-allocator            LoadBalancer   172.20.90.48     abf6453fe41e311eaa72b02706d23ee8-806979722.us-west-2.elb.amazonaws.com    443:31237/TCP      39m                                             
agones-controller-service   ClusterIP      172.20.120.21    <none>                                                                    443/TCP,8080/TCP   39m                                             
agones-ping-http-service    LoadBalancer   172.20.172.167   abf5f999b41e311eaa72b02706d23ee8-1635672743.us-west-2.elb.amazonaws.com   80:30456/TCP       39m                                             
agones-ping-udp-service     LoadBalancer   172.20.196.38    <pending>                                                                 50000:30098/UDP    10m      

Will try to add annotations to helm.tf.
We should wait for this PR to be merged:
https://github.com/kubernetes/kubernetes/pull/87549/files
It seems that Kubernetes code itself does not allow this to happen.

[aLekSer]

It seems that the fix could happen in 1.19 version only as per this comment:
https://github.com/kubernetes/kubernetes/pull/87549#issuecomment-595440964

[roberthbailey]

1.19 is a long way away (especially on EKS). Is there anything we can do in the mean time? Is there a documentation change we can make to explain to EKS users how to fix this manually?

[aLekSer]

Yes, I am also looking into a documentation update.

[TBBle]

The upstream fix for this has landed in k8s master, so will be in 1.19, which should be available in EKS sometime in 2021.

[TBBle]

I didn't notice until recently, that AWS EKS has backported the NLB UDP support from 1.19 as far back as 1.15, per Platform versions:

• 1.15.11.eks.4 or later
• 1.16.13.eks.3 or later
• 1.17.9.eks.2 or later

1.18 doesn't mention it, but the above backports (August 12th) predate 1.18 on EKS (October 13th), so it _should_ have been included there too. My team expects to deploy Agones on an EKS 1.18 platform sometime next month, so I'll try and remember to check that; we disable the UDP-ping service by default, so I'll ask the person doing the deployment to try leaving it on with an NLB (or ideally NLB-IP) annotation.

EKS support for 1.14 ends on December 8th, so I believe this issue is in a good state to close, with the caveat that 1.14 doesn't support it, and never will. However, I have not _tested_ the fix myself yet.

[markmandel]

Is this solved? I think it might be, but I'm not sure.

Can someone confirm, so we can close this issue?

[tenevdev]

I think you still need to set the annotation for the agones-ping Service service.beta.kubernetes.io/aws-load-balancer-type: nlb in the installation sources if you want it to work out of the box. We use kustomize and have an overlay for it.

[TBBle]

I think that would be a user-issue though? I wouldn't expect the Agones source to contain annotations for any/every kind of platform-specific environment.

And deploying things to AWS, I had to use such annotations everywhere (since I wanted NLB everywhere) so this wasn't a shock or a challenge _for me_.

The error could be nicer: "Use NLB mode" would be better than "ELB can't do this", but that's a k8s/AWS problem, not an Agones problem.

Particularly because for AWS, we now have both nlb and nlb-ip types, and the latter is better but requires a (not in-the-box) AWS Load Balancer Controller (due to the migration of cloud-specific controllers out of kubernetes core), so including _either_ annotation is going to be wrong for a significant group of AWS users, and nonsensical for non-AWS users.

I would have to check this, but I think for load balancers, AWS are trying to introduce different annotations, which adds 'external' to the list of possible types, in order to explicitly hand-off to the afore-mentioned AWS Load Balancer Controller.

If there's AWS-specific instructions around, it's worth mentioning the options there, I guess, with appropriate links

[markmandel]

If there are docs to be added to: https://agones.dev/site/docs/installation/creating-cluster/eks/ - that is always appreciated!

[vara-bonthu]

@markmandel Yes, the following annotations needs to be added if we have to replace Classic Load balancers with Network Load Balancers. NLB supports UDP s

I am currently testing the annotations on Agones Helm chart and the following works on EKS 1.19 and Agones 1.15. It creates two NLBs, one for HTTP and other one for UDP ping.

http:
  expose: true
  response: ok
  port: 80
  serviceType: LoadBalancer
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "false"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: ${s3_bucket}
udp:
  expose: true
  rateLimit: 20
  port: 50000
  serviceType: LoadBalancer
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "false"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: ${s3_bucket}

[markmandel]

@markmandel Yes, the following annotations needs to be added if we have to replace Classic Load balancers with Network Load Balancers. NLB supports UDP s

Can someone who clearly knows EKS way better than I do file a PR for the docs? That would be aces.

Sounds like you've got it worked out @vara-bonthu ?