Agones: failed calling webhook "mutations.agones.dev" error while creating fleet on AWS

[itthought]

I am getting below error while creating fleet. Please help me fixing this error
Thanks:

Error from server (InternalError): error when creating "fleet.yaml": Internal error occurred: failed calling webhook "mutations.agones.dev": Post https://agones-controller-service.agones-system.svc:443/mutate?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Please find other details as below:

• kops version
  Version 1.15.0 (git-9992b4055)

• kubectl version
  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

• Agones version: 1.1.0

• Kubernetes version : 1.15.5

• Cloud provider or hardware configuration: AWS (configured cluster using kops)

• kubectl create -f install.yaml
  serviceaccount/agones-controller created
  clusterrole.rbac.authorization.k8s.io/agones-controller created
  clusterrolebinding.rbac.authorization.k8s.io/agones-controller-access created
  clusterrolebinding.rbac.authorization.k8s.io/agones-controller:system:auth-delegator created
  rolebinding.rbac.authorization.k8s.io/agones-controller-auth-reader created
  serviceaccount/agones-sdk created
  clusterrole.rbac.authorization.k8s.io/agones-sdk created
  rolebinding.rbac.authorization.k8s.io/agones-sdk-access created
  customresourcedefinition.apiextensions.k8s.io/fleets.agones.dev created
  customresourcedefinition.apiextensions.k8s.io/fleetautoscalers.autoscaling.agones.dev created
  customresourcedefinition.apiextensions.k8s.io/gameservers.agones.dev created
  customresourcedefinition.apiextensions.k8s.io/gameserverallocationpolicies.multicluster.agones.dev created
  customresourcedefinition.apiextensions.k8s.io/gameserversets.agones.dev created
  service/agones-controller-service created
  service/agones-allocator created
  deployment.apps/agones-allocator created
  clusterrole.rbac.authorization.k8s.io/agones-allocator created
  serviceaccount/agones-allocator created
  clusterrolebinding.rbac.authorization.k8s.io/agones-allocator created
  secret/allocator-client-ca created
  secret/allocator-tls created
  secret/allocator-tls-ca created
  deployment.apps/agones-controller created
  deployment.apps/agones-ping created
  service/agones-ping-http-service created
  service/agones-ping-udp-service created
  apiservice.apiregistration.k8s.io/v1.allocation.agones.dev created
  validatingwebhookconfiguration.admissionregistration.k8s.io/agones-validation-webhook created
  mutatingwebhookconfiguration.admissionregistration.k8s.io/agones-mutation-webhook created
  secret/agones-manual-cert created
  priorityclass.scheduling.k8s.io/agones-system created

• kubectl create -f fleet.yaml
  Error from server (InternalError): error when creating "fleet.yaml": Internal error occurred: failed calling webhook "mutations.agones.dev": Post https://agones-controller-service.agones-system.svc:443/mutate?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Thanks

[aLekSer]

First of all Agones was not tested on version of Kubernetes above 1.12:

Warning
Later versions of Kubernetes may work, but this project is tested against 1.12, and is therefore the supported version. Agones will update its support to n-1 version of what is available across all major cloud providers - GKE, EKS and AKS

You can follow instructions here, they use 1.12 version:
https://agones.dev/site/docs/installation/#create-eks-cluster

[itthought]

thanks aLekSer, I will try on k8s version 1.12

[roberthbailey]

A few other questions:

• how large is your cluster?
• can you verify that the pods in the agones-system namespace are running?
• can you manually reach the agones controller service (from inside the cluster and also from the master node)?

[itthought]

@aLekSer I have tried on k8s version 1.12.0 and 1.12.10 still getting same error

@roberthbailey
I have running 1 master and 2 nodes

• kubectl get nodes
  NAME STATUS ROLES AGE VERSION
  ip-175-31-2-109.ap-south-1.compute.internal Ready node 16m v1.12.0
  ip-175-31-34-180.ap-south-1.compute.internal Ready master 17m v1.12.0
  ip-175-31-40-168.ap-south-1.compute.internal Ready node 16m v1.12.0

• kubectl get pods --all-namespaces
  NAMESPACE NAME READY STATUS RESTARTS AGE
  agones-system agones-allocator-7b5dd87bb9-7sxgv 1/1 Running 0 13m
  agones-system agones-allocator-7b5dd87bb9-bz79c 1/1 Running 0 13m
  agones-system agones-allocator-7b5dd87bb9-hxqtj 1/1 Running 0 13m
  agones-system agones-controller-76c7474f47-nntw2 1/1 Running 0 13m
  agones-system agones-ping-7bbf8c7584-9z7n8 1/1 Running 0 13m
  agones-system agones-ping-7bbf8c7584-cpd7r 1/1 Running 0 13m
  kube-system aws-node-7hp86 1/1 Running 0 5m44s
  kube-system aws-node-qjb5m 1/1 Running 0 5m44s
  kube-system aws-node-zgrsc 1/1 Running 0 5m44s
  kube-system dns-controller-6fcff4d96-8q9d9 1/1 Running 0 18m
  kube-system etcd-manager-events-ip-175-31-34-180.ap-south-1.compute.internal 1/1 Running 0 17m
  kube-system etcd-manager-main-ip-175-31-34-180.ap-south-1.compute.internal 1/1 Running 0 17m
  kube-system kube-apiserver-ip-175-31-34-180.ap-south-1.compute.internal 1/1 Running 2 17m
  kube-system kube-controller-manager-ip-175-31-34-180.ap-south-1.compute.internal 1/1 Running 0 17m
  kube-system kube-dns-68bcf5b6bf-72sm9 3/3 Running 0 16m
  kube-system kube-dns-68bcf5b6bf-xlfzg 3/3 Running 0 18m
  kube-system kube-dns-autoscaler-867b9fd49d-7jqv6 1/1 Running 0 18m
  kube-system kube-proxy-ip-175-31-2-109.ap-south-1.compute.internal 1/1 Running 0 16m
  kube-system kube-proxy-ip-175-31-34-180.ap-south-1.compute.internal 1/1 Running 0 18m
  kube-system kube-proxy-ip-175-31-40-168.ap-south-1.compute.internal 1/1 Running 0 16m
  kube-system kube-scheduler-ip-175-31-34-180.ap-south-1.compute.internal 1/1 Running 0 17m

• I am able to check log on agones controller service from cluster as well as from master node
  kubectl logs agones-controller-76c7474f47-nntw2 -n agones-system
  {"filename":"/home/agones/logs/agones-controller-20191128_064851.log","message":"logging to file","numbackups":99,"severity":"info","source":"main","time":"2019-11-28T06:48:51.477873412Z"}
  {"ctlConf":{"MinPort":7000,"MaxPort":8000,"SidecarImage":"gcr.io/agones-images/agones-sdk:1.1.0","SidecarCPURequest":"30m","SidecarCPULimit":"0","SdkServiceAccount":"agones-sdk","AlwaysPullSidecar":false,"PrometheusMetrics":true,"Stackdriver":false,"KeyFile":"/home/agones/certs/server.key","CertFile":"/home/agones/certs/server.crt","KubeConfig":"","GCPProjectID":"","NumWorkers":100,"APIServerSustainedQPS":400,"APIServerBurstQPS":500,"LogDir":"/home/agones/logs","LogSizeLimitMB":10000},"message":"starting gameServer operator...","severity":"info","source":"main","time":"2019-11-28T06:48:51.477921824Z","version":"1.1.0"}
  ERROR: logging before flag.Parse: W1128 06:48:51.478356 1 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  {"maxPort":8000,"message":"Starting","minPort":7000,"severity":"info","source":"gameservers.PortAllocator","time":"2019-11-28T06:48:51.480597067Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"GameServer"},"message":"Added webhook handler","op":"CREATE","path":"/mutate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.481085646Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"GameServer"},"message":"Added webhook handler","op":"CREATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.481169699Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"GameServerSet"},"message":"Added webhook handler","op":"CREATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.481809419Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"GameServerSet"},"message":"Added webhook handler","op":"UPDATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.481849412Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"Fleet"},"message":"Added webhook handler","op":"CREATE","path":"/mutate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.482082331Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"Fleet"},"message":"Added webhook handler","op":"CREATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.482117293Z"}
  {"groupKind":{"Group":"agones.dev","Kind":"Fleet"},"message":"Added webhook handler","op":"UPDATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.48219905Z"}
  {"groupKind":{"Group":"autoscaling.agones.dev","Kind":"FleetAutoscaler"},"message":"Added webhook handler","op":"CREATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.482480669Z"}
  {"groupKind":{"Group":"autoscaling.agones.dev","Kind":"FleetAutoscaler"},"message":"Added webhook handler","op":"UPDATE","path":"/validate","severity":"info","source":"webhooks.WebHook","time":"2019-11-28T06:48:51.482514529Z"}
  {"message":"Starting http server...","severity":"info","source":"main","time":"2019-11-28T06:48:51.482766803Z"}
  {"message":"Wait for cache sync","severity":"info","source":"metrics.Controller","time":"2019-11-28T06:48:51.483718314Z"}
  {"message":"https server started","server":{"Mux":{}},"severity":"info","source":"https.Server","time":"2019-11-28T06:48:51.483849401Z"}
  {"message":"Running","severity":"info","source":"gameservers.PerNodeCounter","time":"2019-11-28T06:48:51.484404519Z"}
  {"message":"Wait for Allocator cache sync","severity":"info","source":"gameserverallocations.Allocator","time":"2019-11-28T06:48:51.485122356Z"}
  {"crd":"fleetautoscalers.autoscaling.agones.dev","message":"custom resource definition established","severity":"info","source":"fleetautoscalers.Controller","time":"2019-11-28T06:48:51.563533549Z"}
  {"message":"Wait for cache sync","severity":"info","source":"fleetautoscalers.Controller","time":"2019-11-28T06:48:51.563674582Z"}
  {"crd":"gameserversets.agones.dev","message":"custom resource definition established","severity":"info","source":"gameserversets.Controller","time":"2019-11-28T06:48:51.563878101Z"}
  {"message":"Wait for cache sync","severity":"info","source":"gameserversets.Controller","time":"2019-11-28T06:48:51.563918411Z"}
  {"crd":"fleets.agones.dev","message":"custom resource definition established","severity":"info","source":"fleets.Controller","time":"2019-11-28T06:48:51.565098783Z"}
  {"crd":"gameservers.agones.dev","message":"custom resource definition established","severity":"info","source":"gameservers.Controller","time":"2019-11-28T06:48:51.56603449Z"}
  {"message":"Wait for cache sync","severity":"info","source":"gameservers.Controller","time":"2019-11-28T06:48:51.566130263Z"}
  {"message":"Wait for cache sync","severity":"info","source":"fleets.Controller","time":"2019-11-28T06:48:51.566194143Z"}
  {"message":"Wait for ReadyGameServerCache cache sync","severity":"info","source":"gameserverallocations.ReadyGameServerCache","time":"2019-11-28T06:48:51.585330108Z"}
  {"message":"Starting workers...","queue":"autoscaling.agones.dev.FleetAutoscalerController","severity":"info","source":"fleetautoscalers.Controller","time":"2019-11-28T06:48:51.664788127Z","workers":100}
  {"message":"Starting workers...","queue":"agones.dev.FleetController","severity":"info","source":"fleets.Controller","time":"2019-11-28T06:48:51.666460248Z","workers":100}
  {"message":"Starting workers...","queue":"agones.dev.GameServerSetController","severity":"info","source":"gameserversets.Controller","time":"2019-11-28T06:48:51.666469405Z","workers":100}
  {"message":"Running","severity":"info","source":"gameservers.PortAllocator","time":"2019-11-28T06:48:51.666478706Z"}
  {"groupversion":"allocation.agones.dev/v1","message":"Adding Discovery Handler","pattern":"/apis/allocation.agones.dev/v1","severity":"info","source":"apiserver.APIServer","time":"2019-11-28T06:48:51.685587307Z"}
  {"groupversion":"allocation.agones.dev/v1","message":"Adding Resource Handler","pattern":"/apis/allocation.agones.dev/v1/namespaces/","severity":"info","source":"apiserver.APIServer","time":"2019-11-28T06:48:51.685730258Z"}
  {"apiresource":{"name":"gameserverallocations","singularName":"gameserverallocation","namespaced":true,"kind":"GameServerAllocation","verbs":["create"],"shortNames":["gsa"]},"groupversion":"allocation.agones.dev/v1","message":"Adding APIResource","severity":"info","source":"apiserver.APIServer","time":"2019-11-28T06:48:51.685873044Z"}
  {"message":"Starting workers...","queue":"agones.dev.GameServerUpdateController","severity":"info","source":"gameserverallocations.ReadyGameServerCache","time":"2019-11-28T06:48:51.685603874Z","workers":1}
  {"message":"Resetting Port Allocation","severity":"info","source":"gameservers.PortAllocator","time":"2019-11-28T06:48:51.767249397Z"}
  {"message":"Starting workers...","queue":"agones.dev.GameServerControllerDeletion","severity":"info","source":"gameservers.Controller","subqueue":"deletion","time":"2019-11-28T06:48:51.767551371Z","workers":100}
  {"message":"Starting workers...","queue":"agones.dev.GameServerController","severity":"info","source":"gameservers.Controller","time":"2019-11-28T06:48:51.767597864Z","workers":100}
  {"message":"Starting workers...","queue":"agones.dev.GameServerControllerCreation","severity":"info","source":"gameservers.Controller","subqueue":"creation","time":"2019-11-28T06:48:51.767711005Z","workers":100}
  {"message":"Wait for cache sync","severity":"info","source":"gameservers.HealthController","time":"2019-11-28T06:48:51.768146834Z"}
  {"message":"Starting workers...","queue":"agones.dev.HealthController","severity":"info","source":"*gameservers.HealthController","time":"2019-11-28T06:48:51.881747539Z","workers":1}

[itthought]

I have figure out the issue. It was due to AWS VPC and security group policy. Thanks @aLekSer and @roberthbailey for your help

[markmandel]

@itthought is this something we should document somewhere? Maybe the AWS installation docs?

[itthought]

The issue was related to my VPC and security group which I have created for k8s cluster. I was creating kubernetes using kops with existing VPC and security group. Need to make sure following port should be opened between master node security group and node security group

Master node security group Inbound policy

• Type | Protocol I Port | Range Source
  
  
  
  • Custom TCP Rule | TCP | 4003 - 65535 | (nodes security group)
  
  
  • Custom TCP Rule | TCP | 2382 - 4000 | (nodes security group)
  
  
  • All traffic | All | All | (masters security group)
  
  
  • Custom UDP Rule | UDP | 1 - 65535 | (nodes security group)
  
  
  • Custom TCP Rule | TCP | 1 - 2379 | (nodes security group)
  
  

Node security group Inbound policy

• Type | Protocol | Port | Range Source
• All traffic | All | All | (nodes security group)
• All traffic | All | All | (masters security group)