Agones: RollingUpdate shuts down all servers at once

Created on 28 Oct 2019  路  14Comments  路  Source: googleforgames/agones

What happened: I adjusted RollingUpdate settings to replace each Ready GameServer pod one by one using following configuration

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1

But every time I deployed Fleet configuration with this setting it would instantly shut down all Ready servers and replace them with new version.

1) Deployed my yaml and ran kubectl get gs got following 

NAME                             STATE       ADDRESS      PORT   NODE                       AGE
lobby-server-odd-2vmld-2m6h9        Shutdown    34.89.86.    7442   gke-qa-default-pool-3943   39s
lobby-server-odd-2vmld-6mlms        Shutdown    35.246.21.   7812   gke-qa-default-pool-rkw7   39s
lobby-server-odd-2vmld-9n8z9        Shutdown    35.235.55.   7382   gke-qa-default-pool-hjgr   1m
lobby-server-odd-2vmld-c6wzw        Shutdown    35.235.55.   7104   gke-qa-default-pool-hjgr   39s
lobby-server-odd-2vmld-pjttg        Shutdown    35.235.55.   7294   gke-qa-default-pool-hjgr   39s
lobby-server-odd-2vmld-pxqqs        Shutdown    35.235.53.   7139   gke-qa-default-pool-1wwf   39s
lobby-server-odd-2vmld-rb755        Shutdown    34.89.86.    7912   gke-qa-default-pool-3943   39s
lobby-server-odd-2vmld-xqbdl        Shutdown    35.242.175   7098   gke-qa-default-pool-bhwl   39s
lobby-server-odd-2vmld-zrl4n        Shutdown    35.246.21.   7367   gke-qa-default-pool-rkw7   39s
lobby-server-odd-2vmld-zw448        Ready       35.246.21.   7027   gke-qa-default-pool-rkw7   39s

2) Refreshing

NAME                             STATE       ADDRESS      PORT   NODE                       AGE
lobby-server-odd-2vmld-zw448        Ready       35.246.21.   7027   gke-qa-default-pool-rkw7   57s

3) Refreshing

NAME                             STATE       ADDRESS      PORT   NODE                       AGE
lobby-server-odd-2vmld-666zz        Scheduled   34.89.86.    7053   gke-qa-default-pool-3943   4s
lobby-server-odd-2vmld-8fjvp        Scheduled   35.235.53.   7114   gke-qa-default-pool-1wwf   4s
lobby-server-odd-2vmld-chks8        Scheduled   35.242.175   7037   gke-qa-default-pool-bhwl   4s
lobby-server-odd-2vmld-mbqtd        Scheduled   35.235.55.   7255   gke-qa-default-pool-hjgr   4s
lobby-server-odd-2vmld-r9g4p        Scheduled   35.246.21.   7302   gke-qa-default-pool-rkw7   4s
lobby-server-odd-2vmld-t99p2        Scheduled   34.89.86.    7072   gke-qa-default-pool-3943   4s
lobby-server-odd-2vmld-vb9q5        Scheduled   35.235.55.   7561   gke-qa-default-pool-hjgr   4s
lobby-server-odd-2vmld-vkcz6        Scheduled   35.246.21.   7063   gke-qa-default-pool-rkw7   4s
lobby-server-odd-2vmld-x58sb        Scheduled   35.235.55.   7057   gke-qa-default-pool-hjgr   4s
lobby-server-odd-2vmld-zw448        Ready       35.246.21.   7027   gke-qa-default-pool-rkw7   1m

4) Refreshing

NAME                             STATE       ADDRESS      PORT   NODE                       AGE
lobby-server-odd-2vmld-666zz        Ready       34.89.86.    7053   gke-qa-default-pool-3943   11s
lobby-server-odd-2vmld-8fjvp        Ready       35.235.53.   7114   gke-qa-default-pool-1wwf   11s
lobby-server-odd-2vmld-chks8        Ready       35.242.175   7037   gke-qa-default-pool-bhwl   11s
lobby-server-odd-2vmld-mbqtd        Ready       35.235.55.   7255   gke-qa-default-pool-hjgr   11s
lobby-server-odd-2vmld-r9g4p        Ready       35.246.21.   7302   gke-qa-default-pool-rkw7   11s
lobby-server-odd-2vmld-t99p2        Ready       34.89.86.    7072   gke-qa-default-pool-3943   11s
lobby-server-odd-2vmld-vb9q5        Ready       35.235.55.   7561   gke-qa-default-pool-hjgr   11s
lobby-server-odd-2vmld-vkcz6        Ready       35.246.21.   7063   gke-qa-default-pool-rkw7   11s
lobby-server-odd-2vmld-x58sb        Ready       35.235.55.   7057   gke-qa-default-pool-hjgr   11s
lobby-server-odd-2vmld-zw448        Ready       35.246.21.   7027   gke-qa-default-pool-rkw7   1m

What you expected to happen: I expected them to be shut down and replaced one by one instead of all at once

How to reproduce it (as minimally and precisely as possible):
1) Set RollingUpdate configuration to your fleet

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1

And deploy it a few times, I could reproduce it using fleet-example.

Environment:

  • Agones version: 1.0.0
  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-dispatcher", GitCommit:"2e298c7e992f83f47af60cf4830b11c7370f6668", GitTreeState:"clean", BuildDate:"2019-09-19T22:20:12Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.10-gke.5", GitCommit:"e327dc5fe0418e41fa9f7c153ec14da34b61fe85", GitTreeState:"clean", BuildDate:"2019-08-23T23:09:28Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: GCP
  • Install method (yaml/helm): helm
kinbug areuser-experience
Source
picture KamiMay

All 14 comments

That is what I was fixing a while ago. Need to take a look.
/assign alekser

aLekSer picture aLekSer on 29 Oct 2019

@KamiMay thanks for the ticket, can you please clarify?
I can not reproduce it, something is missing in the steps to reproduce. I have changed examples/fleet.yaml to have maxSurge as 1. And RollingUpdate strategy triggers Shutdown in 1 GameServer and Scheduled state in one other GS.
I perform

 kubectl apply -f ./examples/fleet.yaml

And changing GameServer Spec each time - say updating sdkServer.logLevel parameter.
Can you please also provide output of the following command:

kubectl describe fleet [your-fleet-name]
aLekSer picture aLekSer on 29 Oct 2019

I assume you were using 

kubectl scale fleet fleet-example --replicas=1
kubectl scale fleet fleet-example --replicas=10
kubectl scale fleet fleet-example --replicas=1

updates of this kind as from your logs. Am I right?

aLekSer picture aLekSer on 29 Oct 2019

I think that we should not support shrinking size of a fleet with maxSurge and maxUnavailable applied. I mean going from 10, 9, 8 step by step to 1 replica.
Changing he number of Replicas in Kubernetes deployment would result in updating all Pods straight away - all unnecessary Pods would be terminated except new number of Replicas.

aLekSer picture aLekSer on 29 Oct 2019

No I am not changing replica set let me try to explain step by step.

  1. First I start with an empty agones 1.0 set up with no fleets yet spun up.
kami_may@cloudshell:~ (dev)$ kubectl get fleets
No resources found.
  1. Deploy this fleet file using kubectl apply -f command
apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: fleet-example
spec:
  replicas: 2
  scheduling: Packed
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        foo: bar
    spec:
      ports:
      - name: default
        portPolicy: Dynamic
        containerPort: 26000
      health:
        initialDelaySeconds: 30
        periodSeconds: 60
      template:
        spec:
          containers:
          - name: simple-udp
            image: gcr.io/agones-images/udp-server:0.15
---
apiVersion: "autoscaling.agones.dev/v1"
kind: FleetAutoscaler
metadata:
  name: server-autoscaler
spec:
  fleetName: fleet-example
  policy:
    type: Buffer
    buffer:
      bufferSize: 5
      minReplicas: 5
      maxReplicas: 10
  1. Wait for the fleet to be deployed and preview if game servers are there 
kami_may@cloudshell:~ (dev)$ kubectl get gs
NAME                        STATE       ADDRESS      PORT   NODE                                AGE
fleet-example-lds6j-5l29k   Ready       35.235.55.   7372   gke-qa-default-pool-21f0c334-x7bq   39s
fleet-example-lds6j-cwcqs   Ready       35.230.143   7055   gke-qa-default-pool-21f0c334-f98k   40s
fleet-example-lds6j-hhrgv   Ready       35.235.55.   7868   gke-qa-default-pool-21f0c334-x7bq   39s
fleet-example-lds6j-qqdbv   Ready       35.230.143   7175   gke-qa-default-pool-21f0c334-f98k   40s
fleet-example-lds6j-rb7nr   Ready       35.235.53.   7657   gke-qa-default-pool-21f0c334-1wwf   39s
  1. I edit fleet file and change foo: bar label to foo: bar2 and kubectl apply -f edited file
kami_may@cloudshell:~ (dev)$ kubectl apply -f fleet.yml
fleet.agones.dev/fleet-example configured
fleetautoscaler.autoscaling.agones.dev/server-autoscaler unchanged
  1. I preview game server status straight after deploy 
kami_may@cloudshell:~ (dev)$ kubectl get gs
NAME                        STATE      ADDRESS      PORT   NODE                                AGE
fleet-example-wcs7f-6bc29   Ready      35.235.55.   7576   gke-qa-default-pool-21f0c334-x7bq   1m
fleet-example-wcs7f-88zzt   Ready      35.235.55.   7193   gke-qa-default-pool-21f0c334-x7bq   2m
fleet-example-wcs7f-gbd9c   Shutdown   35.235.55.   7218   gke-qa-default-pool-21f0c334-x7bq   1m
fleet-example-wcs7f-r8dvx   Shutdown   35.235.55.   7814   gke-qa-default-pool-21f0c334-x7bq   1m
fleet-example-wcs7f-xdvmv   Shutdown   35.235.55.   7632   gke-qa-default-pool-21f0c334-x7bq   2m

And I can see more than one instance shutting down all at once.

  1. I ran kubectl describe fleet fleet-example and these were events for my deploy 
Events:
  Type    Reason                 Age                    From              Message
  ----    ------                 ----                   ----              -------
  Normal  ScalingGameServerSet   114s                   fleet-controller  Scaling active GameServerSet fleet-example-wcs7f from 5 to 2
  Normal  ScalingGameServerSet   101s                   fleet-controller  Scaling active GameServerSet fleet-example-wcs7f from 2 to 5
KamiMay picture KamiMay on 29 Oct 2019
👍1

Thank you, now I am able to reproduce an issue.
RollingUpdate is not working properly when it used with a FleetAutoscaler.
Ideas of the root cause of the bug is that we have Replicas parameter in a Fleet Spec as 2 and Buffer in FAS as 5, so when we are updating the Fleet Label it is allowed to be decreased to 2 - 1 Max Unavailable which is up to 1. So we should consider more enhanced logic, which take into account not Fleet Spec but Status field of a Fleet.
I am thinking of what is the priority between FleetAutoscaler and Fleet Spec target replicas count.

I have got even more events for that steps:

Events:
  Type    Reason                 Age    From              Message
  ----    ------                 ----   ----              -------
  Normal  CreatingGameServerSet  3m43s  fleet-controller  Created GameServerSet bug-in-fleet-n5mxc
  Normal  ScalingGameServerSet   3m19s  fleet-controller  Scaling active GameServerSet bug-in-fleet-n5mxc from 2 to 5                                                                                    
  Normal  ScalingGameServerSet   42s    fleet-controller  Scaling inactive GameServerSet bug-in-fleet-n5mxc from 5 to 4                                                                                  
  Normal  CreatingGameServerSet  42s    fleet-controller  Created GameServerSet bug-in-fleet-8zjgh
  Normal  ScalingGameServerSet   40s    fleet-controller  Scaling inactive GameServerSet bug-in-fleet-n5mxc from 4 to 3                                                                                  
  Normal  ScalingGameServerSet   38s    fleet-controller  Scaling inactive GameServerSet bug-in-fleet-n5mxc from 3 to 2                                                                                  
  Normal  ScalingGameServerSet   37s    fleet-controller  Scaling active GameServerSet bug-in-fleet-8zjgh from 0 to 1                                                                                    
  Normal  ScalingGameServerSet   34s    fleet-controller  Scaling inactive GameServerSet bug-in-fleet-n5mxc from 2 to 1                                                                                  
  Normal  ScalingGameServerSet   33s    fleet-controller  Scaling active GameServerSet bug-in-fleet-8zjgh from 1 to 2                                                                                    
  Normal  ScalingGameServerSet   32s    fleet-controller  Scaling inactive GameServerSet bug-in-fleet-n5mxc from 1 to 0                                                                                  
  Normal  DeletingGameServerSet  24s    fleet-controller  Deleting inactive GameServerSet bug-in-fleet-n5mxc                                                                                             
  Normal  ScalingGameServerSet   19s    fleet-controller  Scaling active GameServerSet bug-in-fleet-8zjgh from 2 to 5
aLekSer picture aLekSer on 30 Oct 2019
👍1

Seems to be linked to https://github.com/kubernetes/kubernetes/issues/25238
Workaround was to remove replicas: 2 from the file used for kubectl apply -f.

aLekSer picture aLekSer on 30 Oct 2019
👍1

Tested the scenario one more time, after this revert PR merged:
https://github.com/googleforgames/agones/pull/1226
It is working fine if I follow the steps above with no replicas: 2 line. Example yaml config follows.

aLekSer picture aLekSer on 11 Dec 2019

The only noticed thing is when I remove the replicas count from 2 to nothing. It would shutdown all GSs at once, not maxUnavailable, but this could be a separate issue.

aLekSer picture aLekSer on 11 Dec 2019

So to avoid the issue mentioned in the bug, currently we need to use in step 2 this yaml config without replicas: 2 line:

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: fleet-example
spec:
  scheduling: Packed
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        foo: bar
    spec:
      ports:
      - name: default
        portPolicy: Dynamic
        containerPort: 26000
      health:
        initialDelaySeconds: 30
        periodSeconds: 60
      template:
        spec:
          containers:
          - name: simple-udp
            image: gcr.io/agones-images/udp-server:0.16
---
apiVersion: "autoscaling.agones.dev/v1"
kind: FleetAutoscaler
metadata:
  name: server-autoscaler
spec:
  fleetName: fleet-example
  policy:
    type: Buffer
    buffer:
      bufferSize: 5
      minReplicas: 5
      maxReplicas: 10
aLekSer picture aLekSer on 11 Dec 2019

In the above scenario new image should be used 

            image: gcr.io/agones-images/udp-server:0.16

Otherwise all GSs stay in a Scheduled state for more than a minute, and never becomes ready. They got deleted as Unhealthy after some time.

aLekSer picture aLekSer on 11 Dec 2019

k describe fleet fleet-example results:

  Normal  ScalingGameServerSet   11m                 fleet-controller  Scaling active GameServerSet fleet-example-wzs67 from 5 to 0                                                                          
  Normal  ScalingGameServerSet   11m                 fleet-controller  Scaling active GameServerSet fleet-example-wzs67 from 0 to 5

Should be next way (if no replicas line before first apply : 

  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling inactive GameServerSet fleet-example-kswmh from 5 to 4                                                                        
  Normal  CreatingGameServerSet  31m                 fleet-controller  Created GameServerSet fleet-example-z9pmf                                                                                             
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling inactive GameServerSet fleet-example-kswmh from 4 to 3                                                                        
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling inactive GameServerSet fleet-example-kswmh from 3 to 2                                                                        
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling active GameServerSet fleet-example-z9pmf from 0 to 1                                                                          
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling inactive GameServerSet fleet-example-kswmh from 2 to 1                                                                        
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling active GameServerSet fleet-example-z9pmf from 1 to 2                                                                          
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling inactive GameServerSet fleet-example-kswmh from 1 to 0                                                                       
  Normal  DeletingGameServerSet  31m                 fleet-controller  Deleting inactive GameServerSet fleet-example-kswmh                                                                                   
  Normal  ScalingGameServerSet   31m                 fleet-controller  Scaling active GameServerSet fleet-example-z9pmf from 2 to 5
aLekSer picture aLekSer on 11 Dec 2019

@aLekSer sounds like we should add some docs on this here: https://agones.dev/site/docs/guides/fleet-updates/ ?

markmandel picture markmandel on 11 Dec 2019

Yes, that's right. Need to update this part.

aLekSer picture aLekSer on 11 Dec 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

adamhosier picture adamhosier  路  5Comments
alexandrem picture alexandrem  路  5Comments
jmaeso picture jmaeso  路  5Comments
markmandel picture markmandel  路  4Comments
AngryDeveloper picture AngryDeveloper  路  4Comments