Agones: Fleet Rolling Update doesn't seem to be rolling

[markmandel]

What happened:

When updating a Fleet, it doesn't seem to do a rolling update, even when using a Rolling Update strategy. It seems like it deletes the entire old GameServerSet, then creates a new one at the original size.

What you expected to happen:

We should get a gradual rollout of shutting down a set of gameservers, while spinning up another set. Then it should wait for that to complete before moving to the next set.

How to reproduce it (as minimally and precisely as possible):

root@7551c789cd4f:/go/src/agones.dev/agones# kubectl apply -f ./examples/simple-udp/fleet.yaml
fleet.stable.agones.dev/simple-udp created
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl scale fleet simple-udp --replicas=50
fleet.stable.agones.dev/simple-udp scaled
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get flt
NAME         SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY     AGE
simple-udp   Packed       50        50        0           50        27s
root@7551c789cd4f:/go/src/agones.dev/agones#
root@7551c789cd4f:/go/src/agones.dev/agones# kubectl edit flt simple-udp

# change the container port to a different value, that will force an update

root@7551c789cd4f:/go/src/agones.dev/agones# kubectl get gs | grep -v Shut | wc -l
51

You can see that 50 gets created straight away. If you kubectl get gs you can also see all the other gameservers are moved to shutdown.

Anything else we need to know?:

Not sure if it shows anything, but this is what the graph looks like

There is one interesting thing. if we kubectl describe the fleet, the events look correct. Maybe it's moving too fast?

Doesn't look like we are capturing the rolling nature in an e2e test - just the final result (that everything has changed).

root@7551c789cd4f:/go/src/agones.dev/agones# kubectl describe fleet simple-udp
Name:         simple-udp
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"stable.agones.dev/v1alpha1","kind":"Fleet","metadata":{"annotations":{},"name":"simple-udp","namespace":"default"},"spec":{"replicas":2,...
              stable.agones.dev/sdk-version=0.10.0
API Version:  stable.agones.dev/v1alpha1
Kind:         Fleet
Metadata:
  Creation Timestamp:  2019-05-29T22:38:32Z
  Generation:          3
  Resource Version:    4150979
  Self Link:           /apis/stable.agones.dev/v1alpha1/namespaces/default/fleets/simple-udp
  UID:                 7c9ec43a-8262-11e9-83d0-42010aa80085
Spec:
  Replicas:    50
  Scheduling:  Packed
  Strategy:
    Rolling Update:
      Max Surge:        25%
      Max Unavailable:  25%
    Type:               RollingUpdate
  Template:
    Metadata:
      Creation Timestamp:  <nil>
    Spec:
      Health:
      Ports:
        Container Port:  8000
        Name:            default
      Template:
        Metadata:
          Creation Timestamp:  <nil>
        Spec:
          Containers:
            Image:  gcr.io/agones-images/udp-server:0.9
            Name:   simple-udp
            Resources:
              Limits:
                Cpu:     20m
                Memory:  64Mi
              Requests:
                Cpu:     20m
                Memory:  64Mi
Status:
  Allocated Replicas:  0
  Ready Replicas:      50
  Replicas:            50
Events:
  Type    Reason                 Age   From              Message
  ----    ------                 ----  ----              -------
  Normal  CreatingGameServerSet  5m    fleet-controller  Created GameServerSet simple-udp-vx4qx
  Normal  ScalingGameServerSet   5m    fleet-controller  Scaling active GameServerSet simple-udp-vx4qx from 2 to 50
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling inactive GameServerSet simple-udp-vx4qx from 50 to 37
  Normal  CreatingGameServerSet  4m    fleet-controller  Created GameServerSet simple-udp-m96x4
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling inactive GameServerSet simple-udp-vx4qx from 37 to 24
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling active GameServerSet simple-udp-m96x4 from 13 to 26
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling inactive GameServerSet simple-udp-vx4qx from 24 to 11
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling active GameServerSet simple-udp-m96x4 from 26 to 39
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling inactive GameServerSet simple-udp-vx4qx from 11 to 0
  Normal  ScalingGameServerSet   4m    fleet-controller  Scaling active GameServerSet simple-udp-m96x4 from 39 to 50
  Normal  DeletingGameServerSet  4m    fleet-controller  Deleting inactive GameServerSet simple-udp-vx4qx

Environment:

• Agones version: 0.10.0
• Kubernetes version (use kubectl version): 1.11.9
• Cloud provider or hardware configuration: GKE
• Install method (yaml/helm): helm
• Troubleshooting guide log(s): N/A
• Others:

[aLekSer]

I think we should create empty new GS Set first and then it would be scaled in parallel with rest shutting down using this function applyDeploymentStrategy. So Spec.Replicas should be 0 to start with.
https://github.com/GoogleCloudPlatform/agones/blob/e6d91905ccb400b534aca70d2927836ca4043cee/pkg/fleets/controller.go#L306

[markmandel]

Which.. I think is what happens? Or at least, is what is supposed to. We create a second gsSet, then slowly scale that up while we slowly scale down the other one.

[lftakakura]

Any update on this?

[aLekSer]

Hello @lftakakura I am working on a fix. Will send draft PR in a moment.