Agones: GameServer Fleets

[markmandel]

Design

Description

Fleets are a group of warm servers that are available to be allocated to players when needed.

In Kubernetes parlance, they are the Deployment/ReplicaSet to Pods, but for GameServers

Features

• Be able to define a Fleet, with an attached GameServerTemplate (much like a PodTemplate)
• The Fleet ensures there is always replicas number of Healthy GameServers available (assuming resources exist)
• If a GameServer becomes Unhealthy, then delete it and create it anew (we may add more options at a later date).
• If you delete a Fleet, this also deletes the backing GameServers (this should be by default in Kubernetes now anyway)
• A mechanism (TBD) to be able to get an allocated GameServer out of the pool.
  
  
  
  • If a warm server is not available, then a cold one should be started (assuming resources)
  
  
  • An allocated GameServer is moved to a Allocated state on allocation.
  
  
• If the replicas are increased in the Fleet, the number of GameServers is increased to match that number (assuming resources)
  
  
  
  • If the replicas are decreased in the Fleet, the number of GameServers is decreased to match that number.
  
  
  • GameServers that are in an Allocated state will never be deleted during the decrease
  
  
• The GameServer template is changed, then we will mimic a Deployment in that we can do either a Recreate or a RollingUpdate to switch out the waiting warm servers.
  
  
  
  • GameServers that are in an Allocated state will never be deleted during the decrease
  
  
• Validation of the configuration and changes.

Configuration

apiVersion: "stable.agon.io/v1alpha1"
kind: Fleet
metadata:
  name: "fleet-example"
spec:
  # number of GameServers
  replicas: 10
  # deployment strategy for updating the image
  # Lifted directly from https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#deploymentstrategy-v1-apps
  strategy:
    # Recreate or RollingUpdate. Default to RollingUpdate
    type: RollingUpdate
    rollingUpdateDeployment: # optional rolling update config
      maxSurge: "25%"
      maxUnavailable: "25%"
  # A GameServer template
  template:
    # Standard ObjectMeta
    metadata:
      labels:
        mylabel: myvalue
    # GameServer spec
    spec:
      containerPort: 7654
        template:
          spec:
            containers:
            - name: cpp-simple
              image: gcr.io/agon-images/cpp-simple-server:0.1

Allocation

Allocation is done through creating a FleetAllocation record via kubectl or the API.

For example:

apiVersion: "stable.agon.io/v1alpha1"
kind: FleetAllocation
metadata:
  name: "sample-allocation"
spec:
  fleetName: "fleet-example"

The returned value from creating a GameServerAllocation has the details of the allocated server (And moves the GameServer to the state Allocated.)

For example:

apiVersion: "stable.agon.io/v1alpha1"
kind: FleetAllocation
metadata:
  name: "sample-allocation"
spec:
  fleetName: "fleet-example"
status:
  gameserver:
    metadata:
      name: "allocated-game-server"
    spec:
      containerPort: 7654
      template:
        spec:
          containers:
          - name: cpp-simple
            image: gcr.io/agones-images/cpp-simple-server:0.1
    status:
      address: 192.168.99.100
      nodeName: agones
      port: 7373
      state: Allocated

• Allocations can only be created, the only way to remove an allocation (at this stage) is to delete the backing GameServer.

Work Breakdown

• [x] GameServerSet (The GameServerSet to Fleet as ReplicaSet to Deployment) (#156)
• [x] Creating a Fleet creates a GameServerSet (#174)
• [x] Create an Allocation from a fleet (#193)
• [x] Recreate update strategy (#199)
• [ ] Rolling update strategy

Out of scope

• Pre-caching of images. We'll see how testing works without this for now.

Research

• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#replicaset-v1-apps
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#deployment-v1-apps
• https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apimachinery/webhook.go
• https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apimachinery/certs.go

[markmandel]

@EricFortin - I added a section on what to do when the GameServer template changes on a Fleet. I think it should mimic what happens with a Deployment - but we didn't talk explicitly about this. WDYT?

[EricFortin]

@markmandel I think it makes sense and is what I would be expecting of something running on a kubernetes cluster. We just need to adapt it to take the new state into consideration. It should recreate/update only game servers that aren't allocated yet. Moreover, once a non-updated game servers becomes free again, it should be updated before being reallocated. At the beginning, we might solve this issue by recreating any container where games have ended but at some point, games will want persistence for their game server process.

[markmandel]

SGTM. Adjusted the above to make sure it was clear that Allocated GameServers will not get deleted in the rolling update (but we were thinking the same thing)

I had a question about your statement here:

Moreover, once a non-updated game servers becomes free again, it should be updated before being reallocated. At the beginning, we might solve this issue by recreating any container where games have ended but at some point, games will want persistence for their game server process.

My thought here was very much that when a GameServer shutdown, it gets totally deleted, and a new one would replace it. Thus agreeing with you - the update would occur when a replacement gets created in the Fleet's pool.

From my understanding - we were all working from the assumption/opinion that a GameServer will exit when it has completed a session (sounds like this also needs to be documented).

When you said _"but at some point, games will want persistence for their game server process"_ - do you meant the GameServer is going to reset it back to a zero state? rather than exit, or do you mean something else?

[EricFortin]

From my understanding - we were all working from the assumption/opinion that a GameServer will exit when it has completed a session (sounds like this also needs to be documented).

TL;DR. The assumption holds

My point was mainly that games might want to control server life cycle themselves and not tie it to a game session. It could mean having the server becoming available again to host another "session", which is a concept that is missing at the moment. Should we add it?

Having a server process die at the end of a match applies well to a FPS but not necessarily to a PvE but I guess we can still consider that the server process will exit when it is ready, or instructed to become so, to be swapped out by an updated container. It might just take a long time(hours, not weeks).

[cyriltovena]

I'm wondering if the Allocated concept is the same as currently hosting a session ?

[alexandrem]

@EricFortin We most likely want to create new game sessions in a clean environment every time and not reuse previous process.

In a PvE, we wouldn't want cheaters to leave any bad states in the engine memory or on disk.

I suppose there's a trade off to be made between good isolation and initialization time with "hot" server processes. I'd rather have better isolation. What do you think?

[markmandel]

@EricFortin Ah yes - good point (thanks for the explanation). I think when we start looking more at PvE - there are several strategies we could look at - maybe a SDK method of Deallocate() rather than Shutdown() - or maybe it's a configuration option on Shutdown() it returns to the pool rather than gets deleted. But I would defer this to a later date for the moment.

My current thought for getting an Allocated GameServer - is to essentially follow the Deployment paradigm:

• We have a GameServerAllocation CRD that you can then create a via the yaml / Kubernetes api.
• When the GameServerAllocation is created, it's the GameServerAllocation that is returned is populated with the details of the GameServer that is allocated (need 1.9 webhooks for this).
• The GameServer state is also moved to Allocated at the same time.
• GameServerAllocations can't be edited - only created and deleted (current thought being if you delete the GameServerAllocation or the GameServer they are both deleted )?

This stays nicely within the K8s paradigm. WDYT? (I can also write it up above with some examples if that helps)

[markmandel]

@alexandrem I'm more inclined as well to isolation - but I feel like we could try with better isolation (processes get killed), and see what the actual implications are with a real system, and make pragmatic decisions once we have more data (i.e. then add options to move out of Allocation if needed), rather than decide at this stage. WDYT?

[alexandrem]

It's probably too early to decide on the fate of this feature. We certainly can have this discussion about recycling terminated game session resources later.

I'd focus on keeping things really simple at this stage though. Meaning that game server allocation represents a new scheduled pod, then when players leave (or game terminated) we delete the GS pod (fleet will ensure to replace it with a fresh one).

[EricFortin]

Like I said, I am OK with using the simple approach for now. I just wanted to mention some use cases where I know it might cause some problems so we don't make an early mistake. I am all for isolation but in some cases, this goes against certain game mode where the players come and go and game "lives" in between interactions with players so recreating it anew might defeat the feature. It all depends on the level of statefulness of game servers which I do not control.

[alexandrem]

@EricFortin Those longer lived game sessions where players come and go (and remain non populated for some time) are an interesting use case.

I suppose in the end this is controlled solely by the SDK; is game terminated or not. If not terminated, then we definitely want to keep the game session in "allocated" and not touch them. Actually, they probably should have a different state so we can track those who are empty so we can possibly remove them when the fleet size is being decreased.

[markmandel]

Actually, they probably should have a different state so we can track those who are empty so we can possibly remove them when the fleet size is being decreased.

This is an interesting question - one could argue, that if the GameServer becomes empty (maybe for a certain time) this is implementation specific, and the actual game server code should just call SDK.Shutdown() when it's been empty for a while - and then the Fleet will just do what it needs to do at that point forward.

[EricFortin]

For those cases where life cycle is not totally dependant on game sessions, there will most probably be a "god" service that manages it all and can replace an instance by directing players to the new instance and ultimately instructs the old instance to shutdown. This use case also affects scale down event since even if we taint the node, it still can't be recycled if it's hosting a game server. This is out of scope for now but I think we will need a way to provide this feedback so something can react.

[markmandel]

Added some notes about how I think an Allocation should occur. These will definitely need 1.9 features.

[markmandel]

Currently working on GameserverSets (the Fleet version of ReplicaSets) on my fork over here. Will finalise, and then submit for PR before doing the Fleet work that overlays this work.

[justinliew]

I'm very interested in this, as we never did our "fleets" using Kubernetes APIs. Rather, we had a separate service that monitored our pods and did our warming up, scaling up and down, and what not. At the time I wasn't as familiar with K8s as I am now, so I may have come up with something closer to your approach if I did it again.

[markmandel]

Start of "Creating a Fleet creates a GameServerSet" first PR has started over in this repository and branch

[markmandel]

Fleet Allocation work started over here

[markmandel]

Work on doing a straight replace when updating Fleet's GameServer details, can be found over here:
https://github.com/markmandel/agones/tree/feature/fleet-replace

[markmandel]

Rolling update strategy work is here:
https://github.com/markmandel/agones/tree/feature/fleet-rolling