Agones: Add API for exposing fleet status to meta-services

[adamhosier]

This FR proposes the introduction of a new API for exposing information about fleets for consumption by other services that surround a game.

As an example, we may like to have different versions of gameservers deployed (one for PC, PS4, XBOX, Beta, etc.), and have our matchmakers automatically become aware of these versions so they can allocate players into the correct gameservers. We may also want our scalers to be aware of the current allocation state of the fleets to make smarter decisions about scaling out/in our k8s clusters.

We can get this information directly from the k8s API, however it would be more convenient for us to have this exposed over gRPC (behind mTLS), much like the allocation API. This is particularly useful in the multi-cluster setup where authentication into k8s APIs is non-trivial.

We propose introducing a new service, working name agones-informer or agones-discover, which serves the following gRPC service (exact API to be decided):

service InformationService {
    rpc Inform(InformRequest) returns (InformResponse);
}

message InformRequest {}
message InformResponse {
    repeated FleetInformation fleetInformation = 1;
}

message FleetInformation {
    string clusterName = 1;
    string fleetName = 2;
    repeated Label labels = 3;
    int32 allocatedGameservers = 4;
    int32 readyGameservers = 5;
    ...
}

message Label {
    string key = 1;
    string value = 2;
}

Much like the allocation service, this could run in single-cluster mode (where Inform would list fleets installed in the current cluster) or multi-cluster mode (where Inform would list fleets installed in all known clusters by calling down to informers in child clusters). The multi-cluster mode could make use of a new CRD defining cluster topology, or hi-jack the GameServerAllocationPolicy CRD to get this information.

Would you be receptive to this as a concept? Does the approach generally make sense or would there be a nicer way to go about it? If so, we'd be happy to contribute to a more in-depth design and implementation.

[markmandel]

@danieloliveira079 @ilkercelikyilmaz - I think you will find this conversation interesting.

Thanks @adamhosier ! I feel like I understand the impetus for having functionality like this, but I have some concerns and questions :smile:

I'm generally wary of re-implementing the K8s API where possible, it feels like a reinvention of the wheel - but I also understand that sometimes you want a holistic view of what is happening across your entire set of Agones Clusters and Fleets.

I'll also be honest, I'm also very wary of polling operations to determine current state of a system. I'd much rather see a watch/push/subscription type model where possible -- I've seen too many systems be brought down by constant polling.

So this is where it feels to me like an extenion on @danieloliveira079 's great project https://github.com/Octops/agones-event-broadcaster to include Fleet information might be better?

Setting up the infra for multi-cluster coordination is quite extensive and complex, but it's necessary for a imperative command like Allocation. For a system that's only informational, using something that is distributed and decentralised seems far less error prone and easier to manage, as it can be done by installing a single agent on each cluster and have it connect to something like a message bus, without the need to handle security, fail-over, communication and management across multiple clusters.

I'm also curious -- where does your Fleet information come from? Are you doing some kind of CI/CD/GitOps/Terraform/Google Cloud Game Servers, that could also be queried for this information?

Would also love to hear the thoughts of others as well.

[danieloliveira079]

@markmandel Definitely, I will be checking that possibility in the forthcoming days. I can update you all as soon as I get something more concrete. Extending the project to resources of type Fleet is on my roadmap.

@adamhosier I will connect on Slack to get more information about your use case. I've some ideas about this kind scenario.

[danieloliveira079]

@markmandel @adamhosier just a quick update. I've added support for v1.Fleet on the Broadcaster. Moreover, it is possible to watch any type of resource from now on. But let's focus on Fleets :)

The sintaxe has changed and the code below demonstrates how one can create a broadcaster that watches for both v1.GameServer and v1.Fleet resources.

Code: https://github.com/Octops/agones-event-broadcaster/blob/master/cmd/root.go#L64

bc := broadcaster.New(clientConf, broker, duration)
if err := bc.WithWatcherFor(&v1.Fleet{}).WithWatcherFor(&v1.GameServer{}).Build(); err != nil {
  logrus.WithError(err).Fatal("error creating broadcaster")
}

You can find an example of how to publish Fleets events to PubSub on https://github.com/Octops/agones-event-broadcaster/blob/master/examples/pubsub/fleet/main.go

Let me know if you need any help or would like to discuss any further details.

[adamhosier]

@markmandel

I'll also be honest, I'm also very wary of polling operations to determine current state of a system.

I totally understand. Our use case makes this seem more like a query rather than a poll, but we can get that same information from a push-based model without encouraging others to poll.

So this is where it feels to me like an extenion on @danieloliveira079 's great project https://github.com/Octops/agones-event-broadcaster to include Fleet information might be better?

I had not come across this project, this looks very interesting! We'll take a look and sync up with @danieloliveira079.

I'm also curious -- where does your Fleet information come from? Are you doing some kind of CI/CD/GitOps/Terraform/Google Cloud Game Servers, that could also be queried for this information?

We had considered this, however we weren't keen on coupling the operation of our systems to the method of deploying those systems.

I've added support for v1.Fleet on the Broadcaster. Moreover, it is possible to watch any type of resource from now on. But let's focus on Fleets :)

Exciting! We'll be giving this a try over the coming days so expect to hear from me shortly.

[adamhosier]

Quick update - we're happy with the architecture of agones-event-broadcaster and agree that a contribution like this makes more sense in that project than it does here. We'll move further conversation to that repo when we're ready to explore it further.