Agones: YAML packaging

[cyriltovena]

Looking at the big install.yaml file and all the configurations possible, I think it would help people to provide a way of packaging YAML files.

Possible variables:

• namespace
• service account names
• MIN_PORT
• MAX_PORT
• images version (for later)
• RBAC on/off ???

The default variable should reflect the current install.yaml.

We should also update the documentation to add installation step using helm on top of what we already have.

I don't think it's urgent for 0.1 but interesting to have.

/cc @rodcloutier

[markmandel]

So I wonder if this should be made a broader discussion -

• Do we use Helm
• Do we use Ksonnet
• Do we create our own thing?

Each one has it's own pros and cons.

Totally agree on the size of the install.yaml - it's starting to get unwieldy, and I keeping the two versions in sync is getting painful as well, and potentially prone to error.

Top of mind concerns for me:

• Having some kind of template system that will do variable substitution.
• Having some kind of template system that does includes (so we can break up the install.yaml)
• Some kind of scripting ability would be nice - mainly because of webhooks we store some certs and the public key in install.yaml. For production these should be unique to an install, so likely generated one time and able to be rotated via a script.

We should likely make a list of the features we are looking for in an install solution, and see what the best fitting tool is going to be.

Thoughts?

[cyriltovena]

I have no preferences, as far as it cover all your concerns it's fine by me.

[markmandel]

Sounds fair enough. Probably another concern would be how it handles upgrades of installs in a sensible way.

Sounds like time to start a spreadsheet and do some research!

[markmandel]

Regarding "roll your own" one option I was considering was a agonesadm binary (for win/lin/mac) that would be use local Kube credentials to install Agones on a cluster, and would be bundled with the release archives. Would have both a:

agonesadm install - installs Agones
agonesadm yaml - output the yaml for installation, if you want to do it manually

And it could then have a variety of flags for configuring various options, which would work across install and yaml.

I thought this was kind of interesting, as then the Go code could generate the self signed ssl certs within it, as well as do the installation steps.

Not sure if this is reinventing the wheel, but thought I'd throw the idea out there, and see if it stuck.

This is how helm does it:
https://docs.helm.sh/using_helm/#installing-helm

[markmandel]

More research:
Here is how Istio handles the certs for the mutating webhook:
https://istio.io/docs/setup/kubernetes/sidecar-injection.html#installing-the-webhook

Not pretty, but it is _a_ solution.

[cyriltovena]

interesting plugin https://github.com/SUSE/helm-certgen

[markmandel]

Chat from today.
TL;DR - We will go with Helm.

Mark Mandel [1:12 PM]
@Cyril Tovena chatting about Helm - quick thought - if we wanted to do a quick spike, to see what a helm chart for Agones looked like - easy/hard ?


Rod Cloutier [1:12 PM]
Would be fairly easy for us to do.

Mark Mandel [1:12 PM]
Sounds like that may be a nice way forward? Just to try it out, see how it feels?
especially if it will handle the certs issues
and installing helm doesn't look particularly onerous

Rod Cloutier [1:13 PM]
We don`t need to use Helm on both the client and server side. You can generate a yaml from Helm charts and do a `kubectl apply` afterward.

Mark Mandel [1:13 PM]
OH REALLY?
this is why I ask you smart people the questions :smile:
I did not know this
Sounds like Helm is a winner winner chicken dinner?

Cyril Tovena [1:15 PM]
like I said you could try to even get a stable chart
which is good for visibility
I suggest that I give a go with helm without certs and move from there

Mark Mandel [1:17 PM]
SGTM
I'll copy paste this little chat into the ticket, just for posterity/transparency

[cyriltovena]

while reading the doc I found that you can run post installation test : https://docs.helm.sh/developing_charts/#chart-tests

That's just awesome, I'm thinking to use flag switch in the controller container to run a simple gs/pod creation and release control.

WDYT ? I'm planing to introduce this after a first version so probably create a new issue if we all think that's a boomer.

[markmandel]

We do have a need for #37 (End to end tests) - so maybe this helps with both tickets!

[rodcloutier]

Just be careful with this as you now need to install tiller in the cluster to make your tests. This is nice to use a sanity check for the user but I would probably not use this as end to end tests.

[markmandel]

But wouldn't we need to install tiller anyway, so we could install Agones, so we could do end to end tests? :smiley:

[markmandel]

The other thought I had, if there is a way, is that there is a light test for end users, and then there's the _full_ test for the actual e2e tests - if that is possible.

[cyriltovena]

we can do a full test for the user using this ! (create a gs, wait for it to have a pod assigned, communicate)

I'm having hard time with end to end tests are really for us ? is it in an actual cluster ?

[markmandel]

I'm having hard time with end to end tests are really for us ? is it in an actual cluster ?
Apologies, I'm not following?

So one thing I'm very much thinking - we do want automated testing of the Helm chart as well as e2e tests.

So this is likely more a conversation for #37 - but part of that would be to install via helm, and then make sure things work. But we could defer the testing work until we have a valid Helm install.

This actually makes me even more happy about using Helm for this work - it has lots of useful tools!

[cyriltovena]

Sorry midnight comment, but yes I was asking about the scope of e2e tests.

[fooock]

To see the full scope of this feature take a look to this comment

Work breakdown

• [x] Create a Helm chart that can be installed and used on a cluster + docs (see #149)
• [x] Migrate build/ tooling to use Helm
• [ ] Write Helm test (details to be discussed)
• [ ] Chart repository
• [x] Update release process

[markmandel]

@fooock we'll want to add a work item of adding the ability to list namespaces that the gameservers will run in, and create service accounts for them - see #146 and #154 for context.

[markmandel]

I'm currently working on migrating the build tooling to use helm over here, which involves getting it ready for 0.2.

Soon, no more sed!

[cyriltovena]

Every remaining work has it's own issue, let's close it.