Agones: Game server container crash before Ready, should restart, not move to Unhealthy

[markmandel]

What happened:

If the game server container crashes at any stage, the GameServer moved to Unhealthy straight away.

What you expected to happen:

According to: https://agones.dev/site/docs/guides/health-checking/

If any of the GameServer container fails health checking, or exits before the GameServer moves to Ready then, it is restart as per the restartPolicy (which defaults to “Always”)

How to reproduce it (as minimally and precisely as possible):

We will need a e2e test that has a CRASH command on udp-simple that does a sdk.Exit(1) and then test what happens when a crash occurs before Ready and after Ready.

Anything else we need to know?:

Logic can be found here:
https://github.com/googleforgames/agones/blob/master/pkg/gameservers/health.go#L87

Environment:

• Agones version: development
• Kubernetes version (use kubectl version): 1.12
• Cloud provider or hardware configuration: GKE
• Install method (yaml/helm): helm
• Troubleshooting guide log(s): N/A
• Others:

[Colstuwjx]

I also met this issue, version table see below:

agones: 0.11.0 
xonotic version: 0.5

BTW, it seems that our gameserver\fleet crd not maintains POD replica count, if I found the pod is unhealthy, and maunally deleted it, things would be broken, and gameserver/fleet would never try to recreate a new POD...

Thanks.

[markmandel]

BTW, it seems that our gameserver\fleet crd not maintains POD replica count, if I found the pod is unhealthy, and maunally deleted it, things would be broken, and gameserver/fleet would never try to recreate a new POD...

We have an e2e test for this, so _in theory_ this shouldn't happen, but if you are finding this an issue - please file a new bug with detailed replication instructions so that we can reproduce and fix. Thank you! 👍

[aLekSer]

I was trying to develop such E2E test. It seems that we need a separate simple-udp server which would part of the time fail to start say randomly one time out of 2 (os.Exit() before Ready).
I was not able to send something to UDP server before Ready because it does not have allocated ports.
Added such check and test fails on if statement:

func SendGameServerUDP(gs *agonesv1.GameServer, msg string) (string, error) {
    if len(gs.Status.Ports) < 1 {
        return "", errors.New("Port was not assigned to the GameServer")
    }
        ..
}

[markmandel]

I think we could make it a command line argument for the simple-udp startup script, and then overwrite the entrypoint in the configuration (rather than a whole new image)

[aLekSer]

I have created a branch which contains a UT with the part of health checks from https://agones.dev/site/docs/guides/health-checking/#health-failure-strategy
https://github.com/aLekSer/agones/tree/restartPolicyBeforeReady
The test itself which will perform os.Exit() before and after Ready state. If we call before Ready than number in restartCount should be >0 .
And note that GameServer moves to Scheduled state first.
https://github.com/aLekSer/agones/blob/86af4e7c3a9b5b2770f832d2d7c216346f5e023a/test/e2e/gameserver_test.go#L37

[markmandel]

FYI - I think I have a handle on this. It's a bit fun and tricky, but slowly taking it apart.