Agones: Agones controller does not remove deleted pod from game server list

[KamiMay]

I had a fleet of 2 game servers running, one of the game server was allocated, I deleted allocated server by running kubectl delete pod pod-name-xyz this deleted the pod but when retrieving fleet information it still showed that fleet had two game servers running and kept returning allocated server IP and PORT to which we couldn't connect because game server pod has been deleted. Is there any reason why agones does not detect deleted pods?

[markmandel]

Deleting the Pod _should_ delete the backing GameServer:

Logic is here:
https://github.com/GoogleCloudPlatform/agones/blob/master/pkg/gameservers/controller.go#L167

Sounds like we should add a e2e test here:
https://github.com/GoogleCloudPlatform/agones/blob/master/test/e2e/gameserver_test.go

Make sure it always works.

[markmandel]

Oh - I assume this was on 0.9.0-rc?

[KamiMay]

No, it was on 0.8.1.

0.9.0-rc was crashing our matchamker so we had to revert it for QA, we are not sure why it was crashing our matchmaker, so I have not created an issue for that yet. Once I have time for investigation I will look into it, but approaching release, it might not be soon.

[markmandel]

Digging into this some more - deleting a standalone gameserver's pod will make the GameServer move to Unhealthy (it's essentially the same as a crash).

If the GameServer belongs to a Fleet, then the GameServer should be recreated.

Got some more e2e tests working on this.

[markmandel]

So I wrote e2e tests in #684, and they all passed - does that match up to the context you were describing?

[KamiMay]

I forgot to mention that the deleted pod had an allocated game server instance running on it.

[markmandel]

Cool - I'll make sure I test that as well!

[markmandel]

So you'll be pleased to know, that now that we have an e2e test, we can see this happens... sometimes.

Trying to work out why it only happens sometimes though. Somehow managed to hit it just now manually).

:grimacing: inconsistent bugs are the best.

[markmandel]

I think I see where this is... and it was... never really implemented? (or if it was, it was lost). Fixing it!

[djsell]

I've had this happen to me a couple times in v1.0.0

[markmandel]

@djsell - thanks for reporting!

Please open a new issue with repro steps, so we can look into it :+1: