Panel: Kubernetes Support

[Syntax3rror404]

Please make Pterodactyl usable with Kubernetes. If you use Kubernetes with Pterodactyl, your master don't know the containers there are created by the Pterodactyl Deamon :)

[TrixterTheTux]

Why do you need Kubernetes to be aware of the game containers? It wouldn't be interacting with the game containers anyways?

[Yomanz]

scalable Minecraft servers duh

[Syntax3rror404]

No, not scalable Minecraft Servers :D. You can already use the Pterodactyl Panel with K8 by using the Docker Image https://hub.docker.com/r/ccarney16/pterodactyl-panel. But when you using this, your K8 Master dont know the containers there are created by the Pterodactyl Deamon. In this case you doens't have the benefits from the orchestration and the K8 Master can purge the own pods. To solve this problem the deamon must speak with the K8 API server and not with the docker socket. This would totaly awsome :)

[MPV]

@Syntax3rror404 when you say...

But when you using this, your K8 Master dont know the containers there are created by the Pterodactyl Deamon. In this case you doens't have the benefits from the orchestration and the K8 Master can purge the own pods.

...I'm thinking:

In case you don't want Pterodactyl to compete with Kubernetes on using the local Docker Daemon on the actual nodes, maybe an alternative would be to try Docker-in-Docker (DinD)? For example, creating new Pterodactyl "nodes" as simple Kubernetes Pods with an image running the Docker Daemon inside it? See https://hub.docker.com/_/docker for more about DinD and caveats.

[MPV]

If wanting to rewrite the Pterodactyl Daemon to support the Kubernetes API as an alternative to the Docker Daemon, this seems like it might be a good place to start looking: https://github.com/pterodactyl/daemon/blob/ddf4810b3bcc6f2dc8aefc0e4efa942b5e3f7519/src/controllers/docker.js#L524

[MPV]

@TrixterTheTux:

Why do you need Kubernetes to be aware of the game containers? It wouldn't be interacting with the game containers anyways?

A benefit of using Kubernetes instead of the Docker daemon for the Pterodactyl "nodes" might be that you wouldn't need to consider on which node a specific game server will run (as Kube will schedule it for you on a free kubernetes node).

If Pterodactyl had Kubernetes support, I imagine in theory it could possibly even skip the Pterodactyl "Node" concept and only consider the Pterodactyl game "Servers" directly (hence making sense for Pterodactyl to be aware of the game containers running as part of Pods/Deployments in Kube).

[VaasKahnGrim]

It would be pretty nice for being able to automate the creation of new nodes and maybe other services
all through kubernetes instead of having to use several differant softwares for webhosting, VPSs, ptero nodes, and so on.

[TrixterTheTux]

Kubernetes simply doesn't have the feature of automating the provisioning of new nodes (at least out-of-the-box). That's why there's Puppet, Ansible and etc and what MPV is asking wouldn't solve this. And nothing prevents you from running Kubernetes alongside Pterodactyl to already automate the provisioning of additional software you want the node to run.

[MPV]

Kubernetes simply doesn't have the feature of automating the provisioning of new nodes (at least out-of-the-box).

@TrixterTheTux It depends.
If you rent a Kubernetes cluster from a cloud provider, they provide you with "cluster autoscaling", which does just that:

Cluster autoscaler looks for the pods that cannot be scheduled and checks if adding a new node, similar to the other in the cluster, would help. If yes, then it resizes the cluster to accommodate the waiting pods.

https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#cluster-autoscaling
https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler

[MPV]

As inspiration, here's another rough effort at creating game servers using Kubernetes:
https://github.com/dhtech/gaas (Dreamhack "Gaming as a Service")

[DaneEveritt]

I do not have the time nor bandwidth to support this, so I'm not even going to leave it open and pretend it'll happen in any reasonable amount of time.

[sfxworks]

@DaneEveritt if you point me to where the majority of your logic is to start/stop containers I might do this.

Been wanting to take advantage of a UI when it comes to managing the k8s of things (even though I prefer node over php) https://github.com/mcserverhosting-net/charts

[schrej]

The new logic is here: https://github.com/pterodactyl/wings/tree/develop/server

[sfxworks]

https://github.com/mcserverhosting-net/mc-operator Made a Kubernetes operator for it. Once this is battle-tested Ill see about working it into the new logic.

An example CRD would look like this https://github.com/mcserverhosting-net/mc-operator/blob/master/deploy/crds/charts.helm.k8s.io_v1alpha1_serverdeployment_cr.yaml

With a full list of possible values at https://github.com/mcserverhosting-net/charts/blob/master/server-deployment/values.yaml

root@vps207526:~/mcsh-op/mcsh-operator/deploy/crds# kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
mcsh-operator-65f8f6fd4c-ntxfb   1/1     Running   0          52s
vanilla-test-1-0                 1/1     Running   0          15s

[sfxworks]

May 2020 update - https://github.com/mcserverhosting-net/mc-operator
The operator now contains chart references for NameSpace prep (quotas and things), SFTP, and Bungee deployments. Feel free to join me. We (MCSH) will be applying this all out on our cluster soon. Hoping this will be able to be applied to Pterodactyl since it gives a nice json/yaml submission for the dashboard to fire and forget with as well as an SFTP endpoint now.

[DaneEveritt]

If I'm being completely honest and transparent: I have exactly 0 experience with k8s and barely have the time and resources available to me to keep up with Pterodactyl development.

Unfortunately unless someone actively gets involved in the community development channels for this and leads the way with a PR (and is able to show that they'll be able to help maintain it), it likely will never make it into the core.

I can basically guarantee I won't be able to implement this effectively or securely with the resources available to me right now.

:/

[schrej]

While I like the idea of a minecraft operator for k8s, I think it's not a good fit for Pterodactyl as it conflicts with eggs. This would then require to differentiate what can be deployed based on the target node and require two different sources of what game servers are available: eggs and whatever the operator can deploy.

If k8s would be added to pterodactyl, it should probably just be a replacement for docker in wings. K8s would also complicate how ip/port allocations are currently handled. If the current approach should remain, overlay networking would be required, which might degrade performance a bit - something to avoid with game servers. Alternatively wings would need to be able to decide what allocation it will use and communicate it's decision back to the panel.

[sfxworks]

I've never heard of your egg service before. It looks interesting! Correct me if I am wrong, but I don't believe it is a direct conflict with the Minecraft operator since eggs could conform to becoming an operator given the similar objective of managing an application's lifecycle.

On the note of wings and port mapping, there is a service type in k8s called a NodePort that takes the range of (by default) 30000 - 32767, simplifying the process.

On the note of networking, there are Container Network Interfaces that work with k8s to negate the requirement of an overlay network, such as utilizing a routing protocol or using a cloud provider’s routing functionality. Though that would be within the scope of the cluster-admin rather than the operator to setup .

All and all I hope Pterodactyl reconsiders integrating with Kubernetes! Both on-prem and cloud technologies are working with Kubernetes nowadays. I would be happy to contribute, but the most I can do is provide endpoints and helm based operators since I do not know Golang for Wings.

[DaneEveritt]

I'd love to see K8's integration at some point, but right now given my limited time, my inexperience with K8's, and the ever-growing backlog of feature requests and bug reports I think it would be unwise — and most likely extremely detrimental to the project — for me to pursue building out or maintaining yet another system to run servers.

[JeffreyDD]

First of all, compliments on the project, this is probably the nicest game server management solution I've come across so far.

A proper k8s implementation in this project would make a lot of the custom implemented functionality (basically everything which talks to docker) obsolete and unusable, as k8s would handle this (network, storage, actual "servers"). Also the concept of Nodes would be replaced by something like k8s environments (as you see in Gitlab for instance).

It would allow the project from the massive active kubernetes community, but extremely hard to implement in Pterodactyls current architecture, as you'd get a lot of duplication over both backend platforms. A better option would probably be a fork, where we'd remove a lot of the "node" (server) management functionality, and probably drop 90% of everything implemented in the current daemon.

When it comes to networking - anything you can do in docker, you can do in kubernetes. Whether you should is a whole other thing. You'd probably want to use HostPorts together with a nodeName on your pod definition, as this will allow you to reach the container directly on the hosts' IP through specified port. But, in kubernetes, you'd want to let kubernetes manage this for you (that's what it's built for). As in any hosted environment, you'd never want external hosts to connect directly to your backend servers, but in game server hosting this is more the standard than the exception.

All in all, is it possible - yes. Is it something you'd want - probably as well. Is it a good fit for this project - not at all. But if I started a new project now, I'd definitely pursue all options to use kubernetes prior to implement it's functionality by myself, as the ecosystem is so much bigger, and so much is happening on a daily basis. But, you'll need to learn kubernetes first.

Just my 2 cents

[MPV]

Also, many game servers by default expose themselves on the same port on both UDP (game) and TCP (rcon), which isn't supported yet on Kubernetes services: https://github.com/kubernetes/kubernetes/issues/23880

[schrej]

I fully agree with @JeffreyDD, very well explained.
In case someone actually wants to fork Pterodactyl to build a control panel for k8s (or start from scratch), googleforgames/agones might be worth a look.

[MatthewSH]

At that point might as well create a software called Velociraptor or something. 😆

[sfxworks]

Also, many game servers by default expose themselves on the same port on both UDP (game) and TCP (rcon), which isn't supported yet on Kubernetes services: kubernetes/kubernetes#23880

This is actually just an issue requesting support for both protocols with the same service definition.
https://github.com/kubernetes/kubernetes/pull/30253
K8s supports UDP and TCP on the same port. Please take a look at the Service definition and concepts if you wish to know more https://kubernetes.io/docs/concepts/services-networking/service/

I agree with JeffreyDD's post too. I actually didn't know how much the project worked to orchestrate everything and how much the program let docker go about handling things.

Afaik agones and another project was more of a dev kit, so that may go into a route of creating a custom server implementation like with https://github.com/cuberite/cuberite

I will continue work on the operator then for my goals and ambitions. No frontend for the service as of now, but will definitely be looking into this projects resources and a potential fork when we get there! Definitely a great discussion everyone.

[MPV]

K8s supports UDP and TCP on the same port. Please take a look at the Service definition and concepts if you wish to know more https://kubernetes.io/docs/concepts/services-networking/service/

It does, but not for Type: LoadBalancer, as mentioned in the page you linked @sfxworks:

For LoadBalancer type of Services, when there is more than one port defined, all ports must have the same protocol and the protocol must be one of TCP, UDP, and SCTP.

[sfxworks]

@MPV For stateful services like Minecraft with only one replica, you wouldn't put a load balancer in front of it. LoadBalancers are better used for services that scale-up and down.

We at MCSH use NodePort + SRV Records. (We actually started messing with https://hub.docker.com/r/itzg/mc-router which is very useful)

Though if you really wanted, you could follow Metallb's example and use two service definitions like I was referencing https://metallb.universe.tf/usage/

Kubernetes does not currently allow multiprotocol LoadBalancer services. This would normally make it impossible to run services like DNS, because they have to listen on both TCP and UDP. To work around this limitation of Kubernetes with MetalLB, create two services (one for TCP, one for UDP), both with the same pod selector. Then, give them the same sharing key and spec.loadBalancerIP to colocate the TCP and UDP serving ports on the same IP address.

The same concept applies in that you would set the spec.loadBalancerIP the same, but it would also depend on the underlying provider on whether that and UDP is supported (OpenStack's Octavia, for example, doesn't support UDP at all)

[parkervcp]

How many games actually support this kind of load balancing though? Another reason a kube cluster may not be the best way to manage this kind of setup.

I get that you just say "I want one server and have it show up on this IP" and it does the job and routes around all the nodes.

What kind of latency does that potentially add to the games too?

[sfxworks]

@parkervcp

How many games actually support this kind of load balancing though? Another reason a kube cluster may not be the best way to manage this kind of setup.

At the end of the day, its TCP and/or UDP bound to a port. All games support that. I could bind the internal port of a Minecraft server to 1000 with the internal IP via server.properties and 9999 on all nodes and their external IP in the service definition.

I get that you just say "I want one server and have it show up on this IP" and it does the job and routes around all the nodes.
What kind of latency does that potentially add to the games too?

From Node to Container, negligible.
From Node to Node to container in a scenario where you hit a node that the server isn't on, you would just add on the latency between the two nodes.

You can get real technical too, like only allowing a route based on the node that it's on, updating the DNS record as it changes using External Traffic Policy + ExternalDNS or implementing Direct Server Return.

Even something as simple as using IPVS instead of IPTables helps.

Context:

Two customers on different nodes

kubectl get pods -n i-jayt424ckf4u -o wide
NAME                      READY   STATUS      RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
obsidian-brick-server-0   1/1     Running     19         31d   10.34.0.102   a3     <none>           <none>
obsidian-brick-ssh-0      1/1     Running     0          31d   10.34.0.101   a3     <none>           <none>
update-notify-ttkjt       0/1     Completed   0          31d   10.34.0.97    a3     <none>           <none>

kubectl get pods -n i-flxllduwsvxd -o wide
NAME                      READY   STATUS      RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
lapis-redstone-server-0   1/1     Running     1          16d   10.36.0.26   a5     <none>           <none>
lapis-redstone-ssh-0      1/1     Running     0          60d   10.35.0.89   a4     <none>           <none>
update-notify-wtmck       0/1     Completed   0          60d   10.35.0.88   a4     <none>           <none>

In server on a3 pinging a server on a4

kubectl exec -it -n i-jayt424ckf4u obsidian-brick-ssh-0 -- /bin/sh
/ # ping 10.35.0.89
PING 10.35.0.89 (10.35.0.89): 56 data bytes
64 bytes from 10.35.0.89: seq=0 ttl=64 time=1.622 ms
64 bytes from 10.35.0.89: seq=1 ttl=64 time=0.268 ms
64 bytes from 10.35.0.89: seq=2 ttl=64 time=0.406 ms
64 bytes from 10.35.0.89: seq=3 ttl=64 time=0.356 ms
64 bytes from 10.35.0.89: seq=4 ttl=64 time=0.404 ms
64 bytes from 10.35.0.89: seq=5 ttl=64 time=0.449 ms
^C
--- 10.35.0.89 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.268/0.584/1.622 ms

Ping from a3 node to server a4

ubuntu@a3:~$ ping 192.99.16.87
PING 192.99.16.87 (192.99.16.87) 56(84) bytes of data.
64 bytes from 192.99.16.87: icmp_seq=1 ttl=63 time=0.262 ms
64 bytes from 192.99.16.87: icmp_seq=2 ttl=63 time=0.197 ms
64 bytes from 192.99.16.87: icmp_seq=3 ttl=63 time=0.172 ms

A quick example of two random customers in the same region which requires a hop between nodes. Less than a millisecond. For the record, we use IPVS instead of IPTables for routing and also use the Weavenet CNI.

On top of all of that... Applying something like Rook Ceph behind the scenes keeps the data redundant across the servers, which allows for some good failover scenarios.

[MPV]

@sfxworks Well described! 😍

Another detail/context on why mixed protocol load balancing will help a bit in some cases:

How many games actually support this kind of load balancing though? Another reason a kube cluster may not be the best way to manage this kind of setup.

At the end of the day, its TCP and/or UDP bound to a port. All games support that. I could bind the internal port of a Minecraft server to 1000 with the internal IP via server.properties and 9999 on all nodes and their external IP in the service definition.

Yes, games do support changing to a non-standard port.

However it can become a tad more inconvenient for some game clients/consumers. For example, I have run a couple of CS:GO servers in Kubernetes and wanted to configure some web-based game management tools to stats/lit/admin each server, but quite a few tools didn't allow non-standard ports. They either didn't let you configure port at all, or just let you add one port and assumed both UDP and TCP ports on that protocol. I've asked some such tools to reconsider their model but as you can imagine it wasn't deemed worth the effort yet. Hence why I wanted to run load balancer services: so I could get different IP addresses with the same standard port/protocol usage for each server. Another benefit of using services is that I could also use external-dns to have automatically updated DNS records to each server.

[sfxworks]

Thanks, @MPV !
A good idea may be to host the web-based management tool in a k8s pod. That way you could keep a standard port for your game server internally and reference the game's service via its internal IP instead of multiple external IPs and load balancers. Another option, if it is something that must live on your desktop and can't be hosted in a pod, could be to extend your network to your local machine with OpenVPN.

[Soren90]

@sfxworks Well described!

Another detail/context on why mixed protocol load balancing will help a bit in some cases:

How many games actually support this kind of load balancing though? Another reason a kube cluster may not be the best way to manage this kind of setup.

At the end of the day, its TCP and/or UDP bound to a port. All games support that. I could bind the internal port of a Minecraft server to 1000 with the internal IP via server.properties and 9999 on all nodes and their external IP in the service definition.

Yes, games do support changing to a non-standard port.

However it can become a tad more inconvenient for some game clients/consumers. For example, I have run a couple of CS:GO servers in Kubernetes and wanted to configure some web-based game management tools to stats/lit/admin each server, but quite a few tools didn't allow non-standard ports. They either didn't let you configure port at all, or just let you add one port and assumed both UDP and TCP ports on that protocol. I've asked some such tools to reconsider their model but as you can imagine it wasn't deemed worth the effort yet. Hence why I wanted to run load balancer services: so I could get different IP addresses with the same standard port/protocol usage for each server. Another benefit of using services is that I could also use external-dns to have automatically updated DNS records to each server.

I thought i would just add my experience and workaround to this.

We have been using CS:GO with a system called ebot, which lets you easily hosts tournaments. For this to work, it requires that TCP/UDP (game/rcon) is running on the same port, for the bot to be able to control the servers and there is no option to change the RCON port on the bot side. We have used two different workarounds for this issue:

1. NodePort works with mixed protocol, but you need to specify both TCP/UDP seperatly in your manfest. then set the gameserver to run on standard port (27015) and then point another IP to the NodePort on your router. This solution is not redundant, requires manual steps and a router/external loadbalancer that supports VirtualIP's

2. LoadBalancer with Metallb. This solution is what we are using for now. You need to create two services, one for TCP and one for UDP and then set them to the same IP, for example:

apiVersion: v1
kind: Service
metadata:
  name: csgo-server-udp
spec:
  selector:
    app: csgo
  ports:
    - protocol: UDP
      port: 27015
      targetPort: 27015
  type: LoadBalancer
  loadBalancerIP: 10.0.0.2
---
apiVersion: v1
kind: Service
metadata:
  name: csgo-server-tcp
spec:
  selector:
    app: csgo
  ports:
    - protocol: TCP
      port: 27015
      targetPort: 27015
  type: LoadBalancer
  loadBalancerIP: 10.0.0.2

This is pretty inconvenient as well, but it works fine as long as you are using Metallb. This configuration might not work on some cloudproviders.

[DaneEveritt]

The new wings daemon has much better support for custom process environments now as well, so adding K8s and/or Podman should be fairly stright-forward for people to at least play around with implementing. Theres still some work to do in terms of making it truly abstracted, but I think the code should get you 90% of the way there, and you'll just need to modify a few places that configure the environment handling for the server.

[sfxworks]

Awesome. I look forward to that. I would be more than happy to test out integration between the operators I've been making/customers have been using and contribute sample configurations to wings for others.

I should also mention we have a lot of our (non-secretive) manifests open to the public which shows what a namespace running the operators and their configurations would look like.