Ingress-nginx: Please include an example for a StatefulSet

[IainColledge]

Deploying a container which requires persistent storage would appear to best be served by utilising a StatefulSet.

However being able to complete the link from external IP to internal cluster DNS name (i.e. teamcity.default.svc.cluster.local) via an ingress controller and headless service is proving difficult to work out.

Therefore a working example would be a real benefit to kubernetes noobs like me.

If I was doing this using stateless pods then I would just use nginx as a reverse proxy but this is what the nginx ingress controller seems to be for and so would rather adopt best practice than roll my own.

[gianrubio]

Deploying a container which requires persistent storage would appear to best be served by utilising a StatefulSet.

Could you give a real example for this?

[IainColledge]

For sure although I may be going down a rabbit hole being new to Kubernetes. Here is a concrete scenario.

I've containerised Teamcity and want to run it with persistent volume storage for both it's data and config. In Docker I define a volume directory which will hold a data directory and a conf directory and this becomes the mount point for the volume. Under a docker this may just lives locally for example.

As I'm utilising Azure right now there is no easy way to create a blank vhd so looking at options the StatefulSet becomes attractive.

From: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

StatefulSets are valuable for applications that require one or more of the following.

• Stable, unique network identifiers.
• Stable, persistent storage.
• Ordered, graceful deployment and scaling.
• Ordered, graceful deletion and termination.

In the above, stable is synonymous with persistence across Pod (re)schedulings.

Also from http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html

Of course, this is not true at all. At Google, everything runs in a container, including databases. You just need the right tools. Kubernetes 1.5 includes the new StatefulSet API object (in previous versions, StatefulSet was known as PetSet). With StatefulSets, Kubernetes makes it much easier to run stateful workloads such as databases.

If you’ve followed my previous posts, you know how to create a MEAN Stack app with Docker, then migrate it to Kubernetes to provide easier management and reliability, and create a MongoDB replica set to provide redundancy and high availability.

While the replica set in my previous blog post worked, there were some annoying steps that you needed to follow. You had to manually create a disk, a ReplicationController, and a service for each replica. Scaling the set up and down meant managing all of these resources manually, which is an opportunity for error, and would put your stateful application at risk In the previous example, we created a Makefile to ease the management of of these resources, but it would have been great if Kubernetes could just take care of all of this for us.

With StatefulSets, these headaches finally go away. You can create and manage your MongoDB replica set natively in Kubernetes, without the need for scripts and Makefiles. Let’s take a look how.

So as someone new to Kubernetes the ability to maintain state automagically by matching volumes to pods etc when nodes die is quite attractive. And it does using Azure Storage controllers, Persistent volumes and PVCs.

After setting things up with a headless service the next step of using an ingress controller does nto seem to work at all and it may be that I'm just using the wrong tool for the job here. In the end I just created a reverse proxy using nginx and had that point to teamcity using DNS, much as I use with Consul. As it has an external IP and is stateless everything works as expected.

However rolling ones own reverse proxy using nginx does seem to be recreating what from a noob perspective the nginx ingress controller should be able to do.

Hence the request for an example of how to expose a containerised web based service to the Internet, it would be really helpful to noobs like me.

[aledbf]

Closing. StatefulSet is not part of ingress. At ingress level there is no difference between the echoheaders example and any other app (like one using StatefulSet).

[IainColledge]

I'll try again to deploy a StatefulSet to the outside world using an ingress controller.

The examples all have IP addresses afaik and a StatefulSet doesn't, you have to connect using DNS.

For nginx reverse proxy I set in the conf for a location:

set $backend_upstream "http://<StatefulSet name>.<StatefulSet namespace>.svc.cluster.local:<StatefulSet port>";

Is this how nginx controllers work? Using k8s internal DNS?

[aledbf]

@IainColledge no, please check this article https://medium.com/@cashisclay/kubernetes-ingress-82aa960f658e

[IainColledge]

Thanks for that so even though a StatefulSet doesn't give you a cluster IP it shouldn't be a problem then.