Yugabyte-db: Support for Kubernetes-based stateful deployments of YugabyteDB across multiple zones, regions and clouds

[jmirc]

At this time, it is not possible to specify the placement info in the chart.

What is the best way to do it?

[kmuthukk]

hi @jmirc -- looking into this. Will have someone in the team respond soon with appropriate guidance.

[schoudhury]

@jmirc we recently added the documentation on how to set the 3 placement_info flags (placement_cloud, placement_region & placement_zone) using the YugabyteDB Helm Chart - https://docs.yugabyte.com/latest/deploy/kubernetes/oss/helm-chart/#yugabytedb-configuration-flags

As you can see in the above doc, the values for these flags can only be set at an overall cluster level (as opposed to individual node level) and are set manually by the user. So the way to use these flags correctly today entails the following:

1. Create 3 k8s clusters where one cluster is created per zone/region/cloud. The networking between these 3 clusters should be done in such a way that pods in one cluster can discover pods in another cluster (projects such as https://submariner.io/ maybe needed for this purpose esp. if you are using upstream kubernetes).

2. Install 1/3 pods of the YugabyteDB cluster in each k8s cluster using the YugabyteDB Helm Chart while ensuring that each install specifies the placement_info flags accurately.

3. Confirm that you have a single YugabyteDB cluster spanning 3 k8s clusters where YugabyteDB pods in each k8s cluster have their own placement_info.

Next steps on our side

1. We will be enhancing our docs in the next couple days to document the above steps in detail. Will share that doc with you on this GitHub issue as soon as it is available.
2. The 3 k8s cluster approach can be avoided with a single multi-zone k8s cluster that is currently supported only by Google Kubernetes Engine (GKE) and Amazon EKS. We will be adding this documentation as well.
3. We will work towards populating these flags automatically from k8s cluster config so that users will not have to specify them manually. Don't have a specific timeline yet, will update you as soon as we have one.

Question for you
Is the above in alignment with your expectations? If you had something else in mind, please let us know asap so that we can figure out the best course of action.

[jmirc]

Actually, we are using EKS and we deployed one k8s cluster that will deploy one master per AZ. We will deploy as many tserver we need and will be deployed on 3 different AZ too. The documentation seems to work only if I am deploying on a unique AZ.
Let me know if I missed something in the doc.

[schoudhury]

Thanks for the response @jmirc. The installation on Amazon EKS which by default spins up 3-zone k8s cluster follows a similar pattern -- use the helm chart once for each of the 3 zones to spin 1/3 pods of the db cluster and then finally configure the yb-master service (via yb-admin) to note that all the pods (across all the 3 zones) are essentially a single db cluster with replication factor 3. We are documenting these instructions ahead of the generic multi-cluster instructions so that we can unblock you.

Please stay tuned for our update.

[schoudhury]

@jmirc we have published the following docs

1. Deploy YugabyteDB on a multi-zone Amazon EKS cluster
2. Deploy YugabyteDB on a multi-zone/regional Google Kubernetes Engine (GKE) cluster

Both these deployments use the standard YugabyteDB Helm Chart (which by default deploys to a single zone only) to deploy one third of the nodes in the database cluster in each of the 3 zones Kubernetes is deployed in.

Let us know if this works for you.

[jmirc]

Thanks very much. Super useful documentation. I will change how currently I deployed yugabyte.

Another question. Is it better to use a memory based instance type or a cpu based instance type.

[jmirc]

Thanks very much. Super useful documentation. I will change how currently I deployed yugabyte.

Another question. Is it better to use a memory based instance type or a cpu based instance type.

[rkarthik007]

@jmirc - assuming you have enough CPU and memory for the app to work (i.e. the cluster is not starved on either resource), its better to go for a CPU-optimized instance. Note also that if the app is write heavy, the network performance matters (high performance network, this matters on AWS for example where some instance have higher network perf).

[jmirc]

Another question. The deployment works if we are not taking in consideration an availability_zone will never go down. If this happens, the cluster will not be available.

[schoudhury]

@jmirc The cluster will still remain available when zone fails assuming the cluster has been configured with zone-aware replica placement policy. this policy ensures that two replicas of the same tablet (aka shard) can never reside on the same zone. when a zone fails, some tablets lose their leaders. the remaining two follower replicas in the two remaining zones will auto elect a new leader in a matter of seconds and write operations to those impacted tablets can continue thereafter. write unavailability is limited to only the leader re-election time and reads remain available from the follower replicas (albeit a bit stale) all the time. note that YugabyteDB uses Raft distributed consensus protocol for both data replication and leader election.

How Does the Raft Consensus-Based Replication Protocol Work in YugabyteDB? explains the above in fair amount of detail.

[jmirc]

Great thanks for the explanation.

[schoudhury]

Doc for Deploy YugabyteDB on multi-cluster, multi-region GKE clusters is now live. We will do the same for Amazon EKS soon. These docs serve as a template for multi-cloud Kubernetes deployments where each cloud has its own Kubernetes cluster and all the clusters are glued together using a global DNS system.

Closing this issue, pls open a new one if you have questions in this regard.