Kubeapps: Run our chart with multiple replicas

[migmartri]

It is best practice to run our services with at least 2 replicas so it enabled fault tolerance. We should revisit our chart and our apps to be sure that we this can be bumped.

This is part of an effort of making our deployment production ready, which also includes configuring our charts for QOS https://github.com/kubeapps/kubeapps/issues/478

[prydonius]

I believe odd numbers are typically used (e.g. 3, 5, 7), although that might be more important for databases?

The services that we should enable HA deployments for are:

• chartsvc
• dashboard
• tiller-proxy
• frontend nginx

We should document how to enable MongoDB as a replicaset, presumably just pointing to the chart's documentation: https://github.com/helm/charts/blob/master/stable/mongodb/values.yaml#L83

[andresmgot]

mm, should we revisit the MongoDB component? Having a MongoDB replicaset in HA is quite expensive. Having that for a cache system seems a bit of an overkill. WDYT?

[arapulido]

Yes, I agree with Andrés, we shouldn't recommend running MongoDB in HA, as it is just a cache.

[prydonius]

Good point about MongoDB - I agree that there's not much need to do that.

I'm not sure if @andresmgot suggestion was deeper, in that we should consider replacing MongoDB with something more lightweight for caching purposes (e.g. Redis). @andresmgot if that was what you were thinking, can you open a separate issue to discuss that?

[andresmgot]

Well yes, I am not sure if it's worth it but we can probably reduce the resources consumption for a production use case if we use something like Redis or etcd. I can open an issue to discuss that, maybe we want to keep mongodb around if we plan to use it for any future feature (#651).

[migmartri]

I believe odd numbers are typically used (e.g. 3, 5, 7), although that might be more important for databases?

AFAIK, the "suggestion" of an odd number is more associated with data replication in distributed systems, so the nodes can always reach quorum, a single one, even in split brain situation.

In our case, I'd aim to have 2 replicas for all our services except MongoDB. I am not worried about scalability here but about fault tolerance. We should expect K8s to shuffle, kill pods at anytime, so staying at 1 replica does not fit that model. I want to small increase in the number of replicas because soon https://github.com/kubeapps/kubeapps/issues/478 that will have a resource requirement consequence.

I know that keeping Mongo as a single replica does not completely solve that problem but we are reducing the risk. The user can always connect to a managed DB (or we can document how to enable HA) and will have a decent fault tolerant system.

Does it make sense?

[prydonius]

@migmartri that makes sense, thanks. As for MongoDB, as pointed out by @andresmgot for our use of MongoDB a single replica is fine.