Project: MongoDB - how to have a service always connecting to primary

[botzill]

Hi.

Currently, we have 2 services created when deploying mongodb: one which we can customize and other -gvr for internal use. Now, I want to access mongodb from outside and be able to work with DB. Correct me if I'm wrong, but we need to always connect to PRIMARY replica in mongo. Now, PRIMARY node can change and another SECONDARY take it's place that's why we never know which replica set is the PRIMARY. The question is how can we have a service which will always connect to PRIMARY?

Looking at https://github.com/helm/charts/tree/master/stable/mongodb/templates I see that they have 2 separate statefulset - for PRIMARY and SECONDARY - which solves this problem.

Si there any solution in kubedb for this?

Thx!

[tamalsaha]

One way to do this will be to use TCP-SNI to expose each pod of the statefulset externally.

• https://github.com/appscode/voyager/blob/master/docs/guides/ingress/tcp/tcp-sni.md
• https://github.com/appscode/voyager/blob/master/docs/guides/ingress/http/statefulset-pod.md

We are going to look into stable/mongodb does and update this ticket later.

[botzill]

OK, I see what you mean here. We can connect to mongo replica set via

mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl

And it will determine which one is PRIMARY automatically(I guess so).

But, we can have a dynamic number of replicas(by scaling when needed). So, one solution would be to have an ingress/service that, somehow, takes care of it and gives you PRIMARY server to connect to, magically.

Maybe I'm missing smth, please correct me.

Thx.

[botzill]

So, just to clarify things, I just saw that this problem is fixed for PostgresDB by having a label like kubedb.com/role: replica or kubedb.com/role: primary and the service always takes primary. The way it works is perfect and I think the same solution is required for MongoDB. Was there any reason why you didn't implement it like that?

Thx.

[the-redback]

In postgres, the leader election is done by kubedb leader election. So, It is possible to update the annotation while electing new primary.

In MongoDB, however, the mongodb itself elects the new leader. So, kubedb have to watch for leader switch, which is not an efficient way. :(

[botzill]

OK, but such implementation is not helpful either. Having some login into your application related to primary/secondary related to mongo db work is not really ... well how to say it, neat :). Do you have any solution in mind for this scenario? I really like so far kubedb and we plan to use multiple DB. But with this case I'm afraid we can't use mongodb and I'm sure lot's of people will have this scenario in their stack.

Thx a lot so far for all the helpful insights you gave us!

[the-redback]

TODO: solve this issue in either way:

• [ ] Write a doc to point how to use voyager to point to all pods of mongodb statefulsets.
• [ ] Find and use any better solution like proxy, labelselector etc.

[the-redback]

Here is a workaround in helm/chart issue. They used voyager :100: for TCP loadbalancing. https://github.com/helm/charts/issues/1569#issuecomment-446011045

[botzill]

Not sure I understood how they actually solve the primary secondary selection automatically?

We want to have a single endpoint to DB cluster and it's up to cluster to select proper primary which was elected.

[botzill]

Hi @the-redback , can you please clarify this, it's not clear for me how this will solve it?

[sfitts]

We would also like to see the operator label the primary pod -- not so much for routing purposes, but to help other tools we have do their job more efficiently and to help the operations staff quickly identify the current primary.

To do this the operator would probably have to mimic what the MongoDB drivers do which is essentially to poll the members and get their current replication state.

[the-redback]

@sfitts, but polling mongodb members then updating label will cause some downtime in case of leader change. :(

[the-redback]

@botzill, I don't have much knowledge about ingress. But, all they tried is to expose all the pod of the statefulsets.

[sfitts]

@the-redback I'm not sure I follow your comment. If Mongo has to switch primaries there is already a period of time during which the RS is read-only, a fact which is reflected in how the Mongo drivers work today. How does reflecting the result of the switch in the pod meta-data do anything to increase this time?

[sfitts]

@botzill if I'm reading the referenced issue correctly, all they do is arrange to expose a URI for all members of the RS (primary and secondary). Then they use these addresses in the mongo URI given to the client. This lets the client connect to all members and track any changes in leadership directly (since that's functionality which is built into the Mongo driver code).

[devonbarrett]

@sfitts is correct, it is best to let the driver handle detecting the primary rather than creating a separate service which needs to keep track of the primary pod itself.

Currently, this is already possible using the gvr service created by kubeDB:

//full uri
mongodb://user:password@$DBNAME-0.$DBNAME-gvr.$NS.svc.cluster.local:27017,$DBNAME-1.$DBNAME-gvr.$NS.svc.cluster.local:27017,$DBNAME-2.$DBNAME-gvr.$NS.svc.cluster.local:27017?replicaSet=$REPLICASET

The question is whether the created gvr service meant to be used by end users? If it is, it should probably be included in the docs. If it is only supposed to be internal to KubeDB, then it makes sense to create a headless service by default when users create a replica set.

[weenxin]

Maybe we should never need to know which the primary node is in MongoDB cluster . Because

Backgroud

We have 3 nodes of MongDB cluster named rs0 (Primary: 192.168.0.101 Secondary：192.168.0.102 Secondary：192.168.0.103)

Purpose

We want to connect to the MongoDB cluster

Method

1. mongo --host rs0/192.168.0.101:27017,192.168.0.101:27017,192.168.0.101:27017
2. mongo --host rs0/192.168.0.101:27017
3. mongo --host rs0/192.168.0.101:27017

All above three methods work

Reason

We connect to MongoDB Replica Set not any MongoDB instance .
The MongoDB driver(C++ , python , go ... ) will do extra things for use automatically.

[the-redback]

NB: The following information only applies to mongodb replicaset clustering.

The doc is not clear about what should be used as service in user end. I wrote that doc, and I am not sure either.
Certainly, the Mongodb service is not a solution to write operation, because it connects to any one of the instances.
The headless service can be used for write operation (that's the only way I can see right now), but user can't connect to it from outside of cluster. (As our other priorities are almost done (ie, mongodb sharding, stash integration), we can look into it and hopefully we can propose a solution.)

Meanwhile, There two kind of hosts possible,

• Mongodb service as host: <mongodb-name>.namespace.svc

• Governing service as host: <replicaSet-name>/<db-name>-0.<db-name>-gvr.<namespace>.svc,<db-name>-1.<db-name>-gvr.<namespace>,<db-name>-2.<db-name>-gvr.<namespace>

  Here is an example of Governing service as host where DB name is sample-mgo-rs: rs0/sample-mgo-rs-0.sample-mgo-rs-gvr.demo.svc,sample-mgo-rs-1.sample-mgo-rs-gvr.demo.svc,sample-mgo-rs-2.sample-mgo-rs-gvr.demo.svc

  NB: KubeDB ensures the governing services as well. so, user don't need to create them to use as host.

So,

• For read operation: user can use any one of these two hosts
• For write operation: user need to use governing service as hosts

TODO (for kubedb developers/contributors):

• [ ] Find a solution to do write operation from outside of the cluster.
• [ ] Update document.

[zarbis]

@the-redback I want to clarify whether above solution is intended for general use and not only for issue in author's case (access from outside the cluster)?