Charts: [bitnami/mongodb] MongoDB is not rejoining cluster

[estahn]

Which chart:
bitnami/mongodb and 7.13.0

Describe the bug
Given a cluster with 1 primary and 2 secondary pods. We're terminating the primary which results in the secondary to take over. After the primary comes back it doesn't rejoin and the secondary stays "primary". To make the primary rejoin we execute the following on the secondary:

cfg = rs.conf()
rs.reconfig(cfg)

To Reproduce
Steps to reproduce the behavior:

1. Create a cluster with 1 primary and 2 secondary
2. Delete the primary
3. Check logs of secondary
4. See error

Expected behavior
Primary rejoins the cluster.

Version of Helm and Kubernetes:

• Output of helm version: We use ArgoCD, which uses helm template internally.
• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:35:15Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.10", GitCommit:"e3c134023df5dea457638b614ee17ef234dc34a6", GitTreeState:"clean", BuildDate:"2019-07-08T03:40:54Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Additional context
n/a

[estahn]

@horihel makes a good observation in #2461. A cluster with the following settings is working fine:

clusterDomain: cluster.local
replicaSet:
  clusterDomain: true

[javsalgar]

Hi,

This is weird. So the thing is that the search settings in the cluster don't add the cluster.local section and so the resolution fails, right?

[moh-abed]

The clusterDomain change didnt resolve the issue actually, primary did not rejoin unless manually calling rs.reconfig()

[javsalgar]

Hi mod-abed, did setting the values here fix the issue?

https://github.com/bitnami/charts/issues/2460#issuecomment-621169403

[SwannMR]

Hi,

I experienced the same issue while running a cluster with 1 primary and 2 secondary mongos.

After deleting the primary pod there is a log entry that suggests the server cannot resolve the service name.

NETWORK  [replexec-0] getaddrinfo("vision-mongodb-primary-0.vision-mongodb-headless.vision") failed: Name or service not known
REPL     [replexec-0] Locally stored replica set configuration does not have a valid entry for the current node; waiting for reconfig or remote heartbeat; Got "NodeNotFound: No host described in new configuration 7 for replica set rs0 maps to this node" while validating...

A request for a retry loop in getaddrinfo was submitted in the thread below but it looks like nothing came of it.

https://jira.mongodb.org/browse/SERVER-40159

My crude workaround used the initConfigMap and a script which delays the startup of the pod by 30 seconds (sleep 30). That provided enough time to have a valid dns entry by the time mongo started and the primary mongo rejoined the replicaset.

A better fix would be to make sure a valid dns entry exists before starting mongo.

[JoshVee]

As @SwannMR indicated, delaying the startup of the mongodb process resolves this problem.

The issue is caused by a race condition with Kubernetes StatefulSets and the headless service DNS resolution. Kubernetes does not create an Endpoint for the headless service until a Pod is "ready".

https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#using-stable-network-identities

As the mongo process tries to call getaddrinfo immediately at startup, there's not enough time for the Endpoints and DNS records to have been updated.

Therefore, to resolve this issue you need to do two things:

1. Disable the readiness probes for the pod (including for the metrics container, if enabled)
   
   
   
   • this ensures the Pod is marked as Ready as soon as it starts
   
   
2. Add a short delay before the mongodb process is started - or better still, wait for DNS resolution before starting.

I resolved this issue by creating a custom mongo image with a new entrypoint that waits for $MONGODB_ADVERTISED_HOSTNAME to resolve before continuing (see the code examples below). I also disabled the readiness probes.

$ kubectl logs -f mongodb-secondary-0 mongodb-secondary
[INFO] Waiting for mongodb-secondary-0.mongodb-headless.prod ... (1)
[INFO] Waiting for mongodb-secondary-0.mongodb-headless.prod ... (2)
[INFO] Waiting for mongodb-secondary-0.mongodb-headless.prod ... (3)
100.96.22.25    mongodb-secondary-0.mongodb-headless.prod.svc.cluster.local
mongodb 05:31:17.16
mongodb 05:31:17.16 Welcome to the Bitnami mongodb container
mongodb 05:31:17.16 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
mongodb 05:31:17.17 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
mongodb 05:31:17.17 Send us your feedback at [email protected]
mongodb 05:31:17.17
mongodb 05:31:17.17 INFO  ==> ** Starting MongoDB setup **
mongodb 05:31:17.18 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 05:31:17.19 INFO  ==> Initializing MongoDB...
mongodb 05:31:17.20 INFO  ==> Deploying MongoDB with persisted data...
mongodb 05:31:17.20 INFO  ==> Writing keyfile for replica set authentication...
mongodb 05:31:17.22 INFO  ==> Loading custom scripts...
mongodb 05:31:17.22 INFO  ==> ** MongoDB setup finished! **
mongodb 05:31:17.23 INFO  ==> ** Starting MongoDB **

---

Helm

values.yaml

readinessProbe:
  enabled: false

image:
  registry: registry.example.com
  repository: mongodb
  tag: 4.2.6-debian-10-r13

metrics:
  enabled: true
  readinessProbe:
    enabled: false

Docker image

Dockerfile

ARG BASE_IMAGE=bitnami/mongodb:4.2.6-debian-10-r13

FROM ${BASE_IMAGE}

COPY entrypoint.sh /entrypoint.sh

ENTRYPOINT [ "/entrypoint.sh", "/opt/bitnami/scripts/mongodb/entrypoint.sh" ]
CMD [ "/opt/bitnami/scripts/mongodb/run.sh" ]

entrypoint.sh

#!/bin/sh -e

# wait for DNS to resolve
if [ -n "$MONGODB_ADVERTISED_HOSTNAME" ]; then
    i=1
    retries=${BOOT_TIMEOUT:-10}

    until getent hosts "$MONGODB_ADVERTISED_HOSTNAME"; do
        echo "[INFO] Waiting for $MONGODB_ADVERTISED_HOSTNAME ... ($i)"

        i=$((i+1))

        if [ $i -gt $retries ]; then
            echo "[ERROR] Unable to resolve $MONGODB_ADVERTISED_HOSTNAME"
            exit 1
        fi

        sleep 1
    done
fi

# shellcheck disable=SC2086
exec "$@"

[javsalgar]

Hi,

Thank you very much for the information. Question, would setting the headless service with the option publishNotReadyAddresses: true fix this race condition?

[JoshVee]

Thanks for the tip @javsalgar

I just tested that flag and it resolves the issue with regards to the readiness probe. However, due to possible delays in DNS resolution, some sort of delay at the start is still needed.

~1. Disable the readiness probes for the pod (including for the metrics container, if enabled)~

1. Add a short delay before the mongodb process is started - or better still, wait for DNS resolution before starting.

[javsalgar]

Hi,

I see, so I guess it makes sense to optionally allow a delay in DNS resolution before continuing.

[javsalgar]

I will forward this for the team for evaluation. As soon as we have more news, I will update this ticket.

[emrahcetiner]

@javsalgar any update?

[juan131]

Hi everyone,

A new major version of MongoDB (8.0.0) was released including refactoring the way to expose MongoDB nodes outside the K8s cluster, so there's one service per node. It also groups every secondary and primary node under the same statefulset. Please feel free to give it a try since it should address this issue.

More info at https://github.com/bitnami/charts/tree/master/bitnami/mongodb#to-800