Charts: [bitnami/mongodb] can't connect to mongodb replicaset externally

[obeyda]

Which chart:
bitnami/[email protected]

Describe the bug
i used the external service in mongodb chart (see values) but I couldn't connect to the replicaset using mongo shell

• error from mongo shell (remote)

mongo "mongodb://root:$MONGODB_ROOT_PASSWORD@<IP1>,<IP2>,<IP3>/?authSource=admin&replicaSet=rs0&keepAlive=true&poolSize=30&autoReconnect=true&socketTimeoutMS=360000&connectTimeoutMS=360000" --verbose
MongoDB shell version v4.2.8
connecting to: mongodb://<IP1>:27017,<IP2>:27017,<IP3>:27017/?authSource=admin&autoReconnect=true&compressors=disabled&connectTimeoutMS=360000&gssapiServiceName=mongodb&keepAlive=true&poolSize=30&replicaSet=rs0&socketTimeoutMS=360000
2020-08-14T21:34:43.563+0100 D1 NETWORK  [js] Starting up task executor for monitoring replica sets in response to request to monitor set: rs0/<IP1>:27017,<IP2>:27017,<IP3>:27017
2020-08-14T21:34:43.563+0100 I  NETWORK  [js] Starting new replica set monitor for rs0/<IP1>:27017,<IP2>:27017,<IP3>:27017
2020-08-14T21:34:43.563+0100 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to <IP1>:27017
2020-08-14T21:34:43.563+0100 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to <IP3>:27017
2020-08-14T21:34:43.563+0100 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to <IP2>:27017
2020-08-14T21:34:44.971+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:44.971+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 1408ms
2020-08-14T21:34:45.179+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:45.179+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 116ms
2020-08-14T21:34:45.679+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:45.679+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 115ms
2020-08-14T21:34:46.176+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:46.176+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 113ms
2020-08-14T21:34:46.679+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:46.679+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 116ms
2020-08-14T21:34:47.503+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:47.503+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 440ms
2020-08-14T21:34:47.677+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:47.677+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 114ms
2020-08-14T21:34:48.755+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:48.755+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 692ms
2020-08-14T21:34:49.180+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:49.180+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 117ms
2020-08-14T21:34:49.677+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:49.677+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 114ms
2020-08-14T21:34:50.178+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:50.178+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 115ms
2020-08-14T21:34:50.729+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:50.729+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 166ms
2020-08-14T21:34:51.527+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:51.527+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 464ms
2020-08-14T21:34:51.747+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:51.747+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 184ms
2020-08-14T21:34:52.811+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:52.811+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 748ms
2020-08-14T21:34:53.167+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:53.167+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 104ms
2020-08-14T21:34:53.667+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:53.667+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 104ms
2020-08-14T21:34:54.213+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:54.213+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 150ms
2020-08-14T21:34:55.175+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:55.175+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 612ms
2020-08-14T21:34:55.669+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:55.669+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 106ms
2020-08-14T21:34:56.169+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:56.169+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 106ms
2020-08-14T21:34:56.707+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:56.707+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 144ms
2020-08-14T21:34:57.207+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:57.207+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 144ms
2020-08-14T21:34:57.676+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:57.676+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 113ms
2020-08-14T21:34:58.226+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:58.226+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 163ms
2020-08-14T21:34:58.669+0100 W  NETWORK  [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs0
2020-08-14T21:34:58.669+0100 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 106ms
2020-08-14T21:34:58.669+0100 D1 -        [js] User Assertion: InternalError: connect failed to replica set rs0/<IP1>:27017,<IP2>:27017,<IP3>:27017 src/mongo/scripting/mozjs/mongo.cpp 832
2020-08-14T21:34:58.670+0100 E  QUERY    [js] Error: connect failed to replica set rs0/<IP1>:27017,<IP2>:27017,<IP3>:27017 :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-08-14T21:34:58.670+0100 D1 -        [js] User Assertion: Location12513: connect failed src/mongo/shell/shell_utils.cpp 418
2020-08-14T21:34:58.670+0100 I  QUERY    [js] MozJS GC prologue heap stats -  total: 4385355 limit: 0
2020-08-14T21:34:58.673+0100 I  QUERY    [js] MozJS GC epilogue heap stats -  total: 595 limit: 0
2020-08-14T21:34:58.673+0100 D1 -        [main] User Assertion: Location12513: connect failed src/mongo/scripting/mozjs/proxyscope.cpp 320
2020-08-14T21:34:58.674+0100 F  -        [main] exception: connect failed
2020-08-14T21:34:58.674+0100 E  -        [main] exiting with code 1

• when i execute db.isMaster() from one of the replicas:

rs0:OTHER> db.isMaster()
{
        "setName" : "rs0",
        "ismaster" : false,
        "secondary" : false,
        "info" : "Does not have a valid replica set config",
        "isreplicaset" : true,
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2020-08-14T20:37:57.477Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 466,
        "minWireVersion" : 0,
        "maxWireVersion" : 8,
        "readOnly" : false,
        "ok" : 1
}

• when i execute rs.conf().members

rs0:OTHER> rs.conf().members
[
        {
                "_id" : 0,
                "host" : "10.244.8.185:27017",
                "arbiterOnly" : false,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 5,
                "tags" : {

                },
                "slaveDelay" : NumberLong(0),
                "votes" : 1
        },
        {
                "_id" : 1,
                "host" : "mongodb-arbiter-0.mongodb-arbiter-headless.namespace.svc.cluster.local:27017",
                "arbiterOnly" : true,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 0,
                "tags" : {

                },
                "slaveDelay" : NumberLong(0),
                "votes" : 1
        },
        {
                "_id" : 2,
                "host" : "10.244.5.142:27017",
                "arbiterOnly" : false,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 1,
                "tags" : {

                },
                "slaveDelay" : NumberLong(0),
                "votes" : 1
        },
        {
                "_id" : 3,
                "host" : "10.244.4.232:27017",
                "arbiterOnly" : false,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 1,
                "tags" : {

                },
                "slaveDelay" : NumberLong(0),
                "votes" : 1
        }
] 

To Reproduce
Steps to reproduce the behavior:

1. install chart with the following values, this is a diff between the values used and the values in values-production.yaml

--- values-mongodb-bitnami-prod 2020-08-14 22:12:37.249665394 +0100
+++ values-demo.yaml    2020-08-14 21:20:49.972437911 +0100
@@ -20,7 +20,7 @@
   ## Bitnami MongoDB image tag
   ## ref: https://hub.docker.com/r/bitnami/mongodb/tags/
   ##
-  tag: 4.2.8-debian-10-r47
+  tag: 4.2.8-debian-10-r50
   ## Specify a imagePullPolicy
   ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
   ##
@@ -72,17 +72,17 @@
   ## MongoDB root password
   ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#setting-the-root-password-on-first-run
   ##
-  rootPassword: ""
+  rootPassword: r00tPassword
   ## MongoDB custom user and database
   ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#creating-a-user-and-database-on-first-run
   ##
-  # username: username
-  # password: password
-  # database: database
+  username: appUser
+  password: Password
+  database: admin
   ## Key used for replica set authentication
   ## Ignored when mongodb.architecture=standalone
   ##
-  # replicaSetKey: key
+  replicaSetKey: MC3UN74uaL

   ## Existing secret with MongoDB credentials
   ## NOTE: When it's set the previous parameters are ignored.
@@ -98,7 +98,7 @@
 ## Ignored when mongodb.architecture=standalone
 ## Ignored when externalAccess.enabled=true
 ##
-replicaSetHostnames: true
+replicaSetHostnames: false

 ## Whether enable/disable IPv6 on MongoDB
 ## ref: https://github.com/bitnami/bitnami-docker-mongodb/blob/master/README.md#enabling/disabling-ipv6
@@ -188,7 +188,8 @@
 ## extraFlags:
 ##  - "--wiredTigerCacheSizeGB=2"
 ##
-extraFlags: []
+extraFlags:
+  - "--wiredTigerCacheSizeGB=3"

 ## Additional environment variables to set
 ## E.g:
@@ -217,7 +218,7 @@
 ## Number of MongoDB replicas to deploy.
 ## Ignored when mongodb.architecture=standalone
 ##
-replicaCount: 4
+replicaCount: 3

 ## StrategyType for MongoDB statefulset
 ## It can be set to RollingUpdate or Recreate by default.
@@ -231,7 +232,16 @@
 ## Affinity for pod assignment. Evaluated as a template.
 ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
 ##
-affinity: {}
+affinity:
+  podAntiAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+    - labelSelector:
+        matchExpressions:
+        - key: app.kubernetes.io/instance
+          operator: In
+          values:
+          - my-db-release-v4
+      topologyKey: "kubernetes.io/hostname"

 ## Node labels for pod assignment. Evaluated as a template.
 ## ref: https://kubernetes.io/docs/user-guide/node-selection/
@@ -287,12 +297,12 @@
   # choice for the user. This also increases chances charts run on environments with little
   # resources, such as Minikube. If you do want to specify resources, uncomment the following
   # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
-  limits: {}
-  #   cpu: 100m
-  #   memory: 128Mi
-  requests: {}
-  #   cpu: 100m
-  #   memory: 128Mi
+  limits: 
+    cpu: 2000m
+    memory: 6Gi
+  requests: 
+    cpu: 1000m
+    memory: 5Gi

 ## MongoDB pods' liveness and readiness probes. Evaluated as a template.
 ## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
@@ -363,7 +373,7 @@
   create: true
   ## Min number of pods that must still be available after the eviction
   ##
-  minAvailable: 1
+  minAvailable: 3
   ## Max number of pods that can be unavailable after the eviction
   ##
   # maxUnavailable: 1
@@ -385,14 +395,14 @@
   ## If undefined (the default) or set to null, no storageClassName spec is
   ## set, choosing the default provisioner.
   ##
-  # storageClass: "-"
+  storageClass: "managed-premium"
   ## PV Access Mode
   ##
   accessModes:
     - ReadWriteOnce
   ## PVC size
   ##
-  size: 8Gi
+  size: 100Gi
   ## PVC annotations
   ##
   annotations: {}
@@ -445,7 +455,7 @@
 externalAccess:
   ## Enable Kubernetes external cluster access to MongoDB nodes
   ##
-  enabled: false
+  enabled: true
   ## External IPs auto-discovery configuration
   ## An init container is used to auto-detect LB IPs or node ports by querying the K8s API
   ## Note: RBAC might be required
@@ -453,14 +463,14 @@
   autoDiscovery:
     ## Enable external IP/ports auto-discovery
     ##
-    enabled: false
+    enabled: true
     ## Bitnami Kubectl image
     ## ref: https://hub.docker.com/r/bitnami/kubectl/tags/
     ##
     image:
       registry: docker.io
       repository: bitnami/kubectl
-      tag: 1.18.6-debian-10-r15
+      tag: 1.18.8-debian-10-r0
       ## Specify a imagePullPolicy
       ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
       ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
@@ -740,14 +752,14 @@
   ## binding Kafka ServiceAccount to a role
   ## that allows Kafka pods querying the K8s API
   ##
-  create: false
+  create: true

 ## Init Container paramaters
 ## Change the owner and group of the persistent volume(s) mountpoint(s) to 'runAsUser:fsGroup' on each component
 ## values from the securityContext section of the component
 ##
 volumePermissions:
-  enabled: false
+  enabled: true
   ## Bitnami Minideb image
   ## ref: https://hub.docker.com/r/bitnami/minideb/tags/
   ##

2- as mentioned in the release notes: get the external IPs and try to connect using the user and password

Expected behavior
i can connect to the replicaset, and when i shell into a replicaset directly i see either rs0:primary or rs0:secondary.

Version of Helm and Kubernetes:

• Output of helm version:

Client: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"archive", BuildDate:"2020-07-01T16:28:46Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"8a6cdee2dab2329284444e9448bdf8ab165e1c5c", GitTreeState:"clean", BuildDate:"2020-06-24T18:14:29Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

* additional context *
the arbiter pod keeps crashing with error message:

 Node mongodb-0.mongodb-headless.namespace.svc.cluster.local did not become available

finally, I am using Azure (AKS).

[dani8art]

Hi @obeyda I've just launched bitnami/mongodb using these values with a few changes.

--- bitnami/mongodb/values-production.yaml      2020-08-18 10:19:33.527171445 +0000
+++ bitnami/mongodb/values-production.my.yaml   2020-08-18 10:19:38.507017427 +0000
@@ -240,7 +240,7 @@
             - key: app.kubernetes.io/instance
               operator: In
               values:
-                - my-db-release-v4
+                - "{{ .Release.Name }}"
         topologyKey: "kubernetes.io/hostname"

 ## Node labels for pod assignment. Evaluated as a template.
@@ -297,12 +297,12 @@
   # choice for the user. This also increases chances charts run on environments with little
   # resources, such as Minikube. If you do want to specify resources, uncomment the following
   # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
-  limits:
-    cpu: 2000m
-    memory: 6Gi
-  requests:
-    cpu: 1000m
-    memory: 5Gi
+  # limits:
+  #   cpu: 2000m
+  #   memory: 6Gi
+  # requests:
+  #   cpu: 1000m
+  #   memory: 5Gi

 ## MongoDB pods' liveness and readiness probes. Evaluated as a template.
 ## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
@@ -395,14 +395,14 @@
   ## If undefined (the default) or set to null, no storageClassName spec is
   ## set, choosing the default provisioner.
   ##
-  storageClass: "managed-premium"
+  # storageClass: 'managed-premium'
   ## PV Access Mode
   ##
   accessModes:
     - ReadWriteOnce
   ## PVC size
   ##
-  size: 100Gi
+  size: 8Gi
   ## PVC annotations
   ##
   annotations: {}

$ helm upgrade --install mongodb bitnami/mongodb -f bitnami/mongodb/values-production.yaml
$ kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
mongodb-0           2/2     Running   0          5m1s
mongodb-1           2/2     Running   0          3m34s
mongodb-2           0/2     Pending   0          2m40s
mongodb-arbiter-0   1/1     Running   0          5m2s

NOTE: mongodb-2 is pending because of the antiaffinity, it doesn't matter we will use the rest of nodes

$ docker run --rm -ti bitnami/mongodb -- mongo "mongodb://root:r00tPassword@<ip1>,<ip2>/?authSource=admin&replicaSet=rs0&keepAlive=true&poolSize=30&autoReconnect=true&socketTimeoutMS=360000&connectTimeoutMS=360000" --verbose
mongodb 10:12:20.85 
mongodb 10:12:20.85 Welcome to the Bitnami mongodb container
mongodb 10:12:20.85 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
mongodb 10:12:20.85 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
mongodb 10:12:20.85 

MongoDB shell version v4.2.8
connecting to: mongodb://<ip1>:27017,<ip2>:27017/?authSource=admin&autoReconnect=true&compressors=disabled&connectTimeoutMS=360000&gssapiServiceName=mongodb&keepAlive=true&poolSize=30&replicaSet=rs0&socketTimeoutMS=360000
2020-08-18T10:12:20.912+0000 D1 NETWORK  [js] Starting up task executor for monitoring replica sets in response to request to monitor set: rs0/<ip1>:27017,<ip2>:27017
2020-08-18T10:12:20.912+0000 I  NETWORK  [js] Starting new replica set monitor for rs0/<ip1>:27017,<ip2>:27017
2020-08-18T10:12:20.912+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to <ip1>:27017
2020-08-18T10:12:20.912+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to <ip2>:27017
2020-08-18T10:12:20.956+0000 I  NETWORK  [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for rs0 is rs0/<ip2>:27017,<ip1>:27017
2020-08-18T10:12:20.956+0000 D1 NETWORK  [js] creating new connection to:<ip1>:27017
2020-08-18T10:12:20.956+0000 D1 NETWORK  [ReplicaSetMonitor-TaskExecutor] Refreshing replica set rs0 took 44ms
2020-08-18T10:12:20.971+0000 D1 NETWORK  [js] connected to server <ip1>:27017
2020-08-18T10:12:20.985+0000 D1 NETWORK  [js] connected connection!
Implicit session: session { "id" : UUID("22a3acbb-69f5-4ed7-8b83-e929c5f955a5") }
MongoDB server version: 4.2.8
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
2020-08-18T10:12:21.081+0000 I  STORAGE  [main] In File::open(), ::open for '//.mongorc.js' failed with Permission denied
Server has startup warnings: 
2020-08-18T09:58:33.228+0000 I  STORAGE  [initandlisten] 
2020-08-18T09:58:33.228+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-08-18T09:58:33.228+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

rs0:PRIMARY>

rs0:PRIMARY> db.isMaster()
{
        "hosts" : [
                "<ip1>:27017",
                "<ip2>:27017"
        ],
        "arbiters" : [
                "mongodb-arbiter-0.mongodb-arbiter-headless.default.svc.cluster.local:27017"
        ],
        "setName" : "rs0",
        "setVersion" : 3,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "<ip1>:27017",
        "me" : "<ip1>:27017",
        "electionId" : ObjectId("7fffffff0000000000000002"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1597745584, 1),
                        "t" : NumberLong(2)
                },
                "lastWriteDate" : ISODate("2020-08-18T10:13:04Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1597745584, 1),
                        "t" : NumberLong(2)
                },
                "majorityWriteDate" : ISODate("2020-08-18T10:13:04Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2020-08-18T10:13:12.747Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 141,
        "minWireVersion" : 0,
        "maxWireVersion" : 8,
        "readOnly" : false,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1597745584, 1),
                "signature" : {
                        "hash" : BinData(0,"8wLXUtg2Lg0aK3p6bzbOd86yRgY="),
                        "keyId" : NumberLong("6862261281101971460")
                }
        },
        "operationTime" : Timestamp(1597745584, 1)
}

• Could you try to helm del <your_release> and kubectl delete pvc --all in order to deploy a very fresh instance?

• If that doesn't work, could you share with us the following?

  • kubectl describe pod mongodb-0
  • kubectl logs mongodb-0
  • kubectl describe pod mongodb-arbiter-0
  • kubectl logs mongodb-arbiter-0

In addition, you could also add this parameter --set image.debug=true

Thanks!

[obeyda]

After deleting the pvc and the disk from azure portal it worked, it appears that aks was attaching the same disk even after deleting the pvc.
I need to investigate the problem further, but i confirm that on a clean install i can connect to the replicaSet from an external host.

[dani8art]

Good to hear it worked! please, don't hesitate to contact us if you have any other problems. Thanks!

[maitrungduc1410]

For some folks if this help:
I'm having a quite same problem, I'm using AKS and mount my volume using Azure FileShare, first I run a replica set without creating a custom user, empty root password. Then I upgrade using helm upgrade... with a custom username + password, and a root password, then it breaks, I can't connect to replica set from outside.

After removing my volume on FileShare and recreating the replica set, everything works fine.

[dani8art]

Hi @maitrungduc1410 when upgrading the chart you should take care of supplying the same root-password and replicasetKey as they are randomly created even if you don't supply them for the first time, could you try this?

$ helm install mongodb bitnami/mongodb --set architecture=replicaset

$ export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)
$ export MONGODB_REPLICA_SET_KEY=$(kubectl get secret --namespace default mongodb -o jsonpath="{.data.mongodb-replica-set-key}" | base64 --decode)
$ helm upgrade --install mongodb bitnami/mongodb --set architecture=replicaset --set auth.password=my-pasword,auth.username=my-user,auth.database=my-data --set auth.rootPassword=$MONGODB_ROOT_PASSWORD --set auth.replicaSetKey=$MONGODB_REPLICA_SET_KEY

[maitrungduc1410]

Hi @dani8art ,
Thanks for your explanation, now I know the reason my setup was broken. Anyway, my Replicaset now is working fine.

I have another question, may you answer it: my project, first it's deployed with Docker under an Ubuntu VM, the project works ok, queries to MongoDB quite fast

After that because the project scale, I've moved it to K8S, I run 1 standalone MongoDB instance, its volume is mounted to Azure FileShare, and I started seeing my queries getting slow (about 2 times slower when query on a database with ~ 100K documents), now I move it to a MongoDB replicaset already with 4 instances, I hope it could improve read capacity, but seems it's not. With some queries, they make the MongoDB primary instance's CPU usage goes too high and cause bad performance.

I'm wondering: is that in the beginning when deploy under an Ubuntu VM, my DB volume is on same machine, read capacity is fast. After moving to K8S because my Volume is mounted from Azure FileShare and that makes Mongo has poor performance on reading (seems same with writing, but not significant)?

Thank you

P/s: sorry I don't want to create an issue, because my question seems not a bug or feature request

[dani8art]

is that in the beginning when deploy under an Ubuntu VM, my DB volume is on same machine, read capacity is fast. After moving to K8S because my Volume is mounted from Azure FileShare and that makes Mongo has poor performance on reading (seems same with writing, but not significant)?

Yes, I think so. Anyway, you can try to increase your resource like CPU, memory, or even change the storage class in Azure and continue making more performance tests. It seems to be related to the file system but I would do more tests changing these parameters.