Project: MongoDB exporter crash

[runningman84]

The mongodb exporter crashes with this error:

$ kubectl logs graylog-mongodb-0 -n logging -c exporter
Listening on :56790 (scheme=HTTP, secured=no, clientValidation=no)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x508228]

goroutine 16 [running]:
panic(0x73d340, 0xc420016020)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/dcu/mongodb_exporter/collector.(*PreloadStats).Export(0x0, 0xc42006ab40)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/metrics.go:363 +0x38
github.com/dcu/mongodb_exporter/collector.(*ReplStats).Export(0xc420246da0, 0xc42006ab40)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/metrics.go:352 +0x90
github.com/dcu/mongodb_exporter/collector.(*MetricsStats).Export(0xc4201d1b80, 0xc42006ab40)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/metrics.go:435 +0x4c7
github.com/dcu/mongodb_exporter/collector.(*ServerStatus).Export(0xc4201d60a0, 0xc42006ab40)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/server_status.go:111 +0x1f9
github.com/dcu/mongodb_exporter/collector.(*MongodbCollector).collectServerStatus(0xc420010ff0, 0xc42015a1a0, 0xc42006ab40, 0x7235e0)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/mongodb_collector.go:85 +0xe3
github.com/dcu/mongodb_exporter/collector.(*MongodbCollector).Collect(0xc420010ff0, 0xc42006ab40)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/mongodb_collector.go:64 +0x2d4
github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc4200fcfe0, 0xc42006ab40, 0x928900, 0xc420010ff0)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x63
created by github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x326

This is the corresponding pod:

$ kubectl describe pod graylog-mongodb-0 -n logging                                                             
Name:               graylog-mongodb-0
Namespace:          logging
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-11-61.eu-central-1.compute.internal/10.0.11.61
Start Time:         Thu, 25 Apr 2019 09:44:10 +0000
Labels:             controller-revision-hash=graylog-mongodb-7986879874
                    kubedb.com/kind=MongoDB
                    kubedb.com/name=graylog-mongodb
                    statefulset.kubernetes.io/pod-name=graylog-mongodb-0
Annotations:        kubernetes.io/limit-ranger:
                      LimitRanger plugin set: memory request for container exporter; memory limit for container exporter; memory request for init container boot...
Status:             Running
IP:                 10.0.11.253
Controlled By:      StatefulSet/graylog-mongodb
Init Containers:
  copy-config:
    Container ID:  docker://6d0d4ba895040161d7cbda2f4ba68af0b7f359b8122b9ef8c2c29971725bc0a0
    Image:         kubedb/busybox
    Image ID:      docker-pullable://kubedb/busybox@sha256:4415a904b1aca178c2450fd54928ab362825e863c0ad5452fd020e92f7a6a47e
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
    Args:
      -c
      set -xe
                        if [ -f "/configdb-readonly/mongod.conf" ]; then
                          cp /configdb-readonly/mongod.conf /data/configdb/mongod.conf
                        else
                          touch /data/configdb/mongod.conf
                        fi

                        if [ -f "/keydir-readonly/key.txt" ]; then
                          cp /keydir-readonly/key.txt /data/configdb/key.txt
                          chmod 600 /data/configdb/key.txt
                        fi
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 25 Apr 2019 09:44:44 +0000
      Finished:     Thu, 25 Apr 2019 09:44:44 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  248Mi
    Requests:
      cpu:        100m
      memory:     248Mi
    Environment:  <none>
    Mounts:
      /data/configdb from config (rw)
      /keydir-readonly from keydir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from graylog-mongodb-token-hdzpk (ro)
      /work-dir from workdir (rw)
  bootstrap:
    Container ID:  docker://fef525e107c673b466b68dfe6c624c6a29c7020149056fa207e214d27d1bedb1
    Image:         kubedb/mongo:4.1.7
    Image ID:      docker-pullable://kubedb/mongo@sha256:7b16b5445302b498170a26f82dbd4cb1db38f1d35056313becf98474d0a8d0d3
    Port:          <none>
    Host Port:     <none>
    Command:
      peer-finder
    Args:
      -on-start=/usr/local/bin/on-start.sh
      -service=graylog-mongodb-gvr
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 25 Apr 2019 09:44:45 +0000
      Finished:     Thu, 25 Apr 2019 09:44:49 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  256Mi
    Requests:
      memory:  32Mi
    Environment:
      POD_NAMESPACE:               logging (v1:metadata.namespace)
      REPLICA_SET:                 graylog
      AUTH:                        true
      MONGO_INITDB_ROOT_USERNAME:  <set to the key 'username' in secret 'graylog-mongodb-auth'>  Optional: false
      MONGO_INITDB_ROOT_PASSWORD:  <set to the key 'password' in secret 'graylog-mongodb-auth'>  Optional: false
    Mounts:
      /data/configdb from config (rw)
      /data/db from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from graylog-mongodb-token-hdzpk (ro)
      /work-dir from workdir (rw)
Containers:
  mongodb:
    Container ID:  docker://e05e03d8ce13da808711c149e7d297a00e731502b0955f823702549c76dd8558
    Image:         kubedb/mongo:4.1.7
    Image ID:      docker-pullable://kubedb/mongo@sha256:7b16b5445302b498170a26f82dbd4cb1db38f1d35056313becf98474d0a8d0d3
    Port:          27017/TCP
    Host Port:     0/TCP
    Command:
      mongod
    Args:
      --dbpath=/data/db
      --auth
      --bind_ip=0.0.0.0
      --port=27017
      --replSet=graylog
      --keyFile=/data/configdb/key.txt
    State:          Running
      Started:      Thu, 25 Apr 2019 09:44:50 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  248Mi
    Requests:
      cpu:      100m
      memory:   248Mi
    Liveness:   exec [mongo --eval db.adminCommand('ping')] delay=0s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [mongo --eval db.adminCommand('ping')] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MONGO_INITDB_ROOT_USERNAME:  <set to the key 'username' in secret 'graylog-mongodb-auth'>  Optional: false
      MONGO_INITDB_ROOT_PASSWORD:  <set to the key 'password' in secret 'graylog-mongodb-auth'>  Optional: false
    Mounts:
      /data/configdb from config (rw)
      /data/db from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from graylog-mongodb-token-hdzpk (ro)
  exporter:
    Container ID:  docker://b079a51c29f9b20c075e8ba5a639bc59b8979c31ce6bc4ff4034cab5c33796ec
    Image:         kubedb/mongodb_exporter:v1.0.0
    Image ID:      docker-pullable://kubedb/mongodb_exporter@sha256:bdca0f7f09860ba7d42cf2d8ada05f4ecb888a5c9d96e36266b9809809bd4700
    Port:          56790/TCP
    Host Port:     0/TCP
    Args:
      --web.listen-address=:56790
      --web.metrics-path=/metrics
      --mongodb.uri=mongodb://$(MONGO_INITDB_ROOT_USERNAME):$(MONGO_INITDB_ROOT_PASSWORD)@127.0.0.1:27017
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 25 Apr 2019 10:48:37 +0000
      Finished:     Thu, 25 Apr 2019 10:48:43 +0000
    Ready:          False
    Restart Count:  17
    Limits:
      memory:  256Mi
    Requests:
      memory:  32Mi
    Environment:
      MONGO_INITDB_ROOT_USERNAME:  <set to the key 'username' in secret 'graylog-mongodb-auth'>  Optional: false
      MONGO_INITDB_ROOT_PASSWORD:  <set to the key 'password' in secret 'graylog-mongodb-auth'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from graylog-mongodb-token-hdzpk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-graylog-mongodb-0
    ReadOnly:   false
  workdir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  keydir:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  graylog-mongodb-keyfile
    Optional:    false
  config:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  graylog-mongodb-token-hdzpk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  graylog-mongodb-token-hdzpk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                  From                                                  Message
  ----     ------   ----                 ----                                                  -------
  Warning  BackOff  20s (x293 over 65m)  kubelet, ip-10-0-11-61.eu-central-1.compute.internal  Back-off restarting failed container

Maybe the exporter form this fork would work better?
https://github.com/percona/mongodb_exporter

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.98. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[michaelmass]

We had the same problem and it came from the mongo version 4.1.7. We solved this issue by just reverting to 4.0.5.

[richerlariviere]

I have reproduction steps for that issue on Azure AKS:

1. Setup new AKS cluster (I chose to use 1 node)
2. Install the following charts

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ mkdir charts

# Download our charts tarballs locally
$ helm fetch appscode/kubedb -d charts
$ helm fetch appscode/kubedb-catalog -d charts
$ helm fetch stable/prometheus-operator -d charts

# Install prometheus-operator v6.7.2
$ helm install --name prometheus-operator charts/prometheus-operator-6.7.2.tgz \
--values values/prometheus-operator.yaml 

# Install kubedb 0.12.0
$ helm install charts/kubedb-0.12.0.tgz \
--set monitoring.enabled=true,monitoring.agent="prometheus.io/coreos-operator"

# Install kubedb-catalog 0.12.0 
$ helm install charts/kubedb-catalog-0.12.0.tgz

1. Create the following database (based on the doc)

apiVersion: kubedb.com/v1alpha1
kind: MongoDB
metadata:
  name: coreos-prom-mgo
  namespace: default
spec:
  version: "4.1.7-v1"
  terminationPolicy: WipeOut
  storage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  monitor:
    agent: prometheus.io/coreos-operator
    prometheus:
      namespace: default
      labels:
        release: prometheus-operator 
      interval: 10s

1. We can see the the failed deployment and the same error mentioned by OP.

# Stats service is created
$ kubectl get svc | grep coreos-prom-mgo
coreos-prom-mgo                           ClusterIP   10.0.104.19    <none>        27017/TCP           44m
coreos-prom-mgo-gvr                       ClusterIP   None           <none>        27017/TCP           44m
coreos-prom-mgo-stats                     ClusterIP   10.0.252.181   <none>        56790/TCP           42m

# Our mongodb is crashing
$ kubectl get pods | grep coreos-prom-mgo
coreos-prom-mgo-0                                        1/2     Error     4          2m34s

# Get logs from the prometheus exporter
$ kubectl logs coreos-prom-mgo-0 -c exporter
Listening on :56790 (scheme=HTTP, secured=no, clientValidation=no)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x508228]

goroutine 13 [running]:
panic(0x73d340, 0xc420014020)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/dcu/mongodb_exporter/collector.(*PreloadStats).Export(0x0, 0xc4200e8240)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/metrics.go:363 +0x38
github.com/dcu/mongodb_exporter/collector.(*ReplStats).Export(0xc420203060, 0xc4200e8240)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/metrics.go:352 +0x90
github.com/dcu/mongodb_exporter/collector.(*MetricsStats).Export(0xc4201fd0c0, 0xc4200e8240)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/metrics.go:435 +0x4c7
github.com/dcu/mongodb_exporter/collector.(*ServerStatus).Export(0xc4201561e0, 0xc4200e8240)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/server_status.go:111 +0x1f9
github.com/dcu/mongodb_exporter/collector.(*MongodbCollector).collectServerStatus(0xc420082f00, 0xc420176680, 0xc4200e8240, 0x7235e0)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/mongodb_collector.go:85 +0xe3
github.com/dcu/mongodb_exporter/collector.(*MongodbCollector).Collect(0xc420082f00, 0xc4200e8240)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/collector/mongodb_collector.go:64 +0x2d4
github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc420014100, 0xc4200e8240, 0x928900, 0xc420082f00)
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x63
created by github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
        /Users/dc/code/go/src/github.com/dcu/mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x326

[Adriien-M]

This pull request https://github.com/kubedb/installer/pull/16 has been revert?
Because I'm using Mongo 4.1.7 and exporter crashes, I was expecting to use percona exporter but kubedb exporter is still used in the catalog...

In fact, this PR is not included in latest version of KubeDB...

[walk1ng]

Because I'm using Mongo 4.1.7 and exporter crashes, I was expecting to use percona exporter but kubedb exporter is still used in the catalog...

Hi @Adriien-M
Yep, I'm aware of it too. How did you deal with this problem?

[walk1ng]

Hi @tamalsaha, did the PR to use percona mongodb exporter from 0.13.0 has been revert?
How to handle the crush issue from exporter container?

• Thanks

[tamalsaha]

cc: @faem

[faem]

Hi @walk1ng,

What kubedb catalog version are you using?

Percona exporter was added in v0.14.0.

[walk1ng]

Hi @walk1ng,

What kubedb catalog version are you using?

Percona exporter was added in v0.14.0.

Hi @faem
Thanks for your response.
For some reason, we are working with kubedb catalog v0.13.0-rc.0 and mongodb 4.1.13.
I see the fix PRs are related to v0.13.0.

[faem]

Although the PR title says so, for some reasons it wasn't added in v0.13.0.

[walk1ng]

@faem

Although the PR title says so, for some reasons it wasn't added in v0.13.0.

@faem
Do you have any workaround to fix the exporter crush issue in v0.13.0? It's hard to upgrade to v0.14.0+ in our environment currently.