Kubeapps: Kubeapps MongoDB Failed to connect to 127.0.0.1:27017

[yoshi1979]

hello. installing Kubeapps on VMware PKS with Kubeapps up command. the mongodb pod in not working. getting:
ubuntu@ubuntu:~/apps$ kubectl describe pod mongodb-8698b94dd-4vr9x -n kubeapps
Name: mongodb-8698b94dd-4vr9x
Namespace: kubeapps
Node: 7309ca56-06f2-429c-bf18-8caed82df198/172.15.0.3
Start Time: Mon, 13 Aug 2018 16:07:17 -0400
Labels: app=mongodb
created-by=kubeapps
name=mongodb
pod-template-hash=425465088
Annotations:
Status: Running
IP: 172.16.5.6
Controlled By: ReplicaSet/mongodb-8698b94dd
Containers:
mongodb:
Container ID: docker://37927795f74668a59be7fa693ed34a2de2e67111869a8ed0ed527e17ce5e3a9a
Image: bitnami/mongodb:3.6.5-r31
Image ID: docker-pullable://bitnami/mongodb@sha256:65ee9d8b0b2bce6e8fc2ae72e878dbb6577ccc31414f93be0b8395f4382ca31d
Port: 27017/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 100
Started: Mon, 13 Aug 2018 16:12:20 -0400
Finished: Mon, 13 Aug 2018 16:12:25 -0400
Ready: False
Restart Count: 6
Requests:
cpu: 100m
memory: 256Mi
Liveness: exec [mongo --eval db.adminCommand('ping')] delay=30s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [mongo --eval db.adminCommand('ping')] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
MONGODB_ROOT_PASSWORD: Optional: false
Mounts:
/bitnami/mongodb from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bkdkf (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-bkdkf:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bkdkf
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulMountVolume 5m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 MountVolume.SetUp succeeded for volume "data"
Normal SuccessfulMountVolume 5m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 MountVolume.SetUp succeeded for volume "default-token-bkdkf"
Normal Scheduled 5m default-scheduler Successfully assigned mongodb-8698b94dd-4vr9x to 7309ca56-06f2-429c-bf18-8caed82df198
Warning Unhealthy 5m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:07:42.558+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:07:42.560+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:07:53.211+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:07:53.211+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:02.546+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:02.546+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Liveness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:03.865+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:03.866+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:12.408+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:12.408+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Liveness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:14.065+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:14.066+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:22.494+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:22.495+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Liveness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:23.914+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:23.915+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 4m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:32.901+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:32.903+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 3m (x2 over 4m) kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 (combined from similar events): Readiness probe failed: MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
2018-08-13T20:08:52.386+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-08-13T20:08:52.387+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:6
exception: connect failed
Normal Killing 3m kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Killing container with id docker://mongodb:Container failed liveness probe.. Container will be killed and recreated.
Normal Started 3m (x3 over 5m) kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Started container
Normal Created 3m (x3 over 5m) kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Created container
Warning BackOff 3m (x2 over 3m) kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Back-off restarting failed container
Normal Pulled 25s (x7 over 5m) kubelet, 7309ca56-06f2-429c-bf18-8caed82df198 Container image "bitnami/mongodb:3.6.5-r31" already present on machine

[prydonius]

Hi @yoshi1979, couple of questions:

• How have you deployed Kubeapps?
• What are the logs of the failing MongoDB pod - kubectl logs -f pod mongodb-8698b94dd-4vr9x -n kubeapps

[migmartri]

Hi @yoshi1979

In the meantime you might be interested in looking into this other issue which contains a workaround that could potentially be useful to you https://github.com/bitnami/bitnami-docker-mongodb/issues/94#issuecomment-396116710

[yoshi1979]

here is the mongodb log:

ubuntu@ubuntu:~/apps$ kubectl log -f pod/mongodb-8698b94dd-b8gmg -n kubeapps
W0814 02:18:22.028490 29102 cmd.go:353] log is DEPRECATED and will be removed in a future version. Use logs instead.

Welcome to the Bitnami mongodb container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues

nami INFO Initializing mongodb
mongodb INFO ==> Deploying MongoDB with persisted data...
mongodb INFO ==> No injected configuration files found. Creating default config files...
mongodb INFO
mongodb INFO ########################################################################
mongodb INFO Installation parameters for mongodb:
mongodb INFO Persisted data and properties have been restored.
mongodb INFO Any input specified will not take effect.
mongodb INFO This installation requires no credentials.
mongodb INFO ########################################################################
mongodb INFO
nami INFO mongodb successfully initialized
INFO ==> Starting mongodb...
INFO ==> Starting mongod...
2018-08-14T06:18:29.641+0000 I CONTROL [initandlisten] MongoDB starting : pid=41 port=27017 dbpath=/opt/bitnami/mongodb/data/db 64-bit host=mongodb-8698b94dd-b8gmg
2018-08-14T06:18:29.641+0000 I CONTROL [initandlisten] db version v3.6.5
2018-08-14T06:18:29.642+0000 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618
2018-08-14T06:18:29.642+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2018-08-14T06:18:29.642+0000 I CONTROL [initandlisten] allocator: tcmalloc
2018-08-14T06:18:29.642+0000 I CONTROL [initandlisten] modules: none
2018-08-14T06:18:29.642+0000 I CONTROL [initandlisten] build environment:
2018-08-14T06:18:29.643+0000 I CONTROL [initandlisten] distmod: debian81
2018-08-14T06:18:29.643+0000 I CONTROL [initandlisten] distarch: x86_64
2018-08-14T06:18:29.643+0000 I CONTROL [initandlisten] target_arch: x86_64
2018-08-14T06:18:29.643+0000 I CONTROL [initandlisten] options: { config: "/opt/bitnami/mongodb/conf/mongodb.conf", net: { bindIpAll: true, ipv6: true, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: "/opt/bitnami/mongodb/tmp" } }, processManagement: { fork: false, pidFilePath: "/opt/bitnami/mongodb/tmp/mongodb.pid" }, security: { authorization: "disabled" }, setParameter: { enableLocalhostAuthBypass: "true" }, storage: { dbPath: "/opt/bitnami/mongodb/data/db", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: true } }
2018-08-14T06:18:29.651+0000 I STORAGE [initandlisten] exception in initAndListen std::exception: open: Address family not supported by protocol, terminating
2018-08-14T06:18:29.651+0000 I CONTROL [initandlisten] now exiting
2018-08-14T06:18:29.651+0000 I CONTROL [initandlisten] shutting down with code:100

[andresmgot]

this is probably related to https://github.com/kubeapps/kubeapps/issues/340. Is IPv6 forced in your cluster? If that's the case you need to use the latest version of Kubeapps. Please clean up your current installation and use Helm to install Kubeapps as stated in the quickstart guide. That should solve that issue.

[prydonius]

@yoshi1979 if this is still an issue for you, please feel free to reopen!

[roie9876]

i was able to resolve the issue after disable IPv6 for MongoDB POD:
you can edit the mongoDB deployment in kubeapps and add this values:

name: MONGODB_ENABLE_IPV6
value: “no”
image: docker.io/bitnami/mongodb:4.0.1-debian-9-r12