Mongo: Can't stop mongo docker container

Created on 12 Dec 2015 · 28Comments · Source: docker-library/mongo

I am having trouble to stop running mongo docker container. I got following message when I run "docker stop [container name]", Anybody having same issue? the mongodb itself seems to be working fine. just can't stop the container.. so I have to restart VM in order to stop the mongo container.
I am using version 2.6 Dockerfile and build and run with docker-compose.

ERROR: for [container name]  HTTPSConnectionPool(host='X.X.X.X', port=2376): Read timed out. (read timeout=70)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

It seems it is related to http connection error with port 2376, but I couldn't figure out why it is happening, other container built and run by docker-compose works just fine and stopped precisely.

Source

sunao-uehara

All 28 comments

:+1:

alexw23 on 18 Dec 2015

:thumbsup:

I'm having this problem too. Sometimes it happens in other custom containers (for example a PHP container linking MongoDB).

fer2d2 on 31 Dec 2015

I am having this problem also. Would like to debug, but I am too new to the docker ecosystem.

Only restarting the VM works.

mongo 2.4

here is the output from a --verbose kill:

david$ docker-compose --verbose kill
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Trying /Users/david/.docker/config.json
docker.auth.auth.load_config: File doesn't exist
docker.auth.auth.load_config: Trying /Users/david/.dockercfg
docker.auth.auth.load_config: File doesn't exist - returning empty config
compose.cli.command.get_client: docker-compose version 1.5.2, build 7240ff3
docker-py version: 1.5.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
compose.cli.command.get_client: Docker base_url: https://192.168.99.100:2376
compose.cli.command.get_client: Docker version: KernelVersion=4.1.13-boot2docker, Os=linux, BuildTime=Fri Nov 20 17:56:04 UTC 2015, ApiVersion=1.21, Version=1.9.1, GitCommit=a34a1d5, Arch=amd64, GoVersion=go1.4.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=brancheserver', u'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- (u'8f5de3958655a551b1262e27e17c5d494567a8e32ddd972f0ae538a79bc50b77')
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {u'AppArmorProfile': u'',
 u'Args': [u'mongod'],
 u'Config': {u'AttachStderr': False,
             u'AttachStdin': False,
             u'AttachStdout': False,
             u'Cmd': [u'mongod'],
             u'CpuShares': 0,
             u'Cpuset': u'',
             u'Domainname': u'',
             u'Entrypoint': [u'/entrypoint.sh'],
...
Killing brancheserver_dockerized_mongodb_1 ...
Killing brancheserver_dockerized_mongodb_1 ... done

ERROR: for brancheserver_dockerized_mongodb_1  HTTPSConnectionPool(host='192.168.99.100', port=2376): Read timed out. (read timeout=60)
ERROR: compose.cli.main.main: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

davidovich on 31 Dec 2015

@sunao-uehara consider fixing the typo in the issue title, there are surely others with the same problem.

davidovich on 31 Dec 2015

@davidovich

Thanks, I just fixed typo :neutral_face:
Regarding current situation of issue, I updated mongo version 2.6 to 2.6.1, then problem disappeared... I just updated version only. So still I am not sure what the fundamental issues is.

sunao-uehara on 31 Dec 2015

Any ideas on reproducing this so that we can find the cause? Are you all on OSX? Is there a common docker version (docker info)? Maybe a common docker-compose configuration? Has it happened on other containers besides mongo?

yosifkit on 5 Jan 2016

👍1

@yosifkit yes, I am on OSX. I have updated to the mongo 2.6 image and this still happens. Hopefully this can provide you with better insights:

Using docker-compose 1.5.2, and docker info:

Containers: 8
Images: 73
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 89
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: YAUF:7G7V:ZLQJ:IXXG:2CRC:KPVH:O26R:YYLK:FOUW:BDBS:PX42:C4LW
Debug mode (server): true
 File Descriptors: 27
 Goroutines: 51
 System Time: 2016-01-07T13:49:42.657417574Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

The docker-compose.yml file has a simple python flask application (which exits correctly) and a simple link to a direct mongo:2.6 image

Is it perhaps that the container subjected to the PID 1 problem ?
https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

Because the logs correctly say that mongo "Is really exiting now" but stays in that state. Here are the docker-container logs:

dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] now exiting
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] dbexit:
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: going to close listening sockets...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] closing listening socket: 7
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] closing listening socket: 9
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: going to flush diaglog...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: going to close sockets...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: lock for final commit...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: final commit...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: closing all files...
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [signalProcessingThread] closeAllFiles() finished
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [signalProcessingThread] journalCleanup...
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [signalProcessingThread] removeJournalFiles
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [conn4] end connection 172.17.0.3:57284 (1 connection now open)
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [conn5] end connection 172.17.0.3:57285 (1 connection now open)
dockerized_mongodb_1 | 2016-01-07T13:53:11.076+0000 [signalProcessingThread] shutdown: removing fs lock...
dockerized_mongodb_1 | 2016-01-07T13:53:11.076+0000 [signalProcessingThread] dbexit: really exiting now

The output of docker-compose stop:

ERROR: for brancheserver_dockerized_mongodb_1  HTTPSConnectionPool(host='192.168.99.100', port=2376): Read timed out. (read timeout=70)

Output of docker-compose ps after stop:

david$ docker-compose ps
               Name                             Command               State      Ports
----------------------------------------------------------------------------------------
[my flask server name... irrelevant]       /branche/init_and_start.sh ...   Exit 0
brancheserver_dockerized_mongodb_1   /entrypoint.sh mongod            Up       27017/tcp

davidovich on 7 Jan 2016

I had this problem in both Ubuntu 14.04.3 VMWare virtual machine (not sharing directories with the host system) and Mac OS X El Capitan (via docker-machine).

Now I'm not having this problem, using version 2.6.1.

I can't replicate this scenario anymore, but I think that could be related with any Linux kernel update on the Docker host (we had this problem with other containers like PHP-FPM).

fer2d2 on 7 Jan 2016

Same problem on Ubuntu with mongo:2.6:

$ docker --version
Docker version 1.9.1, build a34a1d5
$ docker-compose --version
docker-compose version 1.5.2, build 7240ff3

My mongo definition on docker-compose.yml:

mongo:
  image: mongo:2.6
  volumes:
    - ./docker/conf/mongodb.yml:/etc/mongodb.conf:ro
  command: mongod --config /etc/mongodb.conf

The mongodb.yml file:

storage:
  journal:
    enabled: false

Sometime, mongo process got 100% CPU usage and can be stopped. I tried (in order):

docker-compose stop (^C)
sudo service docker stop # Docker service is stopped, mongo container still running.
kill -9 <mongo_process>

This container is like Chuck Norris.

Let me know if you need additional information.

soullivaneuh on 19 Jan 2016

👍1

Ok I think I found where is the problem.

On my mongo db configuration:

storage:
  journal:
    enabled: false

I disabled journal storage.

If I set it to true or just not specifying configuration, the issue is gone.

So we know where is the problem, but not why.

soullivaneuh on 19 Jan 2016

I have consistently had this problem for a few weeks now. It showed up when I upgraded from Docker 1.8.4 to Docker 1.9.x. I'm on a Mac using vmwarefusion or virtualbox. With or without shared file system support. I'm also using docker-compose 1.5.2. Happens with mongo versions 2.6 and 2.6.1. Happens with --journal and --no-journal. The command I run is mongod --smallfiles.

This bug occurs during CI builds of my somewhat complicated product. I can't reproduce it by simply starting/stopping the container. I am working on making a smaller config to demonstrate the problem.

The "PID 1 problem" hypothesis sounds plausible, from what little I know. I'm running out of things to try so I might merge the mongo 2.6 image with phusion/baseimage and see what happens.

(update: mongo:3.2 image works for me)

ghost on 21 Jan 2016

@damonomad If you want to test if it is the "PID 1" problem, then you shouldn't need all of phusion/baseimage, just use tini.

yosifkit on 21 Jan 2016

After much debugging this is an aufs bug. Which means it will be fixed in the newest release (I just tested boot2docker 1.10-rc1). I was also able to change to overlay and was not able to hit this stopping problem.

yosifkit on 21 Jan 2016

See https://github.com/docker/docker/issues/18180#issuecomment-173436661 for where the current fixes are.

yosifkit on 21 Jan 2016

Upgraded mongo from 2.x to 3.x on ubuntu 14.04 has fixed this for me (so far)

https://docs.mongodb.org/v3.0/tutorial/install-mongodb-on-ubuntu

ghost on 25 Jan 2016

docker 1.10-rc1 was the definitive fix for me. I can use any version of the mongo image with this Docker. Thanks!

ghost on 25 Jan 2016

@damonomad you must mean boot2docker 1.10-rc1, which is not the samething as docker 1.10-rc1

Guys, the issue is a AUFS bug, see https://github.com/docker/docker/issues/18180 at the end for workarounds.

thefallentree on 26 Jan 2016

@thefallentree thanks, yes, I meant boot2docker 1.10-rc1. Which, because I'm on a Mac, I got via the DockerToolbox 1.10-rc1 installer. Upgrading the boot2docker image in my virtual machine was the key thing.

ghost on 26 Jan 2016

:+1:

I am using running docker on Ubuntu 14.04.3 and have had this problem on 1.7.1 and 1.9.1 as well.
My Mongo version is 3.0.7.

purezen on 5 Feb 2016

This issue is fixed in boot2docker 1.10-rc1 and above. Thanks

davidovich on 25 Feb 2016

I'm receiving the following error when trying to stop a mongo:2.4.10 image -

An error occurred trying to connect: Post https://192.168.99.100:2376/v1.21/containers/15e9bb28144b/stop?t=10: EOF
Error: failed to stop containers: [15e9bb28144b]

I'm running the following docker versions:
Docker version 1.9.1, build a34a1d5
docker-machine version 0.5.2 ( 0456b9f )

I've worked for a while with the 3.0.7 mongo image, and had none of this problems. Since I've downgraded the version to suit our production environment better I've started having trouble with stopping the mongo container.
Any idea how to solve this?

yaron-idan on 29 Mar 2016

@yaron-idan, I believe the only solutions are to downgrade the kernel, update aufs, or use a different graph driver; none of which would be very easy in the boot2docker vm since you want to keep docker 1.9.1.

Are there specific reasons to be using 1.9 vs 1.10? If your need to do docker cli commands to your production systems, you could downgrade your docker client on your host to 1.9.1 while running the 1.10 daemon in the VM (older clients can talk to newer daemons, while new clients cannot talk to old daemons).

yosifkit on 29 Mar 2016

I was actually not aware of the new docker version, I've updated it and I think it solved the problem. Thanks for the help.

yaron-idan on 30 Mar 2016

I am also having the same issue. Mongo DB container is not stopping.

Output of sudo docker-compose down command :-

Stopping mongoServer ... 

ERROR: for mongoServer  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=70)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Output of uname -a command

Linux ubuntu 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Output of docker info command

Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 15
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 34
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797GiB
Name: ubuntu
ID: QKF4:ZEQB:UUZQ:QK5G:6HKD:UXVB:AYIK:X5DV:OE2R:7XLZ:HFW6:E3HG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Output of docker-compose -version command

docker-compose version 1.11.2, build dfed245

Please help me to solve this issue.

ravi-stiwari on 12 Jun 2017

Closing old issue, this should be fixed by using a newer docker daemon (1.10+). Feel free to comment on further troubleshooting if you feel this was closed in error or on the solution to help users running into the same problem.

yosifkit on 29 Aug 2017

👎4

Still reproduces on:
Linux IBelieveICanFly 4.2.0-23-generic #28~14.04.1-Ubuntu SMP Thu Dec 31 13:40:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

And last docker version:
Docker version 17.06.1-ce, build 874a737

vusychenko on 6 Sep 2017

@yosifkit I have the same problem:

$ uname -a 
Linux ws212.open.ru 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux

$ docker version 
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:46 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:39:27 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian

$ ps fax | grep mongo -n3
117- 1053 ?        Ssl    1:31 /usr/bin/dockerd -H fd://
118- 1177 ?        Ssl    0:49  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
119- 5892 ?        Sl     0:00      \_ docker-containerd-shim ecd4b0b52c52847383dc784b692037bfb43a0e86d0c6aa94e81f5584862c2771 /var/run/docker/libcontainerd/ecd4b0b52c52847383dc784b692037bfb43a0e86d0c6aa94e81f5584862c2771 docker-runc
120: 5939 ?        Zsl   55:59          \_ [mongod] <defunct>

...

143- 1892 pts/0    Ss     0:00          |   \_ /bin/bash
144: 5822 pts/0    Sl+    0:00          |   |   \_ docker run mongo
145- 6020 pts/1    Ss     0:00          |   \_ /bin/bash
146- 6098 pts/1    Sl+    0:00          |   |   \_ docker stop blissful_raman

Note: the process is a zomby.

Mongo logs:

$ docker run mongo 
2017-11-24T12:45:21.893+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=ecd4b0b52c52
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] db version v3.4.10
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] modules: none
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] build environment:
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten]     distmod: debian81
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] options: {}
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] 
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7473M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] 
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] 
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] 
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] 
2017-11-24T12:45:21.926+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2017-11-24T12:45:21.930+0000 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version" }
2017-11-24T12:45:21.930+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2017-11-24T12:45:21.931+0000 I INDEX    [initandlisten] build index done.  scanned 0 total records. 0 secs
2017-11-24T12:45:21.931+0000 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 3.4
2017-11-24T12:45:21.931+0000 I NETWORK  [thread1] waiting for connections on port 27017
2017-11-24T12:45:35.611+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2017-11-24T12:45:35.611+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2017-11-24T12:45:35.613+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2017-11-24T12:45:35.636+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2017-11-24T12:45:35.636+0000 I CONTROL  [signalProcessingThread] now exiting
2017-11-24T12:45:35.636+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

But the process is not finished and a command docker stop too.

And the most interesting thing is I can stop and remove this container only after a hard reset (by button) my machine.

QIvan on 24 Nov 2017

I had the same problem with Mongo on Ubuntu 18.04.1 using this Dockerfile: https://github.com/docker-library/mongo/blob/6932ac255d29759af9a74c6931faeb02de0fe53e/4.0/Dockerfile

The solution turned out to be a Snap instance of docker installed during the Ubuntu OS setup, running alongside the one I installed via apt-get after the initial installation.

sudo snap remove docker