Moby: Device-mapper does not release free space from removed images

[AaronFriel]

Docker claims, via docker info to have freed space after an image is deleted, but the data file retains its former size and the sparse file allocated for the device-mapper storage backend file will continue to grow without bound as more extents are allocated.

I am using lxc-docker on Ubuntu 13.10:

Linux ergodev-zed 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

This sequence of commands reveals the problem:

Doing a docker pull stackbrew/ubuntu:13.10 increased space usage reported docker info, before:

Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker pull stackbrew/ubuntu:13.10:

Containers: 0
Images: 3
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 413.1 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.8 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker rmi 8f71d74c8cfc, it returns:

Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

Only problem is, the data file has expanded to 414MiB (849016 512-byte sector blocks) per stat. Some of that space is properly reused after an image has been deleted, but the data file never shrinks. And under some mysterious condition (not yet able to reproduce) I have 291.5 MiB allocated that can't even be reused.

My dmsetup ls looks like this when there are 0 images installed:

# dmsetup ls
docker-252:0-131308-pool        (252:2)
ergodev--zed--vg-root   (252:0)
cryptswap       (252:1)

And a du of the data file shows this:

# du /var/lib/docker/devicemapper/devicemapper/data -h
656M    /var/lib/docker/devicemapper/devicemapper/data

How can I have docker reclaim space, and why doesn't docker automatically do this when images are removed?

[vreon]

+1, I'm very interested in hearing some discussion on this subject. My strategy so far has been

• be careful what you build/pull
• be prepared to blow away your /var/lib/docker :neutral_face:

@AaronFriel, which version of Docker are you on? 0.7.1?

[kiorky]

/cc @regilero (link also in to #2276)

[alexlarsson]

Starting from a fresh /var/lib/docker:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
292M -rw-------. 1 root root 100G Dec 12 17:29 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root   89 Dec 12 17:29 /var/lib/docker/devicemapper/devicemapper/json
732K -rw-------. 1 root root 2.0G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

Then after docker pull busybox it grew a bit:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
297M -rw-------. 1 root root 100G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root  181 Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/json
756K -rw-------. 1 root root 2.0G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 1
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 296.6 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

docker rmi busybox does _not_ make the file larger, but makes the space free in the devicemapper pool:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
298M -rw-------. 1 root root 100G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root   89 Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/json
772K -rw-------. 1 root root 2.0G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

The loopback file doesn't grow if we download the image again:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
298M -rw-------. 1 root root 100G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root  181 Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/json
772K -rw-------. 1 root root 2.0G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 1
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 296.6 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

So, it seems like we fail to re-sparsify the loopback file when the thinp device discards a block.

[alexlarsson]

However, if i create a file inside the container fs image it _does_ reclaim the space in the loopback file.
I.e. I did this in the busybox image:

 cd lib
 cat libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 lib
c.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so
.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 > a_file.bin
/lib # ls -l a_file.bin
-rw-r--r--    1 root     root      47090160 Dec 12 16:41 a_file.bin

This grew the data file from 299M to 344M. But when i removed a_file.bin (and waited a bit) it got back to 299M.

So, this seems to me like it is a devicemapper bug. I.e. it forwards discards from the thinp device to the underlying device, but it doesn't discard when removing thinp devices from the pool.

[alexlarsson]

This seem to be a kernel issue, I was looking at working around it bu using BLKDISCARD, but i failed. See this bugs for some details: https://bugzilla.redhat.com/show_bug.cgi?id=1043527

[alexlarsson]

I put my workaround in https://github.com/alexlarsson/docker/tree/blkdiscard, but we're still researching if we can do better than this.

[alexlarsson]

Upstream dm comments on this issue:
http://device-mapper.org/blog/2013/12/17/thin-provisioning-and-discard-support/

[blalor]

Having this problem on CentOS (2.6.32-358.23.2.el6.x86_64) with Docker 0.7.0, as well. Old, but the problem's not isolated to Ubuntu.

[skakri]

Same issue on Arch GNU/Linux 3.12.6-1-ARCH, Docker version 0.7.2.

[blalor]

Still exists on 0.7.0 on CentOS.

[nmz787]

Still exists in 0.7.2 on ubuntu 12.04.3 LTS.

A lot of the space is in docker/devicemapper/devicemapper/data and metadata, but also in docker/devicemapper/mnt

It's neat that I learned you can see the container file systems in docker/devicemapper/mnt/SOME_KIND_OF_ID/rootfs

but it's not neat that my hard disk is almost completely eaten up and only fixable by rmdir -r docker.

[logicminds]

I am having a similar issue while writing docker support for rspec-system. My test VM (docker host) has a 8GB drive and after repeatedly creating images without deleting them my drive fills up. But after removing all images and containers the drive is still 100% full. I figured it was an ID-10T error but just gave up and destroyed the VM all together.

[mengzechao]

Still exist in 0.7.5 on ubuntu 13.04.

[unclejack]

This issue has been fixed by PR #3256 which was recently merged. This fix will be included in a future release.

I'm closing this issue now because the fix has been merged to master.

[alexlarsson]

Note: Its not _fully_ fixed until you also run a kernel with http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6d03f6ac888f2cfc9c840db0b965436d32f1816d in it. Without that the docker fix is only partial.

[logicminds]

What is the work around to remove space. I am using rhel 6.5 and it might be a while to get the new kernel.

Sent from my iPhone

On Jan 21, 2014, at 6:18 AM, Alexander Larsson [email protected] wrote:

Note: Its not fully fixed until you also run a kernel with http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6d03f6ac888f2cfc9c840db0b965436d32f1816d in it. Without that the docker fix is only partial.

—
Reply to this email directly or view it on GitHub.

[alexlarsson]

@logicminds There is no super easy way to recover the space atm. Technically it should be possible to manually re-sparsify the loopback files. But that would require all the non-used blocks to be zeroed or something for easily detection of the sparse areas, which is not done on thinp device removal.

[logicminds]

@alexlarsson Does this also affect OEL 6.5? The OEL 6.5 actually uses the uek 3.8 linux kernel and since I have the option between switching from 2.6 to 3.8 kernel this might be a simple switch for me.

[alexlarsson]

@logicminds I don't even know if that commit is in the upstream kernel yet. That link is from the device-mapper tree. Its definitely not in 3.8.

[alexlarsson]

I'm looking at creating a tool like fstrim that can be used to get back the space.

[kontrafiktion]

@alexlarsson issue https://bugzilla.redhat.com/show_bug.cgi?id=1043527 has been closed, officially because of "unsufficient data". Does that mean the patch will not make it into the kernel? Is it still needed?

[alexlarsson]

@vrvolle The patch that makes the workaround that docker uses is already upstream. There doesn't seem to be any upstream work to make that workaround unnecessary though.

[nicolas-van]

I still have this problem with docker 0.9 on centos 6.5 with the default 2.6.32 kernel.

I'm not sure to understand what you said previously about this commit into device-mapper. Could you confirm that if I migrate my kernel to 3.8 this bug should be solved?

Thanks in advance.

[alexlarsson]

@nicolas-van No, you need this commit: https://github.com/torvalds/linux/commit/19fa1a6756ed9e92daa9537c03b47d6b55cc2316

It is in 3.14, and in may be in various 3.x.y backports

[Jacq]

I've installed some time ago docker to build an image and run it within a container. Then some time later I cleared all the images and containers, including the docker application and main folder.
Now I realize that of a total of 4GB/24GB free/used (df -h), the command du / -sh reports only 10GB so another 10Gb are not being accounted for. That is more less the size of the temp images generated with docker, could it be related to this bug. I've used centos 6.5 and docker 0.9.

[Jacq]

I've removed docker with yum, and devs from /dev/mapper/docker* with dmsetup, and also rm /var/lib/docker -Rf, and still the disk reports with df 10gb used that I cannot find anywhere.

[alexlarsson]

@Jacq Its possible some file is still kept alive by a process that has a file descriptor open to it. Did you reboot? That would ensure that doesn't happen.

[Dieterbe]

I run Linux 3.14.4-1-ARCH, and docker 0.11.1, removed all images and containers.
and the file /var/lib/docker/devicemapper/devicemapper just stuck around, consuming about 1.5GB

here's output from after i was fiddling with some mongodb stuff, i guess the file size must be sparsely allocated because my /var is not even that big.

~/w/e/video_history ❯❯❯ docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-254:3-585-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 948.4 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 1.0 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.14.4-1-ARCH
WARNING: No swap limit support
~/w/e/video_history ❯❯❯ sudo ls -alh /var/lib/docker/devicemapper/devicemapper/data
-rw------- 1 root root 100G Jun  4 14:35 /var/lib/docker/devicemapper/devicemapper/data
~/w/e/video_history ❯❯❯ sudo  du -shc /var/lib/docker/devicemapper/
1.6G    /var/lib/docker/devicemapper/
1.6G    total

[bolasblack]

Excuse me, is the bug fixed now? I did encounter it in gentoo.

[arosboro]

@bolasblack I run gentoo and ran into this issue. Did you figure anything out?

I'm using the latest gentoo-sources which are 3.14.14 for x86_64. I looked at https://github.com/torvalds/linux/commit/19fa1a6756ed9e92daa9537c03b47d6b55cc2316?diff and that patch is applied to my sources. I have docker Docker version 1.1.0, build 79812e3.

@alexlarsson Thanks for bringing your attention back to this closed issue after such a long period of time. It seems like it's still causing trouble. Any word on the status of #4202?

[daniellockard]

This is still an issue! I think I'm going to switch back to using AUFS for the time being.

[shabbychef]

@daniellockard AUFS seems to be unofficially deprecated: #783 and #4704, so good luck with that.

[ccjon]

Yikes... where did that 25GB go? Oh, into that one file.... I am running kernel 3.16.2 f20 64b

The link to the 'workaround' is broken... what is it? and does my kernel support it... if Torvalds committed in 3.14, I suspect fedora should see it in 3.16, no?

[analytically]

+1

[Zolmeister]

+1

[bleib1dj]

+1

[Crashthatch]

+1 Still seems to be happening on Ubuntu Trusty, 3.13.0-36-generic with Docker version 1.3.1, build 4e9bbfa

[4thAce]

+1 also on Trusty. Is it worth checking on 14.10 which uses 3.16?

[AaronFriel]

@SvenDowideit @shykes @vieux @alexlarsson @Zolmeister @creack @crosbymichael @dhrp @jamtur01 @tianon @erikh @LK4D4

Tagging the top committers because this is ridiculous. There needs to be a public conversation about this and an admission from Docker's team that this bug can lead to containers or systems that periodically break and need to be recreated. I already know several people who have had to implement insane devops solutions like periodically reimaging their Docker hosts every week or two because their build bots have so much churn. I introduced this issue close to a year ago and there's been, as near as I can tell, no definitive solution created, and older kernels that are ostensibly supported are not.

Docker team: please do the research to determine which kernel versions are effected, why, and what patch will fix the issue, and document it. Publish that information along with what kernel versions you support, because right now consumers of Docker are getting bit by this issue over and over and over, as evidenced by the fact that I still get emails on this issue every week or two. Seriously, this is a breaking issue and it's been a pain point since before 1.0.

As I see it, there are several possible options to fix this issue in a satisfactory way that would stop the emails I keep getting for +1s on this issue:

1. Notify users when Device-Mapper is being used on an unsupported kernel and provide them with detailed instructions for how to reclaim space, and if possible automatically set up a process to do this in Docker. I would advise that this notice should also be emitted when using the docker CLI against a host that suffers from this problem, so that when remote managing hosts from the docker CLI, users are made aware that some hosts may not reclaim space correctly.
2. Fix the problem (somehow). I don't know enough about kernel development to know what this would entail, but, based on my novice reading, I suggest this:

a. As device mapper is a kernel module, bring a functional, working version of it into the Docker source tree as something like dm-docker

b. Make sufficient changes to dm-docker that it can coexist with device mapper.

c. On affected platforms, install the dm-docker kernel module on installation and default to using dm-docker.

1. Amend your installation docs and the docker.com site to include a warning on affected kernel versions, and add a runtime check to the packages to verify correct device-mapper operation, and if not report it to the user.

This should be a blocking issue for the next stable release of Docker, because it's just plain unacceptable to keep punting on it and leaving users in the lurch.

[damovsky]

Personally I see the CoreOS as the only one stable Linux distro for Docker (until this issue is resolved).

Docker Team: I know that this issue is not caused by your component, but please, help us to use your software also on the others Linux distro. It would be fine if you can also mention this issue as a well known limitation of Docker in documentation, so other people won't waste their time.

Thanks!
Martin

[adamdecaf]

+1 something needs to be done.

It'd be nice if there was something more apparent for this issue than having to dig into the (closed) github issues list. It took a long time to actually discover that this was the underlying issue and it would have been nice to visibility into this issue.

[snitm]

Us upstream Device mapper developers (myself and Joe Thornber) had absolutely _zero_ awareness that this issue is still a problem for people. We fixed the issue immediately once we were made aware of it (by @alexlarsson back in Dec 2013) and tagged it for inclusion in _all_ stable kernels at that time, see: http://git.kernel.org/linus/19fa1a6756ed9e9 ("dm thin: fix discard support to a previously shared block")

Joe Thornber was just made aware that @alexlarsson implemented trim-pool in docker go code. When I pointed it out to him he took on implementing a proper standalone 'thin_trim' tool that will get distributed as part of the 'device-mapper-persistent-data' package (at least on Fedora, CentOS, RHEL), see:
https://github.com/jthornber/thin-provisioning-tools/commit/8e921580554ed91e84bb68ea32a8c2ea47ad6ff3

SO... all being said, users who are running kernels that don't have upstream commit 19fa1a6756ed9e9 ("dm thin: fix discard support to a previously shared block") need to fix that by running kernels that are better supported. I can easily send a note to [email protected] to backfill the fix to any stable kernels that don't have it though. So please let me know which, if any, stable kernel(s) don't have this fix.

Moving forward we'll want docker to take on periodically running 'thin_trim' against the thin-pool device that docker is using. But we'll cross that bridge once 'thin_trim' is widely available in the distros.

[SvenDowideit]

@shabbychef @daniellockard no, AUFs is not deprecated - first up, only one of those issues is closed, and reading on, I'm guessing https://github.com/docker/docker/issues/783#issuecomment-38731137 is worth reading:

Our initial plan was to phase out aufs completely because devmapper appeared
 to be the best option in 100% of cases. That turned out not to be true, there 
are tradeoffs depending on your situation and so we are continuing to maintain both.

[SvenDowideit]

@snitm could you add something to hack/check_config.sh to tell users their kernel doesn't have this patch?

[snitm]

@SvenDowideit unfortunately the change in question isn't identified in an arbitrary kernel. For starters commit 19fa1a6756ed9e9 didn't bump the thin or thin-pool targets' version. But even if it did that version will vary across all the stable kernels (and so that is why version bumps within a 'stable' commit are bad.. as it is cause for hand-editing of all kernels the commit gets backported to).

BUT, users that have a thin and thin-pool target version >= 1.12.0 will all have the fix. So kernels >= 3.15. The docker that users are running would also need to include @alexlarsson's docker.git commit 0434a2ce ("devmapper: Add blkdiscard option and disable it on raw devices")

[snitm]

FYI, running 'dmsetup targets' will list the thin and thin-pool target versions (provided the dm-thin-pool kernel module is loaded).

[dcvii]

Thanks for attention on this. We mentioned it to the booth dudes at Re:Invent last week.

[SvenDowideit]

@snitm so dmsetup targets output should be added to the check-config output?

[AaronFriel]

@snitm

Would it be possible to create an automated test that would create a thin-provisioned device-mapper device, perform some operations on it that would fail to reclaim free space on an unpatched kernel, and report a status code based on that?

[snitm]

@SvenDowideit you're hoping to catch any new offending kernels before they start making use of DM thin provisioning ontop of loopback?

@AaronFriel seems extremely narrow in scope to be so hung up on this particular fix. There is more to enterprise-grade deployment of DM thin-provisioning than making sure this fix is in place (unfortunate reality). The DM thin provisioning target has seen numerous error handling, performance and feature improvements since commit 19fa1a675 went upstream. All of which are important when deploying DM thin provisioning in production.

SO taking a step back, I happen to be able to enjoy working for a company that understands layered products need to be developed in concert with all the underlying layers. And I have little interest in supporting DM thin provisioning for N arbitrary kernels being paired with docker.

And I happen to _strongly_ believe that nobody should be using DM thin provisioning with loopback devices as the backing devices. That was a complete hack that got merged into docker without proper restraint or concern for "what does this look like in production?".

I've realized that a proper deployment of docker on DM thin provisioning requires the enterprise-oriented management features that lvm2-based thin-provisioning configuration provides. So with that in mind I've steadily worked on making a new hybrid management solution work, see this PR:
https://github.com/docker/docker/pull/9006

This work depends on:
1) lvm2 >= 2.02.112
2) DM thin-pool target version >= v1.14.0 (aka changes staged in linux-next for Linux 3.19 inclusion)

RHEL7.1 will have these changes. And RHELAH (Atomic Host) will too. Any other distros/products that want to properly deploy docker on DM thin-provisioning should too.

[SvenDowideit]

@snitm yeah - I'm trying to reduce the number of users that are hit by a misterious breakage, which then takes even more time for someone to realise that it could be this obscure pain, then to try to ask the user to figure out that a lack of some mystery patch is the problem.

and so in context of your remaining info - I want to reduce the number of kernels that will cause this horrid surprise :)

[snitm]

@SvenDowideit OK, I provided the info that docker (or its users) could use to check for DM thinp on loopback compatibility in this comment: https://github.com/docker/docker/issues/3182#issuecomment-63706507

[dbabits]

@snitm, @AaronFriel
I'm running into what seems like this issue on AWS beanstalk, Amazon AMI.
(Running out of space)
Linux ip-172-31-63-145 3.14.23-22.44.amzn1.x86_64 #1 SMP Tue Nov 11 23:07:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ sudo docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): c78088f/1.3.2
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): c78088f/1.3.2

[reiz]

I'm still getting this error with Ubuntu 14.04 and Docker 1.4.1.

[sybrandy]

Don't count on the thin-pool and thin versions being >= 1.12.0 to mean you're O.K. We're running CentOS 6.5 VMs with the 2.6.32-431.29.2.el6.x86_64 kernel and dmsetup targets reports that thin-pool and thin are both v1.12.0. However, I'm stuck with a 40GB data file that isn't freeing itself up.

[LinuXY]

What are the implications running this on CentOS / RHEL 6.5 with this bug? Are you OK with >=100G of free space. I assume this eventually fills any size disk? Or is it bounded at 100G?

[jperrin]

Keep in mind that 6.5 does not have the newest kernel available to centos. I would recommend a simple 'yum update' to 6.6 and a reboot to test with the 2.6.32-504.3.3 kernel.

[ripienaar]

Confirm the very latest kernel and distro updates for CentOS 6 works fine and releases space.

[sybrandy]

ripienaar: Can you explicitly state which CentOS 6 and kernel version you are using? I just want to make sure that when I pass the info along, I have all of the info I need.

thanks.

[jbiel]

As @reiz commented above, this is happening with latest Ubuntu 14.04 + latest docker. dmsetup targets shows thin/thin-pool of v1.9.0 on one of our instances (no active docker containers.) dmsetup targets doesn't show any thin entries on a similar instance with active docker containers. (I'm not really asking for help on this, just adding (hopefully useful) data.)

[ripienaar]

Basic OS stuff:

# uname -a
Linux vagrant-centos65 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS release 6.6 (Final)

# dmsetup targets
thin-pool        v1.14.0
thin             v1.14.0
mirror           v1.12.0
striped          v1.5.6
linear           v1.1.0
error            v1.2.0

# dmsetup status
docker-8:1-393542-pool: 0 209715200 thin-pool 92343 178/524288 4664/1638400 - rw discard_passdown queue_if_no_space

Usage:

# du -h /var/lib/docker/
2.7G    /var/lib/docker/devicemapper/devicemapper
# docker rmi b39b81afc8ca
# du -h /var/lib/docker/
2.5G    /var/lib/docker/devicemapper/devicemapper

Previous to getting onto that kernel it would not recover space.

[lexinator]

@jperrin mentions this be resolved using centos kernel 2.6.32-504.3.3. is it known which kernel commit(set of commits?) that resolve this issue?

I am currently using one of Oracle Enterprise Linux 3.8 UEK.

[donrudo]

Is there any workaround for this situation? at AWS is easy to just re-create the instance but at the local development environment is not good to notice the /var partition is owned in a 90% by device-mapper because of docker; soon won't have space even for logs or journals

[ripienaar]

@donrudo as above - update to latest centos 6 - the 504.3.3 kernel.

[donrudo]

That's OK for centos but some of our devs are using Fedora and others OpenSuse.

[snitm]

All DM thin provisioning changes are sent upstream first. So if Centos 6 is fixed it is also fixed upstream. The various distros need to pull fixes back or rebase more aggressively.

[lexinator]

@ripienaar I'll ask again, do you know which kernel commit(or set of commits) that are required for the fix?

[ripienaar]

@lexinator nope

[TylerMills]

I'm seeing this issue on Ubuntu 14.04 Trusty and kernel 3.13.0-36-generic. Possible solution is attaching AWS EBS storage to the the directory so it can scale indefinitely.

[sirocode]

Guys, this issue really hurts. CoreOS is moving to ext4 now and soon we will have all this troubles on CoreOS as well.

How people can use docker (in dev or prod) while this issue is unfixed for more than a year? Everybody has endless disks?

[jessfraz]

I believe they are moving to ext4 for overlay not devmapper. But that's
just the word on the street...

On Wednesday, January 28, 2015, Sergey [email protected] wrote:

Guys, this issue really hurts. CoreOS is moving to ext4 now and soon we
will have all this troubles on CoreOS as well.

How people can use docker (in dev or prod) while this issue is unfixed for
more than a year? Everybody has endless disks?

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-71800923.

[manjugit]

I am confused. My disk space on a Google instance is just 10G, but the size of the /var/lib/docker/devicemapper/devicemapper/data is shown as 100G. I am not able to reclaim space even after deleting all the containers and images.

[jperrin]

It's a sparse file. Use ls -sh instead of ls -l, or alternately, du -h

[andrewcstewart]

I can report this on Ubuntu 13.04 (probably not surprising).

Is the only real workaround to this right now to provision enough disk space to anticipate growth of devicemapper?

[snitm]

This entire thread is quite ridiculous. One Debian user after another complaining. If you're seeing a problem you should either switch to a distro that is properly maintained or work with your distro maintainers to get them to fix the issue. These distros aren't keeping up with fixes or improvements. I'll be ignoring all future pleading for someone to fix this issue on distro $foo given the fix is clearly upstream already (see CentOS or RHEL). Please open a bug against the broken distro in question and reference this issue.

[dekz]

It's not that ridiculous when Docker maintains they have support on the following platforms:

Docker is supported on the following versions of Ubuntu:

Ubuntu Trusty 14.04 (LTS) (64-bit)
Ubuntu Precise 12.04 (LTS) (64-bit)
Ubuntu Raring 13.04 and Saucy 13.10 (64 bit)

How can you have a blanket support call for 14.04 when this issue is obviously so prevalent. Docker should obviously drop blanket support for certain kernels on certain distributions.

I'm using CentOS and and kernel upgrade seems to have improved the run away devicemapper usage issue.

Regardless, thanks for your work @snitm.

[TylerMills]

For those having this issue on Ubuntu, ensure that you're appropriately cleaning up old docker images/containers. I found that my cleanup script wasn't running when I thought it was, and that's why it was using so much space.

``` #!/bin/bash

Delete all containers

docker rm $(docker ps -a -q)

Delete all images

docker rmi $(docker images -q)
```

[thaJeztah]

@TylerMills note that docker rm will _not_ remove volumes created by the containers. If your containers make use of volumes, you should use docker rm -v to have docker remove the volumes as well.

[bkeroackdsc]

@snitm We just got bit by this as well (using Ubuntu 14.04). While it might not be Docker's fault specifically, they should have a strong interest in fixing this since it reflects poorly on the whole experience.

[JohnMorales]

Also note I've bumped into this problem and this is what I did to fix it:

In my case though, due to device that /var is mounted on reaching 100% usage, the only way to stop the host from going into a kernel panic is to kill the docker process directly (i.e. issuing any docker commands would panic).

Then manually kill any processes using the devicemapper mounts using this command:

grep docker /proc/*/mounts | awk -F/ '{ print $3 }' | xargs kill -9

then I was able to umount any devicemapper mounts using:

for i in /dev/mapper/docker-*; do umount $i; dmsetup remove $i; done

those steps released all the file descriptors which allowed me to reclaim the space/delete files whatever..

At that point in my case, it made more sense to move the /var/lib/docker directory to a mount with more space and configure the docker daemon to use a different home directory using the -g argument

thanks to Alexander Larsson via Adam Miller

[reub]

Following on to @dekz comment, it would probably be a good idea if the installation guide didn't instruct users that kernel versions without this fix are acceptable as is done here: https://docs.docker.com/installation/ubuntulinux/

[xer0x]

Thanks @JohnMorales. A little warning about this from Docker would be a great idea.

[ramantalkin]

+1

[cpoonolly]

+1

[natea]

We recently got bit by this running Ubuntu 14.04.2 LTS (3.16.0-30-generic) and are still scrambling to find a solution. It seems like either switching to CentOS 6.6 (which supposedly has the kernel fix) or mounting a volume that has a larger disk are the only solutions.

Has anyone running Ubuntu found an acceptable workaround?

I agree - Docker should man up and address this issue - it's a serious bug that needs documentation and warning messages, so more people don't naively fall into the same trap.

[sagiegurari]

I agree with @natea
I'm facing same issue with redhat 6.5 and it is putting a big question mark if docker is production ready and we are starting to look for alternatives.
I don't mind even getting a manual workaround to solve the issue, but right now, I'm not aware of any feasible ones.

[thaJeztah]

@sagiegurarie be aware that we bumped the requirements for CentOS to 6.6 because of various issues with 6.5 (and the associated kernel). This requirement is probably not yet on the live docs-website, but will be with the next release, maybe before that.

[sagiegurari]

so issue #11643 answer is not relevant? that means we can't use redhat 6.5?

[bkeroackdsc]

@natea @sagiegurari What works for us is the manual "fix" suggested by @TylerMills

# Delete all containers
docker rm $(docker ps -a -q)
# Delete all images
docker rmi $(docker images -q)

The downside, of course, is that you then have to re-pull all images and run the containers again.

[sagiegurari]

@bkeroackdsc I remember i deleted all containers and images before as well and that didn't solve it, but I'll give it another shot.

[dreamcat4]

Hello! I would like to know if this issue still exists on ubuntu-15.04. Which is now at linux-3.19 kernel. Many thanks.

[jessfraz]

if you are on ubuntu-15.05 with that kernel why no use overlay, it is way
better than devicemapper

On Thu, May 7, 2015 at 11:26 AM, Dreamcat4 [email protected] wrote:

Hello! I would like to know if this issue still exists on ubuntu-15.04.
Which is now at linux-3.19 kernel. Many thanks.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-99968299.

[dreamcat4]

@jfrazelle Oh right. What's it like? Do you use overlay? I have been going under the assumption (up until now) that overlay support was still experimental as of 3.19 kernel.

[jessfraz]

Yes I use overlay :) we are hoping to move it up the list in 1.7 lost of us
use it and love it

On Thursday, May 7, 2015, Dreamcat4 [email protected] wrote:

@jfrazelle https://github.com/jfrazelle Oh right. What's it like? Do
you use overlay? I have been going under the assumption (up until now) that
overlay support was still experimental as of 3.19 kernel.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-99982543.

[jperrin]

@sagiegurari Why would you stick with 6.5? There are a number of updates and fixes you're missing by NOT going to 6.6, including news-worthy CVEs with names.

[sagiegurari]

@jperrin because it's not my decision :)
anyway i'm trying to check with all teams to see if redhat 7 would be ok for production.

[alexDrinkwater]

This seems to still be an issue on CentOS7 (3.10.0-123.el7.x86_64). I setup a new CentOS box and installed docker v1.6.0. The devicemapper data file never decreases size after a sudo docker rmi. I am looking into getting a CentOS6.6 box to try it out on that.

[Zolmeister]

v1.6.0 on Ubuntu 14.04 seemed to have a positive effect (over v1.5.0)

[sagiegurari]

seems to also work good on centos 7 which I guess can also say for redhat 7
but redhat 6.5 isn't, and i suggest to state that in the docs.

[rhvgoyal]

This should be working in latest upstream docker. And last comment from @sagiegurari seems to imply that. (works on centos 7).

If that's the case, we should close this bug.

So is this problem still happening with latest docker? If not, lets close it.

[sagiegurari]

well.... running more tests seems to provide an uncertain results.
lets say its much better, but might still be leaking, just a lot slower.
i'll run more checks to confirm.

[rhvgoyal]

@sagiegurari

Can you give me some more details on what's the issue you are facing and also your environment. Title of the issue says that space is not reclaimed upon image/container deletion, and latest upstream docker should have no such issues.

Can you please take upstream docker, build a dynamic binary, create a fresh thin pool (in a fresh /var/lib/docker/) directory, pull an image, delete it and look at docker info and see if space has been freed or not.

[sagiegurari]

I'll run more tests hopefully next week to provide more exact info.

I admit i see it in our build machine where we create and remove a lot of images (during the build itself) + run the docker registry 2.0 (to which we always push the different images with tag 'latest') + run few containers to do initial testing that everything works fine (test is actually not supposed to run on that machine, but we are in early stage so we create/stop/rm a lot of containers).

our images are really big, sometimes i see virtual size over 2gb, so if there is an issue, it shows itself faster than with usual images that are normally around 400mg.

anyhow, i'll try to run few tests to see behavior and provide more details next week.

[stanislavb]

Using Docker version 1.6.0, build 4749651/1.6.0 on Linux kernel 3.14.35-28.38.amzn1.x86_64

Removing images decreases disk space claimed by docker.

[rhvgoyal]

@vbatts

Can we close this issue I don't think in latest docker we have the problem of reclaiming image/container space from oop files. If something specific comes up, one can open a new issue with specific details.

[sagiegurari]

@rhvgoyal I don't see a reason to rush and close this issue after so many people complain about so many platforms/versions.

[rhvgoyal]

@sagiegurari

I don't know what do we gain by keeping issue open. I know it is fixed upstream. There are so many issues which are open. People are using them as dashboard to note the observations.

I feel that keeping an issue open makes sense only if there is a real reproducible problem. Otherwise this list of issues just keeps on growing and it is hard to figure out which one to focus on.

Nobody stops you from adding more data to this issue once you have that.

[alexDrinkwater]

@rhvgoyal what tag are you referring to by latest upstream?

[rhvgoyal]

@rhvgoyal Say latest commit. I just clone the latest git tree and test that. Does it really matter?

[rhvgoyal]

So many problems are reported on old docker releases and old kernels. I think first step should be to see if same problem is visible on latest docker and relatively newer kernerl. That gives us an idea if it is still a problem or something which is not fixed in older versions.

[alexDrinkwater]

When someone comes back and reads it in the future, yes. I was wanting to know what version it was fixed in.

[rhvgoyal]

We issue discards in the case of loopback devices. It was done long back actually. I think 1.6 should have it.

[rhvgoyal]

@stanislavb tested with 1.6 and it worked for him.

[alexDrinkwater]

If you refer to my first comment in this issue you will see that I also tried with v1.6.0 on centos7 and still had the issue.

[rhvgoyal]

@alexDrinkwater

Ok, are you able to reproduce it? What kind of thin pool did you use (on top of loop devices?). Did you restart docker or it happens very first time you start docker?

In some cases it might happen that docker is using loop devices and then docker is restarted, it can happen that pool was not shutdown because some device was busy somewhere. Upon restart docker
finds the pool already there and it has no knowledge that loopback devices are being used and it does not issue discard. I am wondering if you ran in to that situaiton?

[stanislavb]

CentOS 7

Docker version 1.6.2.el7, build c3ca5bb/1.6.2

Before image cleanup

[root@b2 ~]# docker info
Containers: 1
Images: 320
Storage Driver: devicemapper
 Pool Name: docker-8:2-1076248517-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 10.74 GB
 Data Space Total: 107.4 GB
 Data Space Available: 96.63 GB
 Metadata Space Used: 22.8 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.125 GB
 Udev Sync Supported: true
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.63-11.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 32
Total Memory: 31.52 GiB

[root@b2 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      483443064 13100584 470342480   3% /

After some docker rmi

[root@b2 ~]# docker info
Containers: 1
Images: 143
Storage Driver: devicemapper
 Pool Name: docker-8:2-1076248517-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 7.215 GB
 Data Space Total: 107.4 GB
 Data Space Available: 100.2 GB
 Metadata Space Used: 14.68 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.133 GB
 Udev Sync Supported: true
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.63-11.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 32
Total Memory: 31.52 GiB

[root@b2 ~]# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda2      483443064 9665924 473777140   2% /

[rhvgoyal]

@alexDrinkwater

Ok, just now I booted up a centos7 image and tried docker 1.6 and I don't see the problem. Here are the steps.

[root@centos7-generic ~]# docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64

There is already one image imported.

[root@centos7-generic ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
docker.io/fedora latest ded7cd95e059 2 weeks ago 186.5 MB

Here is the output of docker info.

root@centos7-generic ~]# docker info
Containers: 0
Images: 2
Storage Driver: devicemapper
Pool Name: docker-253:1-50331788-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 525.5 MB
Data Space Total: 107.4 GB
Data Space Available: 20 GB
Metadata Space Used: 892.9 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2

[root@centos7-generic ~]# docker rmi fedora
Untagged: fedora:latest
Deleted: ded7cd95e059788f2586a51c275a4f151653779d6a7f4dad77c2bd34601d94e4
Deleted: 48ecf305d2cf7046c1f5f8fcbcd4994403173441d4a7f125b1bb0ceead9de731

[root@centos7-generic ~]# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-253:1-50331788-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 307.2 MB
Data Space Total: 107.4 GB
Data Space Available: 20.22 GB
Metadata Space Used: 733.2 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2

Notice Data usage dropped from 525MB to 307MB. So deleting an image did get me the space back.

[rhvgoyal]

Ok, I forgot to paste the size of loop file before and after the operation. here it is.

• Before image pull.
  $cd /var/lib/docker/devicemapper/devicemapper
  $ du -h data
  293M data
• After image pull
  [root@centos7-generic devicemapper]# du -h data
  499M data
• After image deletion
  [root@centos7-generic devicemapper]# du -h data
  293M data

So my loop file actually shrunk after image deletion. I think that's what you are complaining about.

[rhvgoyal]

@stanislavb

Are you using loop files? They are not visible in your docker info. I think you are running into same issue where when docker started pool was already there.

Can you check the size of loop backing files (/var/lib/docker/devicemapper/devicemapper/data) and make sure that file shrinks after image deletion. (du -h ).

[stanislavb]

@rhvgoyal
CentOS 7, Docker version 1.6.2.el7, build c3ca5bb/1.6.2

[root@b2 ~]# du /var/lib/docker/devicemapper/devicemapper/data
7041272 /var/lib/docker/devicemapper/devicemapper/data
[root@b2 ~]# docker rmi $(docker images -q)
...
[root@b2 ~]# du /var/lib/docker/devicemapper/devicemapper/data
5617292 /var/lib/docker/devicemapper/devicemapper/data

[rhvgoyal]

@stanislavb

Ok, So loop device size is going down indeed.

Also I verified from the code. We will issue discard even if upon restart docker found thin pool already there.

    // By default, don't do blk discard hack on raw devices, its rarely useful and is expensive
    if !foundBlkDiscard && (devices.dataDevice != "" || devices.thinPoolDevice != "") {
            devices.doBlkDiscard = false
    }

As per above code, discards are disabled only if user either passed in a data block device or a thin pool. Given that we did not pass either, discards are still on. That's why it is working for you.

@alexDrinkwater Now we have two data points that it is fixed in 1.6. Do you still have concerns in closing this issue.

[stanislavb]

CentOS 6.6

Docker version 1.5.0, build a8a31ef/1.5.0

Before image cleanup

[root@b1 ~]# docker info
Containers: 2
Images: 2996
Storage Driver: devicemapper
 Pool Name: docker-8:2-21890120-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 43.24 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 117.1 MB
 Metadata Space Total: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System: <unknown>
CPUs: 32
Total Memory: 31.33 GiB

[root@b1 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      475957304 53995488 397777856  12% /

Checked size of data file at 1929 images remaining

[root@b1 ~]# du -h /var/lib/docker/devicemapper/devicemapper/data
30G /var/lib/docker/devicemapper/devicemapper/data

After some docker rmi

[root@b1 ~]# docker info
Containers: 2
Images: 497
Storage Driver: devicemapper
 Pool Name: docker-8:2-21890120-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 13.39 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 27.87 MB
 Metadata Space Total: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System: <unknown>
CPUs: 32
Total Memory: 31.33 GiB

[root@b1 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      475957304 25606060 426167284   6% /
[root@b1 ~]# du -h /var/lib/docker/devicemapper/devicemapper/data
13G /var/lib/docker/devicemapper/devicemapper/data

[rhvgoyal]

Ok, so even in 1.5 it works.

[alexDrinkwater]

I tested this on a new machine.

Docker version 1.6.0, build 8aae715/1.6.0

Before pull:

/dev/mapper/os-var                             3.9G  750M  2.9G  21% /var

Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:3-247472-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 307.2 MB
 Data Space Total: 107.4 GB
 Data Space Available: 3.308 GB
 Metadata Space Used: 733.2 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.634 GiB

after pull:

/dev/mapper/os-var                             3.9G  815M  2.9G  23% /var

Containers: 0
Images: 22
Storage Driver: devicemapper
 Pool Name: docker-253:3-247472-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 639.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 3.24 GB
 Metadata Space Used: 1.438 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.634 GiB

remove image:

sudo docker rmi 617ef0e7677f
Untagged: docker.io/ghost:latest
Deleted: 617ef0e7677fbff322b8f3af29b9795314a333876d66b49e1ddede15229dccea
Deleted: e68d1ee297168a0b3c3e21ff7a9ab14f7df8edf4525e008fcaf3abc83c731387
Deleted: 4eee85e0d29b6836d3d46ff3b84b0292626df5c8428da0aeeb916d33b6fb7642
Deleted: d0e1169bd5617096df1776902e191f23c07ac0fb5b365c5664fd3d4a554e4c8e
Deleted: e983dda26a2392947fffa4cc37b36251b84bbc74c95ce9c353b80a9c8e63d70f
Deleted: 64b0f4c09fde536675106331c31636f8478ed0cf5c2c7aa274d464216e470b3f
Deleted: c0313e19f503a4770c00250049419a014972ba8f3458524d61b377b8fd289ef0
Deleted: ff1093062f402021a7574b3c40692d2c6dc7aec07d66e17caa6c35df19bad091
Deleted: 37b39063e0c0f3c1a8c90d304ad7ba6b47e14242ff60f6f5d091b280258e0ff3
Deleted: 6e878a0c2422a5496d6bfc5eaf1facbc48f66a8e437fdd7db18d8944b20f7072
Deleted: a10b93053713fb726beaea993bad52b39ca92c5ce6b646dbc7d9cd96016ee9bc
Deleted: 1324d506b72f2a602a8f55c5224708e8ff02dec86b5fe46462a1d73aafb32075
Deleted: 70014f2a472b03d0cfde21d99c601db25f62de1d6c8497cadd6e05743c09d5a1
Deleted: f81216b6a47e2f51c80ff56044a5120d4b8cb76e9ea5990ba08ed073e97fd429
Deleted: 286e04b15dcfe587da0d8b6b359d6ae74c3ef299b868183859508304153ceaca
Deleted: 1dbb5edeebca3ae23a32fcf600015df645a788747553287548ce67126b206ab7
Deleted: 8807133d30b36044c670a06b087b6b61082b660a61fef651f0871283c5505bff
Deleted: 4c0283973ca2335a9ae82168956e4add2e6f2f13fd31e16473d695912e34d974
Deleted: 95d6d696e46933c9d063b5d6328b1a92b988462f1277d74d42bbbdd568efc220
Deleted: 80565b90e8eb693f03cea0411aadb45f21f2bcfe39f6b0fda8cf351eaee1f81b
Deleted: df2a0347c9d081fa05ecb83669dcae5830c67b0676a6d6358218e55d8a45969c
Deleted: 39bb80489af75406073b5364c9c326134015140e1f7976a370a8bd446889e6f8

after delete:

/dev/mapper/os-var                             3.9G  814M  2.9G  23% /var

Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:3-247472-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 307.2 MB
 Data Space Total: 107.4 GB
 Data Space Available: 3.24 GB
 Metadata Space Used: 733.2 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 7.634 GiB

[alexDrinkwater]

Let me know if you any more details of my machine or versions etc.

[rhvgoyal]

@alexDrinkwater

Looks like you are mounting a separate device on /var. What's the file system on that device and what are the mount options?

[rhvgoyal]

Also can you give me following.

• dmsetup table
• dmsetup status
• cat /etc/sysconfig/docker-storage

[rhvgoyal]

Can you try the simpler case of keeping /var on root itself with xfs file system. Trying to narrow down the problem a bit.

[rhvgoyal]

@alexDrinkwater

I mounted another disk with ext4 partition on /var/lib/docker so that loopback backing files are on ext4 (instead of xfs) and still things are working for me.

/dev/vdb1 on /var/lib/docker type ext4 (rw,relatime,seclabel,data=ordered)

Really not sure what's special about your setup.

[rhvgoyal]

To be specific,

• Before pull
  /dev/vdb1 9.8G 338M 8.9G 4% /var/lib/docker
• After image pull
  /dev/vdb1 9.8G 544M 8.7G 6% /var/lib/docker
• After image delete
  /dev/vdb1 9.8G 338M 8.9G 4% /var/lib/docker

[rhvgoyal]

@alexDrinkwater

The other major difference between your setup and my setup is kernel version. I am using "3.10.0-229.el7.x86_64" while you seem to be using "3.10.0-123.el7.x86_64". Can you upgrade your kernel and give it a try.

[alexDrinkwater]

Thanks for the help @rhvgoyal. I wouldn't be able to update the kernel today. But here are the settings you asked for:

/dev/mapper/os-var /var ext3 rw,relatime,data=ordered 0 0

vgdocker-lvdocker: 0 62906368 linear 8:17 2048
os-tmp: 0 8388608 linear 8:2 5244928
os-var: 0 8388608 linear 8:2 13633536
os-swap: 0 5242880 linear 8:2 2048
os-root: 0 8388608 linear 8:2 22022144
docker-253:3-247472-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing 

vgdocker-lvdocker: 0 62906368 linear 
os-tmp: 0 8388608 linear 
os-var: 0 8388608 linear 
os-swap: 0 5242880 linear 
os-root: 0 8388608 linear 
docker-253:3-247472-pool: 0 209715200 thin-pool 79 179/524288 4688/1638400 - rw discard_passdown queue_if_no_space 

DOCKER_STORAGE_OPTIONS=

I don't have any custom storage options configured.

[rhvgoyal]

Found it. I think it has something to do with ext3 file system. I can see the problem too. Try a different file system, say ext4 or xfs and you would not face the problem.

/dev/vdb1 on /var/lib/docker type ext3 (rw,relatime,seclabel,data=ordered)

• Before pull image
  [root@centos7-generic-2 devicemapper]# du -h data
  293M data
• After pull image
  [root@centos7-generic-2 devicemapper]# du -h data
  501M data
• After image removal
  [root@centos7-generic-2 devicemapper]# du -h data
  501M data

So for some reason if loopback file is on ext3 filesystem, it does not work. May be ext3 is old enough that loop discards don't work on it.

[alexDrinkwater]

Thanks, as soon as I saw the filesystem was ext3 I thought that could be it. I guess if you were able to reproduce the problem on ext3 then we could call this issue closed. It will be a good reference for others.

[stanislavb]

For the reference, my working examples were Docker 1.6.2 on xfs filesystem and Docker 1.6.0 and 1.5.0 on ext4.

[rhvgoyal]

@vbatts

Can we close this. Space reclaim works just fine on xfs and ext4. ext3 seems to be too old to support it.

[thaJeztah]

@rhvgoyal @vbatts IIRC, there's a compatibility-check that checks the underlying file-system. Perhaps we should add a check for ext3 if that is the issue here?

edit:
for reference; daemon/graphdriver/overlay/overlay.go#L115

[rhvgoyal]

Loop driver uses fallocate() to punch a hole in file when discard comes in. Linux man page of fallocate() verifies that it is not supported on ext3.

/*
Not all filesystems support FALLOC_FL_PUNCH_HOLE; if a filesystem doesn't support the operation, an error is returned. The operation is supported on at least the following filesystems:
* XFS (since Linux 2.6.38)
* ext4 (since Linux 3.0)
* Btrfs (since Linux 3.7)
* tmpfs (since Linux 3.5)
*/

[rhvgoyal]

@thaJeztah

If somebody wants to run docker on ext3, I guess we should allow that. Just that they will not get discard support hence loop file size will not shrink when image/containers are deleted.

[thaJeztah]

@rhvgoyal how about showing this in docker info? Similar to the Udev Sync Supported output.

[rhvgoyal]

@thaJeztah

We already have an entry in docker info "Backing filesystem". I am not sure why does it say extfs instead of being precise about ext2/ext3/ext4.

May be a warning at startup time in logs should do here. Something similar to warning about using loop devices for thin pool.

[rhvgoyal]

@thaJeztah

And I think we should do it only if lot of people are impacted by this. If not, then we are just creating more work and code and it might not be worth it.

[vbatts]

the syscall.Statfs_t struct, with the Type field on ext2, ext3, and ext4 all return 0xef53, and that is what we are using to detect filesystem magic.

[thaJeztah]

the syscall.Statfs_t struct, with the Type field on ext2, ext3, and ext4 all return 0xef53, and that is what we are using to detect filesystem magic.

Bummer. Would have been good to have that information to make it easier to identify reported issues.

I guess, we should just close this then?

[vbatts]

closing as this is an issue with ext3 being outdated. Please use ext4 or xfs

[dbabits]

do you think this could be better documented in the main Docker documentation somehow?

Whether the issue is with the filesystem or not, look at how much confusion this has generated across two recent issues.

Thanks.

[dbabits]

reterring to issue #9786

[thaJeztah]

@dbabits yes, I think mentioning that ext3 has some issues could be worth mentioning in the docs. No idea yet what an appropriate location would be.

[tomlux]

Have same/similar problem with XFS.
Was on kernel "3.10.0-123.el7.x86_64", but updated now on "3.10.0-229.el7.x86_64".

Eveything is deleted (container, images) but data still contains 100GB.
Any ideas, help?

[root@docker0101 ~]# ls -alh /data/docker/devicemapper/devicemapper/
total 80G
drwx------ 2 root root 32 Jun 8 16:48 .
drwx------ 5 root root 50 Jun 9 07:16 ..
-rw------- 1 root root 100G Jul 16 21:33 data
-rw------- 1 root root 2.0G Jul 17 09:20 metadata

[root@docker0101 ~]# uname -a
Linux docker0101 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@docker0101 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

[root@docker0101 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

[root@docker0101 ~]# docker images -a
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

[root@docker0101 ~]# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-253:0-268599424-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 85.61 GB
Data Space Total: 107.4 GB
Data Space Available: 40.91 MB
Metadata Space Used: 211.4 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 40.91 MB
Udev Sync Supported: true
Data loop file: /data/docker/devicemapper/devicemapper/data
Metadata loop file: /data/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.7.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 11.58 GiB
Name: docker0101

[root@docker0101 ~]# dmsetup table
vg1-lvol0: 0 167772160 linear 8:16 2048
docker-253:0-268599424-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing

[root@docker0101 ~]# dmsetup status
vg1-lvol0: 0 167772160 linear
docker-253:0-268599424-pool: 0 209715200 thin-pool 71359 51606/524288 1306264/1638400 - ro discard_passdown queue_if_no_space

[root@docker0101 ~]# cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS=

[root@docker0101 ~]# rpm -qi docker
Name : docker
Version : 1.6.2
Release : 14.el7.centos
Architecture: x86_64
Install Date: Fri 17 Jul 2015 09:19:54 AM CEST
Group : Unspecified
Size : 33825026
License : ASL 2.0
Signature : RSA/SHA256, Wed 24 Jun 2015 05:43:12 AM CEST, Key ID 24c6a8a7f4a80eb5
Source RPM : docker-1.6.2-14.el7.centos.src.rpm
Build Date : Wed 24 Jun 2015 03:52:32 AM CEST
Build Host : worker1.bsys.centos.org

[root@docker0101 ~]# ls -al /var/lib/docker
lrwxrwxrwx 1 root root 12 Jun 9 08:21 /var/lib/docker -> /data/docker

[root@docker0101 ~]# mount
/dev/sda5 on /var type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/vg1-lvol0 on /data type xfs (rw,relatime,attr2,inode64,noquota)

[jperrin]

@tomlux The devicemapper loopback mode you're using is mostly meant as a way to easily toy around with docker. for serious work, the loopback will be slower, and will have some limitations. I'd very highly recommend having a read over http://www.projectatomic.io/docs/docker-storage-recommendation/

You'll get better performance, and won't hit things like this, assuming you've applied all the system updates.

[rhvgoyal]

@tomlux

can you add "-s" to ls. That will give you actual blocks allocated to data and metadata files. Right now it is showing apparent size of the files.

docker info output is intriguing though. It seems to show high usage.

[tomlux]

[rhvgoyal]

@tomlux

So actual size of data and metadata files looks small. You can use "ls -alsh" so that sizes are more readable.

So data file size seems to be around 79MB and metadata file size is around 202KB.

I think somehow thin pool's stats about number of blocks used does not look right. Does a reboot of the machine fix the problem?

[tomlux]

I did a reboot after the kernel update without success.

[rhvgoyal]

Ok, so loop files are big and pool thinks it has lots of used blocks. So something is using those blocks. Can you give me output of.

• docker ps -a
• docker images
• ls /var/lib/dokcer/devicemapper/metadata/ | wc -l
• shutdown docker and run following.
  thin_dump /var/lib/docker/devicemapper/devicemapper/metadata | grep "device dev_id" | wc -l

This will tell how many thin devices are there in the pool.

[tomlux]

Hmm,
now we hae DEAD containers but no images.
Did already a reboot.

Seems to be something wrong with the devicemapper.
I don't want to spam this issue.
I can also "rm -f /var/lib/docker" and rebuild my containers. Everything is scripted.

[rhvgoyal]

do "dmsetup status"

Looks like pool is in bad shape and most likely it needs to be repaired.

[tomlux]

[swachter]

Hi,

I seemed to have the same problem under Ubuntu 14.04. However the cause where unwanted volumes (cf. blog http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/). Running the command

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

released a lot of disk space.

[bkeroackdsc]

This isn't fixed in any meaningful way. On a fully up-to-date Ubuntu 14.04.x installation (the latest LTS release) and with the latest version of Docker (installed via $ wget -qO- https://get.docker.com/ | sh), Docker will continuously leak space with no easy way to reclaim. docker stop $(docker ps -q) && docker rm $(docker ps -q -a) && docker rmi $(docker images -q) only releases a small amount of space.

The only way to reclaim all space is with the following hack:

$ sudo service docker stop
$ sudo rm -rf /var/lib/docker
$ sudo service docker start

Which then requires re-pulling any images you might need.

[thaJeztah]

@bkeroackdsc could that space also be related to "orphaned" volumes?

[sagiegurari]

I agree with @bkeroackdsc this is not solved.
I asked @rhvgoyal why he wanted to close this case so much.
at the end, I dropped from docker specifically because of this issue.
it is killing this product.

orphaned or not orphaned, there is no good, easy, way to cleanup space.
there needs to be a docker cli option for cleanup, a good status report cli option and also some sort of monitoring for the diskspace as this issue happens to too many people on many platforms.

[thaJeztah]

@bkeroackdsc @sagiegurari

Reason I'm asking is that orphaned volumes are not related to this issue,
not related to devicemapper.

Orphaned volumes are not a bug, but a misconception that volumes defined
for a container are automatically deleted when the container is deleted.
This is _not_ the case (and by design), because volumes can contain data
that should persist after a container is deleted.

_To delete a volume together with a container_, use docker rm -v [mycontainer].
Volume management functions will be added (see https://github.com/docker/docker/pull/14242 and https://github.com/docker/docker/issues/8363),
and will allow you to manage "orphaned" volumes.

A growing size of /var/lib/docker does not have to be an indication
that devicemapper is leaking data, because a grow in size of that directory
can also be the result of (orphaned) volumes that have not been cleaned up
by the user (docker stores all its data in that path).

[sagiegurari]

I really hope those 2 items will give the needed capabilities.
I did understand the second item, but the first (#14242), doesn't explain at the top what it is about apart of that its a volume api (meaning, not sure what capabilities it gives).

[thaJeztah]

@sagiegurari it's part of the requirements to implement image volume management (there are some other open PR's issues). End goals is to make volumes a first-class citizen in Docker, that can be created/deleted/management separate of the containers that make use of them.

[rxacevedo]

@swachter Thanks for posting that workaround, I reclaimed 6GB with the aforementioned image.

[nathwill]

we fixed the leaky volume issue with a dirty hack, since it was preventing the docker daemon from starting up before the service timed out on our high-churn docker hosts:

PRODUCTION [[email protected] ~]$ cat /etc/systemd/system/docker.service.d/docker-high-churn.conf 
[Service]
ExecStartPre=-/bin/rm -rf /var/lib/docker/containers
ExecStopPost=-/bin/rm -rf /var/lib/docker/volumes

which fixes the issue without flushing the pre-cached images.

[rhvgoyal]

Can we discuss the issue of leaky volumes in a separate issue. Discussing it here gives the impression that it is a device mapper issue while it is not.

[brian-dlee]

@tomlux @rhvgoyal

Did you ever come to a conclusion on what occurring in the CentOS 7 box? My docker host is nearly identical, and I was experiencing the same issue. I followed through to the point in which @rhvgoyal asked to run the thin_dump command. After, I went to start the docker daemon up and it wouldn't start. I've just deleted /var/lib/docker and restarted since then, but I just wanted to know if a resolution was found as I (as well as others) may run into it again.

[root@Docker_Sandbox_00 devicemapper]# thin_dump /var/lib/docker/devicemapper/devicemapper/metadata | grep "device dev_id" | wc -l
102

[root@Docker_Sandbox_00 devicemapper]# systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
 [root@Docker_Sandbox_00 devicemapper]# systemctl -l status docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: exit-code) since Tue 2015-10-27 08:24:47 PDT; 37s ago
     Docs: https://docs.docker.com
  Process: 45244 ExecStart=/usr/bin/docker daemon -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 45244 (code=exited, status=1/FAILURE)

Oct 27 08:24:45 Docker_Sandbox_00 systemd[1]: Starting Docker Application Container Engine...
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.512617474-07:00" level=info msg="[graphdriver] using prior storage driver \"devicemapper\""
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.526637164-07:00" level=info msg="Option DefaultDriver: bridge"
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.526719113-07:00" level=info msg="Option DefaultNetwork: bridge"
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.589016574-07:00" level=warning msg="Running modprobe bridge nf_nat br_netfilter failed with message: modprobe: WARNING: Module br_netfilter not found.\n, error: exit status 1"
Oct 27 08:24:46 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:46.625324632-07:00" level=info msg="Firewalld running: true"
Oct 27 08:24:47 Docker_Sandbox_00 docker[45244]: time="2015-10-27T08:24:47.142468904-07:00" level=fatal msg="Error starting daemon: Unable to open the database file: unable to open database file"
Oct 27 08:24:47 Docker_Sandbox_00 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 27 08:24:47 Docker_Sandbox_00 systemd[1]: Failed to start Docker Application Container Engine.
Oct 27 08:24:47 Docker_Sandbox_00 systemd[1]: Unit docker.service entered failed state.

[root@Docker_Sandbox_00 devicemapper]# df -ah
Filesystem                                  Size  Used Avail Use% Mounted on
/dev/vdb1                                    20G   20G   24K 100% /var/lib/docker
[root@Docker_Sandbox_00 devicemapper]# du -sh /var/lib/docker/devicemapper/devicemapper/data
20G     /var/lib/docker/devicemapper/devicemapper/data

[brian-dlee]

@tomlux @rhvgoyal

I found the source of my issue. Despite how it appeared, it was unrelated to the issues in the thread. It's just stemmed from a misunderstanding with how docker worked. All, the dead containers were still holding on to the disk space they had allocated during their execution. I just had to remove all the container carcasses to free up the disk space. I do remember this coming up in this thread, but I thought is only regarded mounted volumes, not the disk space allocated by the container.

# Beware this removes ALL containers
docker rm -f $(docker ps -aq) 

@tomlux This may have been your issue as well since your output of docker ps -a showed several Dead containers.

[awolfe-silversky]

docker rm is not freeing container's disk space

boot2docker recent in OS X VirtualBox. OS X fully patched.

I'm building a fat (47 GB) container and it has a problem indicating I should rebuild the container. So I stopped the container and did docker rm. Double-checking using docker ssh 'df -h', I find disk usage is still 47 GB. The container has 75 GB.

So I'm going to need to kill the docker VM again.

Can we get this done?

[thaJeztah]

@awolfe-silversky is disk-space _inside_ the VM returned? If it's outside the VM, this may be unrelated.

[thaJeztah]

@awolfe-silversky also; did you remove the _image_ as well? removing just the container may not help much if the image is still there

[SvenDowideit]

@awolfe-silversky this issue is about devicemapper - and if you're using docker-machine/boot2docker, then you're much more likely to be running aufs. I also wonder if you've docker rmi'd your big image.

its worth running docker images, and docker info to see if things really are as dire as you make it sound :)

(yes, if you still have the vm, and the image is removed, then we should open a new issue and debug further, as you've found a weird corner case)

[awolfe-silversky]

I did not remove the image. It is from an internal docker registry.
I used an awk script to size the images - total 6.9 GB.

docker images -a | awk '(FNR > 1) { imgSpace = imgSpace + $(NF - 1); }
END { print "Image space is " imgSpace; }'
Image space is 6909.01

It's dirty but I know that all the image sizes are in MB.

Here's how I was trying to diagnose usage:

 file:///Andrew-Wolfe-MacBook-Pro.local/Users/awolfe/DataStores
awolfe_10063: docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

 file:///Andrew-Wolfe-MacBook-Pro.local/Users/awolfe/DataStores
awolfe_10064: docker-machine ssh 'awolfe-dbhost' 'df -h'
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     7.0G    123.8M      6.9G   2% /
tmpfs                     3.9G         0      3.9G   0% /dev/shm
/dev/sda1                71.0G     47.2G     20.2G  70% /mnt/sda1
cgroup                    3.9G         0      3.9G   0% /sys/fs/cgroup
none                    464.8G    379.9G     84.8G  82% /Users
/dev/sda1                71.0G     47.2G     20.2G  70% /mnt/sda1/var/lib/docker/aufs

[ror6ax]

Right now I have box with 0 images.
docker volume ls -qf dangling=true shows nothing.
docker volume ls shows a lot of volumes, which are, by definition, orphaned, since there's no-images to own them.
docker volume rm $(docker volume ls) shows lots of such messages:

Error response from daemon: get local: no such volume
Error response from daemon: Conflict: remove 6989acc79fd53d26e3d4668117a7cb6fbd2998e6214d5c4843ee9fceda66fb14: volume is in use - [77e0eddb05f2b53e22cca97aa8bdcd51620c94acd2020b04b779e485c7563c57]

Device mapper directory eats up 30 GiG.
Docker version 1.10.2, build c3959b1
CentOS 7, 3.10.0-327.10.1.el7.x86_64

[ror6ax]

Data Space Used: 33.33 GB
 Data Space Total: 107.4 GB
 Data Space Available: 915.5 MB
 Metadata Space Used: 247 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 915.5 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)

Also, why is default installation uses 'strongly discouraged' storage option?
Why wasn't I told so at installation?

[ir-fuel]

I have exactly the same problem here on an Amazon Linux EC2 instance.

Linux ip-172-31-25-154 4.4.5-15.26.amzn1.x86_64 #1 SMP Wed Mar 16 17:15:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

On instances where I install new docker images on a regular basis the only solution is to do the following:

service docker stop
yum remove docker -y
rm -rf /var/lib/docker
yum install docker -y
service docker start

I don't really think such a thing is acceptable in a production environment

some extra info:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G   20G     0 100% /

[mercuriete]

As this bug is there for years and is seem that is not closed yet, could you put in the docker documentation about devidemapper how to destroy safety all docker information?
i mean, in this page: https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/
put something like "Cleaning device mapper" and how to do it.

I will try to do rm -rf /var/lib/docker but I'm not feel confortable doing that. Can somebody tell me if is safe?

I am using gentoo linux in my daily laptop and I tried docker for learning but is filling up my disk and reinstall the whole system is not an option because is not a VM and reinstall gentoo takes time.

Thank you for your work.

[ir-fuel]

@mercuriete On your dev machine just uninstall docker, delete the directory and reinstall it. Works fine.

[aleks-f]

@ir-fuel: I just did that and now I have this:

$ sudo service docker-engine start
Redirecting to /bin/systemctl start  docker-engine.service
Failed to start docker-engine.service: Unit docker-engine.service failed to load: No such file or directory.
$ uname -a
Linux CentOS7 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[ir-fuel]

I'm using service docker start

[mercuriete]

@ir-fuel thanks, it works fine. :+1:

[asarkar]

Reinstalling docker to release disk space is the most ridiculous answer that I came across while looking for a solution for this issue. Not only that's a waste of time, it's not even allowed in most environments. It's a good way to get paid if you're an hourly worker.

[ir-fuel]

I completely agree. It's amazing that a product like Docker just keeps on eating away disk space with nothing you can do about it except for uninstall/reinstall.

[ror6ax]

Checking in to this issue yet another time to see nothing has changed. +1

[awolfe-silversky]

This issue is marked closed. We need a resolution. No workaround, no reconfiguration. What is the real status, and what are the configuration settings that are implicated? Dropping and recreating a production Docker node is not acceptable.

[ir-fuel]

And what is the alternative? How do we go about avoiding this?

[ror6ax]

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande" [email protected] wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-228068397, or mute
the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

[blalor]

There's always rkt…

[ripienaar]

the general bitchy unhelpful snark is no doubt why no-one from upstream cares to give you proper answers.

[asarkar]

no one forced you to use Docker.

That's like Oracle telling a Java developer to use PHP due to a JVM bug. That's also not consistent with the elevator pitch here

Three years ago, Docker made an esoteric Linux kernel technology called containerization simple and accessible to everyone.

I'm sure a lot of people are grateful that Docker took off like it did and that couldn't have happened without volunteering from the community. However, it shouldn't be this hard to admit that it has it's problems too without implicitly dropping the "I'm a upstream contributor so shut up and listen" line whenever someone brings up an unlikable point.

[ror6ax]

Wait. I did report an issue, provided the details of my machine and setup,
which I'm not obliged to. None of devteam responded to my and others bug
reports in half a year period. Now I stated this fact, you call my behavior
bitchy? Do you even open-source? I'm looking for Go project to work on, and
it will not be Docker, I give you that. Is this your goal?
On 23 Jun 2016 16:45, "gregory grey" [email protected] wrote:

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande" [email protected] wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/3182#issuecomment-228068397,
or mute the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

[thaJeztah]

First of all, if you still have this problem, please open a new issue;

Wait. I did report an issue

You replied on a 3 year old, closed issue; following the discussion above, the original issue was resolved. Your issue _may_ be the same, but needs more research to be sure; the errors you're reporing indicate that it may actually be something else.

I really recommend to open a new issue, not commenting on a closed issue

provided the details of my machine and setup, which I'm not obliged to.

You're not _obliged_ to, but without any information to go on, it's unlikely to be resolved. So, when reporting a bug, please include the information that's asked for in the template:
https://raw.githubusercontent.com/docker/docker/master/.github/ISSUE_TEMPLATE.md

None of devteam responded to my and others bug reports in half a year period.

If you mean "one of the maintainers", please keep in mind that there's almost 24000 issues and PRs, and less than 20 maintainers, many of whom doing that besides their daily job. Not every comment will be noticed especially if it's on a closed issue.

If this is not recommended feature to use - why is it silent and default one?

It's the default if _aufs_, _btrfs_, and _zfs_ are not supported, you can find the priority that's used when selecting drivers; see daemon/graphdriver/driver_linux.go. It's still above overlay, because unfortunately there's some remaining issues with that driver that _some_ people may be affected by.

Automatically selecting a graphdriver is just to "get things running"; the best driver for _your_ situation depends on _your_ use-case. Docker cannot make that decision automatically, so this is up to the user to configure.

If you do not care for people using devicemapper - I might be even ok with this.

Reading back the discussion above, I see that the upstream devicemapper maintainers have looked into this _multple times_, trying to assist users reporting these issues, and resolving the issue. The issue was resolved for those that reported it, or in some cases, depended on distros updating devicemapper versions. I don't think that can be considered "not caring".

Also, why is default installation uses 'strongly discouraged' storage option?

Running on loop devices is fine for getting docker running, and currently the only way to set up devicemapper automatically. For production, and to get a better performance overall, use direct-lvm, as explained in the devicemapper section in the storage driver user guide.

Why wasn't I told so at installation?

That's out of scope for the installation, really. If you're going to use some software in production, it should be reasonable to assume that you get yourself familiar with that software, and know what's needed to set it up for your use case. Some maintainers even argued if the warning should be output at all. Linux is not a "holding hands" OS (does your distro show a warning that data loss can occur if you're using RAID-0? If you have ports opened in your Firewall?)

[shadowmint]

Deeply reluctant as I am, to once again resurrect this ancient thread, there is still no meaningful advice in it about how to work around this issue on an existing machine encountering this issue.

This is my best effort at a tldr; for the entire thread; I hope it helps others who find this thread.

Issue encountered

Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're using ext3.

Resolution

You're out of luck. Upgrade your file system or see blowing docker away at the bottom.

Issue encountered

Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're _not_ using ext3 (eg. system currently using xfs or ext4)

Resolution

You may be able to reclaim space on your device using standard docker commands.

Read http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/

Run these commands:

docker volume ls
docker ps
docker images

If you have nothing listed in any of these, see blowing docker away at the bottom.

If you see old stale images, unused containers, etc. you can perform manual cleanup with:

# Delete 'exited' containers
docker rm -v $(docker ps -a -q -f status=exited)

# Delete 'dangling' images
docker rmi $(docker images -f "dangling=true" -q)

# Delete 'dangling' volumes
docker volume rm $(docker volume ls -qf dangling=true)

This should reclaim much of the hidden container space in the devicemapper.

Blowing docker away

Didn't work? You're out of luck.

Your best bet at this point is:

service docker stop
rm -rf /var/lib/docker
service docker start

This will destroy all your docker images. Make sure to export ones you want to keep _before_ doing this.

Ultimately, please read https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production; but I hope this will assist others who find this thread.

If you have problems with using advice above open a new ticket that specifically addresses the issue you encounter and link to this issue; do not post it here.

[gg7]

rm -rf /var/lib/docker

You can also use nuke-graph-directory.sh.

[gdring2]

After removing the files as stated above, I can't start Docker any more:
Mar 02 04:53:40 pmdc001b dockerd[29522]: Error starting daemon: error initializing graphdriver: open /var/lib/docker/devicemapper/devicemapper/data: no such file or directory

[pauvos]

Just ran into this issue on CentOS 7.3 and didn't want to debug devmapper issues that exist for more than 3 years, so I followed this DC/OS guide, purged the original packages, switched over to overlayfs and everything seems to work fine now: https://dcos.io/docs/1.7/administration/installing/custom/system-requirements/install-docker-centos/ (had to modify the ExecStart command for docker version 17.03 though -> "dockerd --storage-driver=overlay")

Server Version: 17.03.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
...
Operating System: CentOS Linux 7 (Core)

(purging volumes, images and containers didn't help. Deleting stuff in /var/lib/docker led to the problem described by @gdring2 )

[invisibleroads]

Running docker system prune freed a lot of space on my machines.

https://docs.docker.com/engine/reference/commandline/system_prune/

[Noah-Huppert]

Well, this is kind of... lame.

In my case I found this issue after I uninstalled Docker and deleted the the /var/lib/docker directory, so I couldn't run the equivalent of service docker stop ... service docker start.

I found that my system was not reporting the space from deleting /var/lib/docker as freed (I had ~14 GB sitting in what seemed like limbo).

The fix to this is to simply reload your file system, in my case I just rebooted and the space was reclaimed.

[shahaf600]

I can't believe it's still an issue! come on guys, i'm still having that

[thaJeztah]

@shahaf600 what version of docker are you running? Also see my comment above; https://github.com/moby/moby/issues/3182#issuecomment-228298975

Without details there's not much to say about your situation; your case could be caused by a different issue, but resulting in a similar result.

[liusi123]

good

[misterbigstuff]

after buying one of these pieces of garbage and seeing the state of support, i returned it.

[darkpixel]

There's your first problem @misterbigstuff...you bought something that's open source?

[misterbigstuff]

and returned it