Kubevirt: Devicemapper: Can't set cookie dm_task_set_cookie failed

[sylvainmouquet]

Hello,

I have this error with docker version 17.06.0-ce :
docker -v Docker version 17.06.0-ce, build 02c1d87

devmapper: Error activating devmapper device for '6bf91878789809febd403ed5e87f715f4c9a2d3b7f257e90fbe3d34dd0f8e816-init': devicemapper: Can't set cookie dm_task_set_cookie failed

When i try to restart docker, with
service docker restart

i have this error :
Jul 17 14:11:50 dockerd[46867]: time="2017-07-17T14:11:50+02:00" level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead" Jul 17 14:11:50 dockerd[46867]: time="2017-07-17T14:11:50.919640600+02:00" level=info msg="libcontainerd: new containerd process, pi d: 46875" Jul 17 14:11:51 dockerd[46867]: time="2017-07-17T14:11:51.922847500+02:00" level=warning msg="failed to rename /mnt/block_drive/dock er/tmp for background deletion: rename /mnt/block_drive/docker/tmp /mnt/block_drive/docker/tmp-old: file exists. Deleting synchronously" Jul 17 14:11:52 dockerd[46867]: time="2017-07-17T14:11:52.123141700+02:00" level=warning msg="devmapper: Usage of loopback devices i s strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section." Jul 17 14:11:52 dockerd[46867]: Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem ve rification failed: devicemapper: Can't set cookie dm_task_set_cookie failed Jul 17 14:11:52 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE Jul 17 14:11:52 systemd[1]: Failed to start Docker Application Container Engine.

For solving i restart the VM and restart docker

[rmohr]

Hi @sylvainmouquet,

Thanks for reporting this! Are you using our Vagrant setup, or do you run your own infrastructure? We have seen this in our CentOS based Vagrant setup (https://github.com/kubevirt/kubevirt/issues/241), but it should be fixed in CentOS already.

Running

echo 'y' | sudo dmsetup udevcomplete_all

on affected nodes should be sufficient, instead of restarting docker.

[sylvainmouquet]

It's a Centos VM on VMWare (not vagrant)
cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)
If the same problem occurs i will try your solution and post the result here

[sylvainmouquet]

I have the problem this morning on the same machine. I have tried your solution :

echo 'y' | sudo dmsetup udevcomplete_all
This operation will destroy all semaphores with keys that have a prefix 3405 (0xd4d).
Do you really want to continue? [y/n]: 128 semaphores with keys prefixed by 3405 (0xd4d) destroyed. 0 skipped.

I have restart the docker build and it works

[sylvainmouquet]

The problems is arrived this night with our CI
3 docker build are started in //

first
Step 9/15 : RUN cd /usr/share/nginx/html && npm install && node_modules/gulp/bin/gulp.js env && node_modules/gulp/bin/gulp.js dev_jenkins && node_modules/gulp/bin/gulp.js manifest layer does not exist
devmapper: Error activating devmapper device for 'ef9c7fb27919c930a8a73e7e78acefa519a25517698a393ee07c186c0c0b71c0-init': devicemapper: Can't set cookie dm_task_set_cookie failed

second
Error removing intermediate container 9331ff934f74: driver "devicemapper" failed to remove root filesystem for 9331ff934f74c52313bb188c6aa1d127ca0ed6d76c9625022769837633935c52: failed to remove device ea139ffbc51a68c4420f9b19d279caa060292aa0cf4cf03d279ba85ae5cc0a4a: devicemapper: Can not set cookie: dm_task_set_cookie failed

third
devmapper: Error activating devmapper device for '4523ecadb82bad5946c1748f6f3be88df3c105b97f2db083966487dda80566b0-init': devicemapper: Can't set cookie dm_task_set_cookie failed

$ echo 'y' | sudo dmsetup udevcomplete_all
This operation will destroy all semaphores with keys that have a prefix 3405 (0xd4d).
Do you really want to continue? [y/n]: 128 semaphores with keys prefixed by 3405 (0xd4d) destroyed. 0 skipped.
$ echo 'y' | sudo dmsetup udevcomplete_all
This operation will destroy all semaphores with keys that have a prefix 3405 (0xd4d).
Do you really want to continue? [y/n]: 0 semaphores with keys prefixed by 3405 (0xd4d) destroyed. 0 skipped.

[sylvainmouquet]

Same error this morning. Does the devicemapper allow to build multiple image in parralel ?

docker info

Containers: 8
 Running: 1
 Paused: 0
 Stopped: 7
Images: 248
Server Version: 17.06.0-ce
Storage Driver: devicemapper
 Pool Name: docker-253:0-204103596-pool
 Pool Blocksize: 65.54kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 8.269GB
 Data Space Total: 107.4GB
 Data Space Available: 14.24GB
 Metadata Space Used: 23.41MB
 Metadata Space Total: 2.147GB
 Metadata Space Available: 2.124GB
 Thin Pool Minimum Free Space: 10.74GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /mnt/block_drive/docker/devicemapper/devicemapper/data
 Metadata loop file: /mnt/block_drive/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.633GiB
Name: *******
ID: 7HT3:WIHY:7S6V:MBC5:U6HF:GE5A:3KU4:CG5H:3HVC:OQSQ:ZLI6:EMUY
Docker Root Dir: /mnt/block_drive/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use --storage-opt dm.thinpooldev to specify a custom block storage device.

[rmohr]

@sylvainmouquet I think that some people solved massive parallel builds by increasing the semaphore count. @davidvossel I think you mentioned that. In general, I don't think anyone here can really help you with that issue. It might make sense to go through https://github.com/moby/moby/issues/33603 and https://github.com/projectatomic/docker/pull/256. As far as I know, this issue should be fixed on an up-to-date CentOS.

[sylvainmouquet]

I have changed the number of semaphore from 128 to 1024. I post the result here

[xinqiyang]

echo 'y' | sudo dmsetup udevcomplete_all

this is ok ,thanks .

[zephinzer]

okayed for echo 'y' | sudo dmsetup udevcomplete_all too given error:

xxxxx | ERROR: for <container_name>  Cannot create container for service <service_name>: devmapper: Error activating devmapper device for '<container_id>-init': devicemapper: Can't set cookie dm_task_set_cookie failed

kernel: Linux 3.10.0-514.10.2.el7.x86_64
system: centos-release-7-3.1611.el7.centos.x86_64
docker -v: Docker version 17.06.0-ce, build 02c1d87

[junoteam]

Hello, I have the same type of error:

08:04:46 docker: Error response from daemon: devmapper: Error activating devmapper device for '4380222828fc9f0e3e8dfd832611a519e62c78df67f2ff47c9cd26e6127d63f9-init': devicemapper: Can't set cookie dm_task_set_cookie failed.

On Ubuntu 14.04
AWS

[junoteam]

When I'm trying to remove container, I get this:

docker rm -f 3f4f83744a68
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 3f4f83744a68c77340dc08ada1989016d02eb5b44c5efa7f5844cd26219b2a77: failed to remove device 48f7ab1f979e64b9e53de6072627bf0c0a423f8cce89885b3ff8dcd1e29152ee: devicemapper: Can not set cookie: dm_task_set_cookie failed

[junoteam]

After I did this:

root@ip-10-0-8-107:~# echo 'y' | sudo dmsetup udevcomplete_all
This operation will destroy all semaphores with keys that have a prefix 3405 (0xd4d).
Do you really want to continue? [y/n]: 128 semaphores with keys prefixed by 3405 (0xd4d) destroyed. 0 skipped.

I successfully removed containers:

root@ip-10-0-8-107:~# docker rm -f 3f4f83744a68
3f4f83744a68
root@ip-10-0-8-107:~# docker rm -f f667397b8213
f667397b8213

Do someone knows why this error happens?

[sylvainmouquet]

I have found the scenario in my platform. The problem arrived because we have configured 2 continuous integration tools with same jobs on the same virtual machine.
The fact to run 2 docker build with the same Dockerfile generated the problem

[fabiand]

Yes, it seems to be docker related. We actually thought (@rmohr has the details) that a fix landed in (at least) CentOS. But maybe that fix was not enough …

Thanks for the workaround @junoteam

[junoteam]

@fabiand This is only a workaround. The problem will be back after day or three. Unfortunately, I still can't find the root of this problem.

[mlushpenko]

Having the same problem.
echo 'y' | sudo dmsetup udevcomplete_all this allowed me to remove containers, but what I am getting from this discussion is that the fix is just temporal, any final solution like changing some system settings?

System info

~$ uname -a
Linux amsddoppe10 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

:~$ docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:20:36 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:21:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

[elliotpeele]

I'm seeing this issue when attempting to launch a service in docker swarm on CentOS 7 using devicemapper storage driver on Docker CE 17.06.1.

x8nueqqb1txj        tenant1_rabbitmq.2       docker.example.com/foo/rabbitmq:17w12   s01w04.example.com   Running             Running about an hour ago
8t1azr0cupq2         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"
4qtslqtvvhlm         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"
uw87lugpcxli         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"
e5zqejda5n23         \_ tenant1_rabbitmq.2   docker.example.com/foo/rabbitmq:17w12   s01w06.example.com   Shutdown            Rejected about an hour ago   "devmapper: Error activating d…"

Each one of the failed tasks have an error like devmapper: Error activating devmapper device for '0a1bbd565d4619df84af2f06c85fe4dbda24fa0aecb4bd8ee7689f0238113ac2-init': devicemapper: Can't set cookie dm_task_set_cookie failed.

# docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:31:53 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 23:01:50 2017
 OS/Arch:      linux/amd64
 Experimental: true

uname -a
Linux s01m01.example.com 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[hatdropper1977]

Suggested permanent fix?

I fixed this issue by simply upgrading my Linux kernel from 3.10.0 to 4.4.83 (Current LT).

Background info:

• I faced this very issue when I upgraded from _Docker version 1.13.1-cs4, build e46aec0_ to _Docker version 17.06.1-ce, build 874a737_
• Unmounting all docker volumes and then restarting the daemon provided a short lived fix
  
  
  
  • It worked for one run of _docker-compose_ and then broke on the second
  
  
• Running the following command provided a temporary fix as well, I needed to run it before each __docker_compose__ build.

$ echo 'y' | sudo dmsetup udevcomplete_all

• I initially upgraded the kernel from 3.10.0 to 3.10.107. This _did not_ fix the issue.

[svscorp]

I wonder did someone report it to Docker?

Docker version 17.06.0-ce, build 02c1d87

Same issue here.

[elliotpeele]

Filed an issue against Docker: https://github.com/docker/for-linux/issues/85

[osman-masood]

I'm on Ubuntu 14.04 and tried @hatdropper1977's fix above -- upgraded the Linux kernel to 4.2.0, and so far the issue hasn't reoccurred.

Docker version 17.06.1-ce build 874a737.

[hatdropper1977]

@osman-masood As an alternate approach, if you have the ability to wipe out the server and start with a fresh install, I would recommend it. Using a fresh version of Centos 7, with Kernel 3.10.0 I could install the most recent version of Docker (17.06.1) and it works without an issue. The process of upgrading from a machine running Docker 13 to Docker 17 appears to have broken it.

[osman-masood]

@hatdropper1977 Interesting, thanks for the tip. After running smoothly for about a day, now the containers can't restart automatically, and drop the dm_task_set_cookie failed error message:

time="2017-09-01T05:55:23.653665786Z" level=error msg="fatal task error" error="devmapper: Error activating devmapper device for '32226eae16a58dcbb6401a76afcc0e874d4d4137b63f3f0bea1062bde3d0785d-init': devicemapper: Can't set cookie dm_task_set_cookie failed" module="node/agent/taskmanager" node.id=umq1q3tfm1v7d39h4xazkhb33 service.id=j0tuemye49pvcov1ss90cjp4e task.id=1o6gaz931et5382cubes0a46z 

[ankitgupta07]

Did anybody face this problem on version 17.03.1? Basically I want to know if downgrading from 17.06 to 17.03.1 will work or not.

[elmariofredo]

For others like me who miss it, it's already fixed in 17.06.2-ce see https://docs.docker.com/release-notes/docker-ce/#17062-ce-2017-09-05