Swarmkit: If a Docker node dies, the container will get rescheduled on any host randomly completely ignoring my labels!

[yank1]

Hi

There is a blog "The Sad State of Docker" from Blog

Labels and constraints are a neat Docker feature that lets me schedule containers to run on hosts with a specific characteristic. For example, I can set an arbitrary label on a Docker node that I can reference later when running a container such as disk type, node name, foo, or bar. In theory when I run a container with a label, it should only run on a host that uses the specified label and fail if the request cannot be fulfilled. With Swarm 1.12, my labels rarely work. If a Docker node dies, the container will get rescheduled on any host randomly completely ignoring my labels!

Is it true ?

[dongluochen]

I'm not sure which Docker version the blog post was referring to. At Docker 1.12 (not RCs) node labels and constraints were tested against node failure case. We need more information on node label configuration and service options.

In theory when I run a container with a label, it should only run on a host that uses the specified label and fail if the request cannot be fulfilled.

This is not accurate. Swarm 1.12 supports constraints on service, not container. Constraints are specified in README.

[dongluochen]

Here are 2 test cases on service constraints with Docker 1.12.1. You may follow the steps to validate the result. This test uses node.labels. If you see problem with other constraints, we appreciate your feedback.

In a 3 node cluster, set node label priority to different level.

ubuntu@ip-172-19-241-145:~$ docker node ls
ID                           HOSTNAME           STATUS  AVAILABILITY  MANAGER STATUS
0cno8bg1ye5kg6gu0yklkqx72    ip-172-19-241-146  Ready   Active        Reachable
77yrknu2sif0wczt5tig2ppwu *  ip-172-19-241-145  Ready   Active        Leader
edmmp6b3618h6g2uz682bervn    ip-172-19-241-144  Ready   Active
ubuntu@ip-172-19-241-145:~$ docker node update --label-add priority=1 ip-172-19-241-144
ip-172-19-241-144
...
ubuntu@ip-172-19-241-145:~$ docker node inspect ip-172-19-241-144 --pretty | grep -i priority
 - priority = 1
ubuntu@ip-172-19-241-145:~$ docker node inspect ip-172-19-241-145 --pretty | grep -i priority
 - priority = 2
ubuntu@ip-172-19-241-145:~$ docker node inspect ip-172-19-241-146 --pretty | grep -i priority
 - priority = 2

The first test is to validate service cannot run if nodes do not meet service constraints.

Add a service with constraint node.labels.priority==1. It'd select node ip-172-19-241-144.

ubuntu@ip-172-19-241-145:~$ docker service create --name p1service --constraint node.labels.priority==1 redis
3a5r9wbydkfqdnxj03vgq3yyk
ubuntu@ip-172-19-241-145:~$ docker service ps p1service
ID                         NAME         IMAGE  NODE               DESIRED STATE  CURRENT STATE           ERROR
a1y4myy8n6qld50wuyuq3ch2j  p1service.1  redis  ip-172-19-241-144  Running        Running 23 seconds ago

Shutdown node ip-172-19-241-144. Note the active instance is in Allocated state not assigned to any node.

ubuntu@ip-172-19-241-145:~$ docker node ls
ID                           HOSTNAME           STATUS  AVAILABILITY  MANAGER STATUS
0cno8bg1ye5kg6gu0yklkqx72    ip-172-19-241-146  Ready   Active        Reachable
77yrknu2sif0wczt5tig2ppwu *  ip-172-19-241-145  Ready   Active        Leader
edmmp6b3618h6g2uz682bervn    ip-172-19-241-144  Down    Active
ubuntu@ip-172-19-241-145:~$ docker service ps p1service
ID                         NAME             IMAGE  NODE               DESIRED STATE  CURRENT STATE               ERROR
bkx40ugcr6g1cmewu9ffeavll  p1service.1      redis                     Running        Allocated 19 seconds ago
a1y4myy8n6qld50wuyuq3ch2j   \_ p1service.1  redis  ip-172-19-241-144  Shutdown       Running about a minute ago
ubuntu@ip-172-19-241-145:~$ docker service ls
ID            NAME       REPLICAS  IMAGE  COMMAND
3a5r9wbydkfq  p1service  0/1       redis

Restart node ip-172-19-241-144 and the service instance starts running again.

ubuntu@ip-172-19-241-145:~$ docker node ls
ID                           HOSTNAME           STATUS  AVAILABILITY  MANAGER STATUS
0cno8bg1ye5kg6gu0yklkqx72    ip-172-19-241-146  Ready   Active        Reachable
77yrknu2sif0wczt5tig2ppwu *  ip-172-19-241-145  Ready   Active        Leader
edmmp6b3618h6g2uz682bervn    ip-172-19-241-144  Ready   Active
ubuntu@ip-172-19-241-145:~$ docker service ps p1service
ID                         NAME             IMAGE  NODE               DESIRED STATE  CURRENT STATE            ERROR
bkx40ugcr6g1cmewu9ffeavll  p1service.1      redis  ip-172-19-241-144  Running        Running 9 seconds ago
a1y4myy8n6qld50wuyuq3ch2j   \_ p1service.1  redis  ip-172-19-241-144  Shutdown       Complete 13 seconds ago
ubuntu@ip-172-19-241-145:~$ docker service ls
ID            NAME       REPLICAS  IMAGE  COMMAND
3a5r9wbydkfq  p1service  1/1       redis

The second test case is to validate tasks select the right nodes according to constraint on node failure.

Create a service with 3 replicas and validate they run on nodes with label priority=2.

ubuntu@ip-172-19-241-145:~$ docker service create --name p2service --replicas 3 --constraint node.labels.priority==2 redis
dwz9ocvwme1mup8br4j6wigeu

ubuntu@ip-172-19-241-145:~$ docker service ps p2service
ID                         NAME         IMAGE  NODE               DESIRED STATE  CURRENT STATE                   ERROR
534p6m0nloszl3zo3bk2834th  p2service.1  redis  ip-172-19-241-145  Running        Running less than a second ago
9jzxne4vvuyhiudo4ogpnarpd  p2service.2  redis  ip-172-19-241-146  Running        Running less than a second ago
a0vwpy0f2iuvtnp55xdvkx8b8  p2service.3  redis  ip-172-19-241-146  Running        Running less than a second ago

Reboot node ip-172-19-241-145, after it comes back reboot node ip-172-19-241-146. See the tasks bouncing between the 2 nodes. The tasks are never run on node ip-172-19-241-144.

ubuntu@ip-172-19-241-146:~$ docker service ps p2service
ID                         NAME             IMAGE  NODE               DESIRED STATE  CURRENT STATE                ERROR
5v5vgw2vocz62veykh8t8ke5t  p2service.1      redis  ip-172-19-241-145  Running        Running about a minute ago
534p6m0nloszl3zo3bk2834th   \_ p2service.1  redis  ip-172-19-241-145  Shutdown       Complete about a minute ago
48h3hqjalcqd3e4e8z8nhpib1  p2service.2      redis  ip-172-19-241-145  Running        Running about a minute ago
cpkuw4jsltu69dxm5wnak6z6z   \_ p2service.2  redis  ip-172-19-241-145  Shutdown       Complete about a minute ago
9jzxne4vvuyhiudo4ogpnarpd   \_ p2service.2  redis  ip-172-19-241-146  Shutdown       Complete 4 minutes ago
c3lfud25q625g4xjiqklm3pom  p2service.3      redis  ip-172-19-241-146  Running        Running 4 minutes ago
a0vwpy0f2iuvtnp55xdvkx8b8   \_ p2service.3  redis  ip-172-19-241-146  Shutdown       Complete 4 minutes ago

[dperny]

@yank1 does this answer your question sufficiently?

[yank1]

Thanks @dongluochen . The information is very helpful to us.

[dperny]

Closing because it's answered. Let us know if you have any more trouble!