Swarmkit: Global Servives should honor constraints

[aluzzardi]

Low priority but global services should only consider nodes that match constraints.

/cc @dongluochen

[dongluochen]

For now constraints are in scheduler filters which is after orchestrator. For a global service, orchestrator creates one task for each node. If a node fails the constraint, scheduler would not move it to ASSIGNED state so it remains in ALLOCATED. It would be re-evaluated and run if the constraint passes in the future. Do you think such tasks should not be created at all?

dchen@vm2:~$ sudo ./swarmctl -s /var/lib/docker/swarm/control.sock node ls
ID             Name  Membership  Status  Availability  Manager status
--             ----  ----------  ------  ------------  --------------
2sw5uj0yn6401  vm4   ACCEPTED    READY   ACTIVE
35clkmutzasnv  vm2   ACCEPTED    READY   ACTIVE        REACHABLE *

dchen@vm2:~$ sudo ./swarmctl -s /var/lib/docker/swarm/control.sock service create --name hello --constraint node.name==vm4 --mode global --image redis
e779nsffqir9i0bz8ubkoaf27

dchen@vm2:~$ sudo ./swarmctl -s /var/lib/docker/swarm/control.sock service ls
ID                         Name   Image  Instances
--                         ----   -----  ---------
e779nsffqir9i0bz8ubkoaf27  hello  redis  global

dchen@vm2:~$ sudo ./swarmctl -s /var/lib/docker/swarm/control.sock task ls
ID                         Service  Desired State  Last State                Node
--                         -------  -------------  ----------                ----
4gj50iw8s6kr3hu0vz17stss9  hello.0  RUNNING        RUNNING 11 seconds ago    vm4
e0whmekg7986tbxsv0c5h8dss  hello.0  RUNNING        ALLOCATED 12 seconds ago  vm2

[aaronlehmann]

ping @aluzzardi

[aluzzardi]

I think it's a low priority optimization.

Yes, if the constraint only matches a handful of nodes, we shouldn't create a ton of tasks for nothing

[aaronlehmann]

Think this is worth improving for 1.13?

I think at least we should improve the UX. The swarm service tasks output will be awful for global services restricted to a few nodes.

[aluzzardi]

I think so but as a P3. The functionality works, it's just ugly

[SSchwaiger]

For me, this is an important ability.

I have 2 worker nodes which are behind a F5 load balancer and a manager node. I want a Layer 7 router (i.e. nginx) to run on each of the nodes behind the LB. If I simply start a replicated service --replicas 2 it is not guaranteed that the tasks are assigned to different nodes.

Has anybody ideas on that?

[dperny]

@SSchwaiger the thing you want is #308, which is being worked on.

[anarcher]

Hi,

When a node changed/removed the node's label, swarm doesn't do anything for global mode services.

docker@swarm1:~$ service create --name redis --mode global --constraint="node.labels.type==web" redis

docker@swarm1:~$ docker service ps redis
ID                         NAME       IMAGE  NODE    DESIRED STATE  CURRENT STATE            ERROR
4vvfrhut88y6kkqyazjyuntz0  redis      redis  swarm1  Running        Allocated 6 seconds ago
bvojf85sqtu42yfdlc5adf59q   \_ redis  redis  swarm2  Running        Preparing 6 seconds ago
29ayszqv6ptk3z0i7vla31w8l   \_ redis  redis  swarm3  Running        Preparing 6 seconds ago

swarm2,swarm3 have a label type=web. I added the label (type=web) to swarm1 node.

docker@swarm1:~$ docker node update --label-add "type=web" swarm1
swarm1
docker@swarm1:~$ docker service ps redis
ID                         NAME       IMAGE  NODE    DESIRED STATE  CURRENT STATE            ERROR
4vvfrhut88y6kkqyazjyuntz0  redis      redis  swarm1  Running        Preparing 8 seconds ago
bvojf85sqtu42yfdlc5adf59q   \_ redis  redis  swarm2  Running        Running 30 seconds ago
29ayszqv6ptk3z0i7vla31w8l   \_ redis  redis  swarm3  Running        Running 41 seconds ago

It's works!
And then I remove swarm3 node's label

docker@swarm1:~$ docker node update --label-rm type swarm3
swarm3
docker@swarm1:~$ docker service ps redis
ID                         NAME       IMAGE  NODE    DESIRED STATE  CURRENT STATE               ERROR
4vvfrhut88y6kkqyazjyuntz0  redis      redis  swarm1  Running        Running 22 seconds ago
bvojf85sqtu42yfdlc5adf59q   \_ redis  redis  swarm2  Running        Running about a minute ago
29ayszqv6ptk3z0i7vla31w8l   \_ redis  redis  swarm3  Running        Running about a minute ago

docker@swarm1:~$ docker node inspect --pretty swarm3
ID:                     4twjz7na3248fpoi45nb97khb
Hostname:               swarm3
Joined at:              2016-09-07 13:55:05.340205463 +0000 utc
Status:
 State:                 Ready
 Availability:          Active
...

It seems that swarm doesn't do anything with global mode service and node label removing.
Is it right?

ps) Global services should only consider nodes that match constraints is important to me. In many my case, I would like to deploy with global mode (Sometimes I called it "static-deployment"). Node - service simple matching is enough. like https://github.com/spotify/helios)
ps) Docker 1.12 swarm mode is quite stable. I'm impressed. Thanks.

[dperny]

@anarcher There is actually a different issue open for this already, #1009.

EDIT: Also, I'm glad you're impressed with swarm mode!

[aaronlehmann]

Closing because #1570 was merged.

[Deepak-Vohra]

Global services are by definition _global_ and should not honor constraints. A service wouldn't be global if only running on some of the nodes in a Swarm.

[aaronlehmann]

We considered the name fill instead of global, as in "fill in all nodes that match the constraints". Ultimately, global was determined to be least confusing. global services do consider constraints, and this can be very useful. It's entirely optional to use constraints with a global services, so you can simply not set any constraints if you want the service to run on every single node.