Swarmkit: Orchestrator does not handle dynamic updates in tasks scaling

[abronan]

Considering the following scenario using docker integration:

$ docker service scale test=1000

# Directly after
$ docker service scale test=50

The orchestrator goes through scheduling the 1000 tasks before scaling down to 50.

The orchestrator should dynamically handle that case and only converge on the latest input value (in this case 50). This is risky to assume that any input value is a valid one and we should allow a user to correct a potential mistake, otherwise the cluster can be made slow or unresponsive by providing very high and unreasonable values.

See docker/docker#24027 for more context.

/cc @stevvooe @nishanttotla (couldn't find an issue tracking this but let me know if this looks like a dupllicate)

[stevvooe]

Isn't this handled in the orchestrator?

[aaronlehmann]

I'm not sure whether we should change the behavior. When you scale up, it creates a certain number of tasks and puts them in the scheduling pipeline. The current design is quite elegant because the stages of the pipeline are not aware of desired state, so tasks advance through the pipeline even if you scale down the service while this is happening. I don't think the tasks actually run once they get assigned if their desired state is set to "shutdown" (if that's not true, we might revisit it).

So in unusual cases when something is scaled up and then back down immediately, we have to do some unnecessary allocator/scheduler work, but we have the advantage of a very simple pipeline and avoid a class of bugs where tasks could get stuck in the pipeline.

On Jul 11, 2016, at 11:25, Alexandre Beslic [email protected] wrote:

Considering the following scenario using docker integration:

$ docker service scale test=1000

Directly after

$ docker service scale test=50
The dispatcher goes through scheduling the 1000 tasks before scaling down to 50.

The dispatcher should dynamically handle that case and only converge on the latest input value (in this case 50). This is risky to assume that any input value is a valid one and we should allow a user to correct a potential mistake, otherwise the cluster can be made slow or unresponsive by providing very high and unreasonable values.

See docker/docker#24027 for more context.

/cc @stevvooe @nishanttotla (couldn't find an issue tracking this but let me know if this looks like a dupllicate)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[abronan]

@stevvooe right, updated description and label.

[abronan]

@aaronlehmann While I agree with you that the design gets the job done and reduces the complexity of the scheduling pipeline, I'm thinking about the worst case. What happens if you type:

docker service scale redis=100000000000000000000000000000000000

You can make your cluster unresponsive to further requests/maintenance operations for several minutes/hours with one single command. Not only that but you can also disrupt other services.

[abronan]

/cc @aluzzardi Would like to hear your opinion on this one.

[aluzzardi]

@abronan @aaronlehmann We do something similar for rolling updates: If a second update for the same service comes in, we cancel out the previous one.

There are some components that cannot run in parallel (e.g. scheduler), however, for some others (orchestrator) we could probably do something similar as an optimization later on: start every service reconciliation in its own go-routine, and if a second update comes from the first service, kill the go-routine before spawning a new one.

I think this would help in other situations too: right now, docker service scale redis=100000000000000000000000000000000000 would block the orchestrator for quite a while. With the go-routine approach, since we are using batch writes, multiple services would nicely contend together for a raft lock, resulting in smoother parallel service management.

[dongluochen]

@aluzzardi The problem is not in orchestrator, but the way dispatcher handles task updates. In the following example, orchestrator updates service tasks right away. Dispatcher sequentializes task state change, and pushes it out gradually. It's a 3 managers (drain) and 2 workers setup. Dispatcher rate limiting should be in effect.

==> start a service
==> update replicas to 500
ubuntu@ip-172-19-241-145:~$ docker service ls
ID            NAME        REPLICAS  IMAGE        COMMAND
2f65uo172gga  stresstest  49/500    redis:3.0.7

==> task assigned to a node (dockerd log from leader)
Jul 22 18:04:51 ip-172-19-241-145 dockerd[32031]: time="2016-07-22T18:04:51.289861541Z" level=debug msg="Assigning to node 4fm1frh6nof1oeux2zh4x2byx" task.id=6dia23d6xezj5eay9pvyqtajt

==> after 2 minute set replicas to 5
ubuntu@ip-172-19-241-145:~$ docker service update --replicas 5 stresstest
stresstest
ubuntu@ip-172-19-241-145:~$ docker service ls
ID            NAME        REPLICAS  IMAGE        COMMAND
2f65uo172gga  stresstest  59/5      redis:3.0.7
ubuntu@ip-172-19-241-145:~$ docker service tasks stresstest
ID                         NAME            SERVICE     IMAGE        LAST STATE             DESIRED STATE  NODE
evigtzpzuyqb45h1palxsrk8z  stresstest.30   stresstest  redis:3.0.7  Running 2 minutes ago  Running        manager
021x9wpr3iumk8k81wh252qty  stresstest.125  stresstest  redis:3.0.7  Running 2 minutes ago  Running        manager
00doe6g8feo9f2kpsz4166u1v  stresstest.337  stresstest  redis:3.0.7  Running 18 hours ago   Running        ip-172-19-241-144
00n50ixfwk6qs1ts4rz4qzwzr  stresstest.458  stresstest  redis:3.0.7  Running 17 hours ago   Running        manager
03u8801db6410skawj6v3kar1  stresstest.880  stresstest  redis:3.0.7  Running 17 hours ago   Running        ip-172-19-241-144
ubuntu@ip-172-19-241-145:~$ date
Fri Jul 22 18:07:11 UTC 2016

==> 7 minutes later, the task changed to RUNNING and push to node
Jul 22 18:14:06 ip-172-19-241-145 dockerd[32031]: time="2016-07-22T18:14:06.066039974Z" level=debug msg="task status updated" method="(*Dispatcher).processTaskUpdates" module=dispatcher state.transition="ASSIGNED->RUNNING" task.id=6dia23d6xezj5eay9pvyqtajt

==> another 2 minutes later dispatcher set task to SHUTDOWN
Jul 22 18:16:35 ip-172-19-241-145 dockerd[32031]: time="2016-07-22T18:16:35.447105547Z" level=debug msg="task status updated" method="(*Dispatcher).processTaskUpdates" module=dispatcher state.transition="RUNNING->SHUTDOWN" task.id=6dia23d6xezj5eay9pvyqtajt

[aluzzardi]

@stevvooe Not sure how we handle overlapping updates on the agent side?

On the orchestrator for rolling updates, for instance, a new update for the same service cancels the previous update.

If we receive new state from the dispatcher, could we cancel the previous local reconcilation?

[stevvooe]

If we receive new state from the dispatcher, could we cancel the previous local reconcilation?

This is fairly tricky and will probably lead to instability. It may be worthwhile to analyze the reconciliation performance and see if we reduce the likelihood of overlap.

It's probably better to do everything we can to not sending confusing updates to the agent (I can give you the control systems speel on why this is a problem due to system response time and state hysteresis). If we can coalesce and rate limit the updates, it will make the agent much more stable in this regard.

[aluzzardi]

@aaronlehmann Would it make sense to handle each service update in its own goroutine?

[aaronlehmann]

In other words, making Update blocking? Not sure I see the advantage.

On Aug 4, 2016, at 17:25, Andrea Luzzardi [email protected] wrote:

@aaronlehmann Would it make sense to handle each service update in its own goroutine?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[aluzzardi]

@aaronlehmann Sorry, I realize how service update and the update supervisor can be confusing.

Forgetting entirely about rolling updates and update supervisor, would it make sense to have the orchestrator spawn each reconciliation in its own goroutine?

1) docker service scale foo=10000 -> spawn a goroutine
2) docker service scale foo=10 -> cancel previous goroutine, spawn a new one

[aluzzardi]

I think the orchestrator, unlike the scheduler, can run in parallel.

The above would mean that large reconcilations (e.g. scaling to a large number or something like that) wouldn't block other services.

It also means they are cancellable.

It also might make the code simpler.

[aluzzardi]

I need to look at the code to figure out how it'd make things simpler, but I remember we were doing a few dances around for some reason that we could avoid.

Also, things such as the UpdateSupervisor wouldn't need to exist anymore: reconciling state (whether it's changing the number of instances or doing a rolling update) could be entirely blocking since the orchestrator is already running in parallel.