Swarmkit: [proposal] new node management

[vieux]

After a couple of meetings with solomon, @al and I would like to propose some
ideas regarding the node management.

It's a big change compared to what we have today, but the UX is pretty neat:

• agent -> worker
• a node is a docker engine
• a node is either a worker or a worker+manager
• init create a manager and a worker
• join add a new worker to the cluster
• then from a manager, you can change the role of a worker to be promoted manager
• if you want a "manager only" set the role of a worker to manger and set it's availability to drain, so no task will be scheduled on it.

See this example workflow, we have 2 machines, node-1 and node-2:

# create a cluster
node-1$ docker swarm init [--accept-policy=always]
9ta6bzhbc72e

node-1$ docker swarm info
Node: 1
Manager: 1
Leader: 9ta6bzhbc72e
Raft:
        HeartbeatTick   5s
        ElectionTick    15s
        MaxSizePerMsg   256
        MaxInflightMsgs 16


# list nodes
node-1$ docker node list
ID            Name       IP             Status   Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY    ACTIVE        MANAGER


# switch to “manager only”
node-1$ docker node update --state=drain node-1 #or --self

node-1$ docker node list
ID            Name       IP             Status   Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY    DRAIN         MANAGER


# join a 2nd engine
node-2$ docker swarm join node-1:2377
dsa7321nsad9


# accept node (only if --accept-policy=always wasn’t supplied)
node-1$ docker node list
ID            Name       IP             Status  Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY   ACTIVE        MANAGER
dsa7321nsad9  node-2     10.0.0.2       READY   PENDING       WORKER

node-1$ docker node update --state=active node-2

node-1$ docker node list
ID            Name       IP             Status  Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY   ACTIVE        MANAGER
dsa7321nsad9  node-2     10.0.0.2       READY   ACTIVE        WORKER


# promote a node to manager
node-1$ docker node node-2 update --role=manager

node-2$ docker node list
ID            Name       IP             Status  Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY   DRAIN         MANAGER
dsa7321nsad9  node-2     10.0.0.2       READY   ACTIVE        MANAGER

node-2$ docker swarm info
Nodes: 2
Managers: 2
Leader: 9ta6bzhbc72e
Raft:
        HeartbeatTick   5s
        ElectionTick    15s
        MaxSizePerMsg   256
        MaxInflightMsgs 16


# demote a manager
node-2$ docker node-2 update --role=worker #also works from node-1

node-1$ docker node ls
ID            Name       IP             Status  Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY   DRAIN         MANAGER
dsa7321nsad9  node-2     10.0.0.2       READY   ACTIVE        WORKER


# remove a node
node-1$ docker node rm dsa7321nsad9

node-1$ docker node list
ID            Name       IP             Status  Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY   DRAIN         MANAGER

As you can see, a node doesn't ask to me promoted as a manager and needs to be accepted from a manager, it's the other way around, you promote a node from the manager.
Basically the manager change the role of a node in raft, the agent will pick up the change and start it's manager process.

@diogomonica do it think this workflow is doable regarding the certs ?
@stevvooe @aaronlehmann how do you feel about the agent spawning the manager process when it detect a change in raft ?
@tonistiigi @dnephin how do you feel about the UX ?

[tonistiigi]

I believe when you add a node it should also be in the manager role by default. People just starting to try it out don't know the exact definitions of the components and having both is the safest way to run small clusters. What if MANAGER and WORKER both remain roles, when you do cluster join you can specify if you want manger-only or worker-only and then in docker node update this can be overridden or if no properties are set it would just enable the roles that the node requested.

From the UX side we are expecting user to switch between engines and this is very painful with the current DOCKER_HOST= method. So maybe we should think if there is a way to store the node/connection-strings mapping on the client side and automatically find the correct engine to talk to. This would also mean that the accept process could be handled by the client automatically.

[aaronlehmann]

how do you feel about the agent spawning the manager process when it detect a change in raft ?

Is there a good reason for the agent and manager to be separate processes if they're this tightly integrated? It would be a little easier not to have to send a bunch of state to a separate process, and monitor it to make sure it keeps running, etc.

I think in this design the flow for being promoted to a manager would work something like this:

• swarmctl requests a change to raft showing the node should become a manager.
• This change gets committed to raft.
• The dispatcher observes the change to the node in raft, and tells the associated agent to upgrade itself to a manager.
• The agent requests a ou=manager certificate. Since the node is marked in raft as having manager privileges, the CA should grant this certificate. It installs the certificate separately from the agent cert, and only uses it for manager responsibilities.
• The agent starts up a manager subsystem, which attempts to join the raft cluster, using parameters given to it by the dispatcher in the upgrade message. If it succeeds, a configuration change gets committed to raft adding it as a member of the cluster. Otherwise, it keeps trying with some kind of backoff.

As far as I can tell right now, it seems workable. But it's quite a big design change late in the process, and I'm not sure I'm comfortable with that.

I believe when you add a node it should also be in the manager role by default.

I disagree. We need to encourage people to be very selective about which nodes are allowed to be managers. Managers will get manager certificates, which are very sensitive, and we shouldn't issue them by default. Also, managers will join the raft cluster, and if the default is for every node to be a manager, it would be easy to end up with a huge raft cluster, when there is no reason for it to have more than 7 nodes (and it does not scale well to larger sizes).

[dnephin]

For the most part I like this change. I'm not sure how "the agent will pick up the change and start it's manager process" is going to work with the current integration. Does it just spawn another process? I think this would be much cleaner when they run as containers.

I believe when you add a node it should also be in the manager role by default.

I think there is a potentially bad UX here, where join and init do the same thing except for one argument. If we're going to have them do the same thing, we should make just a single command.

I like that in this proposal they do different things.

[aaronlehmann]

if you want a "manager only" set the role of a worker to manger and set it's availability to drain, so no task will be scheduled on it.

I think there needs to be some one-step way of setting up a node to be only a manager. In the time between creating a new node and draining it, containers could be scheduled there. This could be unacceptable to production users who insist that their managers do not run containers.

I still think it's better to keep agents and managers as separate concepts instead of conflating them.

[vieux]

@dnephin init is to create a new cluster, creating the certs etc... / join is to join another cluster I don't think they are similar.

@aaronlehmann we were thinking of maybe a docker cluster init --state=pause so handle this case.
@aaronlehmann we came up with this, because solomon would like to have a single docker node list with every node, it means every node should have a common base (for example you also want to see the engine version on managers), that's why we were thinking that maybe agent could be this common base.

[abronan]

Was going to comment exactly like @aaronlehmann but I saw his comment first luckily.

Also small nit on the UI, we should have the information of who is the leader somehow but we can't add a new column or this will be confusing with Workers being listed as well so maybe:

# promote a node to manager
node-1$ docker node node-2 update --role=manager

node-2$ docker node list
ID            Name       IP             Status  Availability  Role
9ta6bzhbc72e  node-1     10.0.0.1       READY   ACTIVE        MANAGER (*)
dsa7321nsad9  node-2     10.0.0.2       READY   ACTIVE        MANAGER

I think the UI is great but I'm afraid this is going to confuse many users on the importance of the Manager compared to a Worker and we'll end up with users treating a Manager like a Worker (which can be dangerous). Those are managed differently, have a different tolerance to failures and overall have different ways to recover from those failures. Users might end up restarting the nodes or doing rolling updates of the daemon (Workers) considering Managers are just regular nodes which is far from being safe and can end up in a pretty disastrous state in the end.

[stevvooe]

I took some time to write up a startup workflow that feels a bit more natural.
The big difference is that we allow a node to be a manager and agent. The two
datasets are separated, since they don't share functionality. We still have a
single command to get to a working cluster very easily with the advantage that
agent and manager concerns are appropriately separated.

Cluster Bootstrap

The first step is configure a local node. We support API methods that can
change the configuration for cluster membership. These could also be set by
flags.

The main command for configuring the local node is the docker cluster init or
docker cluster join command. It operates on node-local engine configuration
through the local API.

Any state configured by docker cluster init or docker cluster join (or
corresponding API calls) can be cleared by docker cluster leave.

For a summary, we have the following:

docker cluster init [--role=(manager|agent), ...]
docker cluster join [--role=(manager|agent), ...] [<manager>, ...]
docker cluster leave 

Command can take multiple --role arguments. Default for --role is
--role=agent --role=manager. Setting any --role clears defaults. Depending
on the role, the manager addresses will be used to join a cluster.

To bootstrap a cluster, call the following:

$ docker cluster init

This can take initial cluster configuration that cannot be changed (maybe). The
above is shorthand for:

$ docker cluster init --role=manager --role=agent

Such a node runs a manager where tasks will be scheduled and dispatched to the
single agent.

Let's say the above was run on cluster0. From another node, We can join as an
agent/manager from another node with the following command:

$ docker cluster join cluster0:4242

Now we have a cluster with two nodes that will schedule tasks on each other.
It will join the quorum for cluster0 and provide another agent to dispatch
tasks. Both nodes are manager and agent. At this point we have the following:

# From a manager node
$ docker node ls
ID            Name       IP             Status  Availability 
9ta6bzhbc72e  cluster0   10.0.0.1       READY   ACTIVE
dsa7321nsad9  cluster1   10.0.0.2       READY   ACTIVE

Using the manager command, we see the following:

# From a manager node
$ docker manager ls
ID            Name       IP             Status 
9ta6bzhbc72e  cluster0   10.0.0.1       LEADER
dsa7321nsad9  cluster1   10.0.0.2       FOLLOWER

This isn't very robust. Let's add a third, manager only node:

$ docker cluster join --role=manager cluster0:4242

Our docker node ls output is the same, but we have following managers:

# From a manager node
$ docker manager ls
ID            Name       IP             Status 
9ta6bzhbc72e  cluster0   10.0.0.1       LEADER
dsa7321nsad9  cluster1   10.0.0.2       FOLLOWER
deadbeaf0123  cluster2   10.0.0.3       FOLLOWER

Now, we have a fairly robust quorum. Let's balance this out by adding another
separate agent node:

$ docker cluster join --role=agent cluster0:4242

And, now we have three agents:

$ docker node ls
ID            Name       IP             Status  Availability 
9ta6bzhbc72e  cluster0   10.0.0.1       READY   ACTIVE
dsa7321nsad9  cluster1   10.0.0.2       READY   ACTIVE
beefdad13375  cluster3   10.0.0.4       READY   ACTIVE

From any node, we can directly leave the cluster using the docker cluster leave command. It resets the cluster state for a node and returns it to a
regular docker engine. This command _must_ be run directly on the target node.

Having to run commands on the target node is obnoxious. We define the following
to manage the nodes through the cluster API:

docker node info
docker node update
docker node rm 
docker manager info
docker manager update
docker manager rm

The above commands interact with the cluster API and allow one to evict nodes
and managers as they see fit.

[aluzzardi]

Just for the sake of argument, here's what a NodeSpec of a joint agent/manager could look like:

message NodeSpec {
    Annotations annotations = 1 [(gogoproto.nullable) = false];

    enum Availability {
        option (gogoproto.goproto_enum_prefix) = false;

        // Active nodes.
        ACTIVE = 0 [(gogoproto.enumvalue_customname) = "NodeAvailabilityActive"];

        // Paused nodes won't be considered by the scheduler, preventing any
        // further task to run on them.
        PAUSE = 1 [(gogoproto.enumvalue_customname) = "NodeAvailabilityPause"];

        // Drained nodes are paused and any task already running on them will
        // be evicted.
        DRAIN = 2 [(gogoproto.enumvalue_customname) = "NodeAvailabilityDrain"];
    }

    enum Acceptance {
        option (gogoproto.goproto_enum_prefix) = false;

        PENDING = 0 [(gogoproto.enumvalue_customname) = "NodeAcceptancePending"];
        ACCEPT = 1 [(gogoproto.enumvalue_customname) = "NodeAcceptanceAccept"];
        REJECT = 2 [(gogoproto.enumvalue_customname) = "NodeAcceptanceReject"];
    }

    enum Role {
        option (gogoproto.goproto_enum_prefix) = false;

        WORKER = 0 [(gogoproto.enumvalue_customname) = "NodeRoleWorker"];
        MANAGER = 1 [(gogoproto.enumvalue_customname) = "NodeRoleManager"];
    }

    Role role = 2;

    Acceptance acceptance = 3;

    // Availability allows a user to control the current scheduling status of a
    // node.
    Availability availability = 4;
}

I do believe though that given the time constraints, @stevvooe's proposal might be the most realistic.

[dnephin]

The proposal from @stevvooe sounds like what we had originally in the docker cli 1.12 proposal, and what we started to implement.