Swarmkit: API: Events stream

[aluzzardi]

The API should provide a way to watch for changes.

There are various solutions to do that:

• A single event bus, similar to the engine (Events() RPC)
• A per-type watch (e.g. WatchTasks(), WatchServices(), ...)
• Blocking get with sequencer. (e.g. GetService() then GetService(IfModified: lastServiceVersion)

/cc @stevvooe @vieux

[aluzzardi]

/cc @stevvooe

I'm putting this back into the Icebox. We should do this, not sure the time horizon yet (probably not 1.13).

WDYT?

[thaJeztah]

I still think it's a very useful feature to have. Quite some projects use events to (re) configure services and to hook into docker. Not being able to use events related to Swarm mode sounds like quite a limitation.

I'm a bit worried that those projects will now implement all kind of workarounds, and by the time Swarm events _are_ implemented, it's too late.

[cleishm]

For my use case, I would like to know when an operator initiates a change to the scale of a particular service (and the change is confirmed within the swarm cluster).

The use case is a replicated database, which needs to know how many instances of the database process should be reachable in the swarm so that it can determine how many replicas constitutes a majority.

For example, if the service is set to scale=3, then a minimum of 2 replicas would need to agree on a distributed transaction for that tx to be applied. If the operator then changes the scale to 5, then 2 replicas is no longer a sufficient majority - it is now 3. I'd like a documented way to know when such changes occur.

[allingeek]

I'm not sure how I can justify putting an orchestration abstraction into production without proper monitoring. Event streams are critical for operational insight. These events power everything from workflow automation to operator incident response. To punt on a complete event stream implementation for at least two releases feels like serious blow. People are going to fill the gaps (which is difficult given the new hits to integration points) or they're going to use a different orchestration system.

Events we need:

• service create
• service update
• service scale
• service correction (correcting drift from desired state)
• service delete

These events would augment the existing container/network/volume events we have today and hint at orchestration intent rather than contextless change reporting.

[BretFisher]

Also swarm node events. In several current deployment projects, we're trying to come up with reliable ways to handle node activities as well, esp when mixed with AWS ASG's:

• node promote/demote
• node add/remove
• node health change
• manager add/remove
• manager/raft health change (sneaking this in: including "yellow" state if even count of managers, which assumes we have manager health checks for that)

[alexellis]

I would also like to see this and I think it may be important for jobs, i.e. imperative one-shot tasks.

[thaJeztah]

@alexellis also see https://github.com/docker/docker/issues/23880 for batch/jobs

[alexellis]

Thanks

[Yaytay]

One of the most important things for me is to know node health - in particular to know when a node returns after being down.
Currently I monitor all events on each node, but there is nothing at all to inform me when a downed node (that is already a swam worker) returns.

[aluzzardi]

Folks - @dongluochen is currently working on this, we are hoping to ship this in the next release

[stevvooe]

@aluzzardi Is there a protobuf proposal?

[dongluochen]

@stevvooe I'll create a protobuf proposal.

[aluzzardi]

I don't think there'll be changes in the protos. Events can be implemented in Docker as long as swarmkit exposes a Watch gRPC interface which @aaronlehmann has a prototype for

[dongluochen]

Here is current design for Swarm events. Feedbacks are welcomed.

Swarm leader subscribes to gRPC flow from Swarmkit which consists of raft memory store changes. @aaronl is working on the Watch interface. The leader compares each raft change with recorded state and emits Swarm events when necessary. Events are cached on leader up to amount limit or time limit (e.g., store events for past 24 hours). On the first iteration leader would not preserve events before it becomes leader, i.e., Swarm event history are not kept on leader switch. It requires raft event replay to re-gerenate history Swarm events, or persistent storage for Swarm events. When client detects connection failure, it should re-estabilish event stream.

Event category

The first iteration supports the following events. Task events are not supported as they are transient steps of service orchestration. I think it should be reported at service level. How to report service convergence/failure is a topic to explore.

cluster update (cluster spec update)

node join/remove (should report node pending)
node promote/demote
node status change (Ready, Down)
node availability change (Active, Pause, Drain)
node leadership change  

service create
service update
service scale (this could be just update)
service rollback
service remove

network create
network remove

secret create
secret remove

CLI

Docker Swarm events follows docker events CLI design. The CLI could be docker swarm events. Swarm event stream is separated from Docker events on the local daemon. I don't find valid use cases to mix local daemon events with Swarm events in one stream.

$ docker swarm events --help

Usage:  docker swarm events [OPTIONS]

Get real time events from the cluster

Options:
  -f, --filter filter   Filter output based on conditions provided
      --format string   Format the output using the given Go template
      --help            Print usage
      --since string    Show all events created since timestamp
      --until string    Stream events until this timestamp
      --node-only       Show only node events
      --service-only    Show only service events
      --network-only    Show only network events

The filtering flag -f or --filter allows filtering on node/service/network. It follows docker events processing of filters. It will support filters:

service [service=<name or id>]
node [node=<name or id>]
network [network=<name or id>]

The event stream is for program consumption as well as human inspect. It follows docker events format, to be refined.

$ docker swarm events
2017-01-05T00:35:41.241772953+08:00 service create (image=nginx:latest, mode=replicated, replicas=3, name=nginx-server)
2017-01-05T00:35:58.859401177+08:00 network create (driver=overlay, name=tnet)

API

GET /swarm/events

Example response, to be refined.

HTTP/1.1 200 OK
Content-Type: application/json
Server: Docker/1.13.1 (linux)
Date: Fri, 29 Apr 2016 15:18:06 GMT
Transfer-Encoding: chunked

{
  "status": "",
  "id": "",
  "Type": "service",
  "Action": "create",
  "Actor": {
    "ID": "nginx-server",
    "Attributes": {
      "image": "nginx:latest",
      "mode": "replicated",
      "replicas": "4",
    }
  },
  "time": 1461943101,
}

[stevvooe]

@dongluochen Looks reasonable.

Could we please avoid the cruft in the swarm events API? This means things like using proper RFC3339 timestamps and not sending deprecated fields (status and id).

[aluzzardi]

/cc @ehazlett @dhiltgen

I'm heavily against using a different API endpoint or CLI command. There are very painful efforts going on to convergence Docker & Swarm APIs together and we should avoid anything that adds yet another API difference.

Regarding the events per se, I think we should do something as simple as possible:

• CRUD events for all objects
• State changes for objects that have a State

cluster update (cluster spec update)
Sounds good, there's only update.
I think we could leave cluster events aside for now - we don't expose the ClusterSpec to the user.

node join/remove (should report node pending)
node promote/demote
node status change (Ready, Down)
node availability change (Active, Pause, Drain)
node leadership change

I think those fit CRUD+state.

Since promote/demote and availability change (Active, Pause, Drain) are NodeSpec changes, these could be represented as a simple node update.

For node status change (Ready, Down), those can in fact be emitted as state

The only one truly aside are leadership changes - those are not in the store.

service create
service update
service scale (this could be just update)
service rollback
service remove

I think those should just be CRUD as well. rollback and scale are just updates.

[aaronlehmann]

service update

Would this show updates to the spec, or any update to the service object?

Since promote/demote and availability change (Active, Pause, Drain) are NodeSpec changes, these could be represented as a simple node update.

Sounds reasonable, but I wonder if these will be hard to deal with. Will the events contain the old and new spec? If not, it seems very hard to detect things like a node becoming drained. I think that's the kind of thing people want events for.

[aluzzardi]

From my experience as en event user, what I have to do without events is:

for {
  c := GetContainer(foo)
  if changed(c) {
    doStuff(c)
  }
  time.Sleep(n)
}

With events I can get notified when something happens to an object so I don't have to sleep and poll in my code. @dhiltgen @wsong ?

[dongluochen]

Looking at Docker events API, I think it has tried to describe what happens, not just a simple notification. For example, Docker containers report the following events: attach, commit, copy, create, destroy, detach, die, exec_create, exec_detach, exec_start, export, health_status, kill, oom, pause, rename, resize, restart, start, stop, top, unpause, update. Not providing a meaningful event would be a change from docker events, and may reduce its usage.

Swarm events can reuse current Docker events CLI like docker events --swarm. API can be GET /events?source=swarm. Some options and filters like node-only are meaningful only with --swarm. I think they should be clear enough for users.

[thaJeztah]

Slightly orthogonal, but we should have a proper look at the docker events api, because there are some issues in the way event / object attributes are returned (for example, labels and actual properties are combined in a single map; not sure what lead to that implementation, but it's, erm, 💩 ).

[stevvooe]

@thaJeztah @dongluochen We should also look to other examples of event API. Typically, it is bad practice to embed event data within the event itself, as you may operate on out of date information by the time you handle it. It would be best to include only the information about the event (who, what, where, when).

[allingeek]

@stevvooe Right. While the practice of operating on data embedded within an event directly is an anti-pattern omitting that data presumes the use-case. Many event stream consumers simply want to log and move on. The best A good practice is to emit the data (maybe occasionally even in duplicate or out of order D: ) and let the consumer decide how to use it.

[stevvooe]

Many event stream consumers simply want to log and move on. The best A good practice is to emit the data (maybe occasionally even in duplicate or out of order D: ) and let the consumer decide how to use it.

Sure. In that case, you log "service foo with id abc created". If you need more, you lookup the current state of the object.

Consumers that don't require the whole state shouldn't pay the cost of including the extra data.

Perhaps, carrying state should be an option.

[qmhu]

@dongluochen which swarm version will have this feature , can watch the node change event is very necessary when management many swarm clusters.

[bjornmagnusson]

@dongluochen Is there any ETA for this feature?

[dongluochen]

Current plan is to release event API with Docker-17.05 or 17.06. It has upstream dependence on store Watch API.

[aaronlehmann]

(Note the store watch API is #2034)

[kaikuchn]

@dongluochen you said that task events will not be supported for the first iteration, instead this will be reported at the service level. Does that mean that I will be able to know during a rolling update when a task was successfully started?

I.e, when I do a rolling update with some delay, could I execute some script - by watching the event stream - right after one task was updated?

[dongluochen]

@kaikuchn The event support is merged https://github.com/moby/moby/pull/32421. We don't support task monitoring in this first iteration. Instead you can get events on service updatestate.

# update image of a service
2017-04-06T18:11:19.122732546Z service update 9vvofszhb6iv4k3tmphras96u (image.new=nginx:1.10.3@sha256:6202beb06ea61f44179e02ca965e8e13b961d12640101fca213efbfd145d7575, image.old=nginx:latest@sha256:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582, name=nginx)
2017-04-06T18:11:19.126619069Z service update 9vvofszhb6iv4k3tmphras96u (name=nginx, updatestate.new=updating, updatestate.old=nil)
2017-04-06T18:11:41.552581741Z service update 9vvofszhb6iv4k3tmphras96u (name=nginx, updatestate.new=completed, updatestate.old=updating)

User might still want to get update on each task. We will investigate. There will be a lot of task events overwhelming other events. I'm thinking make it a daemon option so users can enable it if they want to.

[datacarl]

We don't support task monitoring in this first iteration. Instead you can get events on service updatestate

@dongluochen I was hoping to use the events to update a load balancer's configuration of available servers. E.g. if a container is rescheduled for any reason I'd like to get an event for it so I can update the LB with the new IP/domain name for the container. Will that be possible after this update is shipped?

[alexellis]

I would say task-level events are important for any kind of dynamic or serverless platform too. +1 for task-level events.

[dongluochen]

The current plan is to add an option to enable task level events. There are several issues related to functionality and scalability.

1. Swarm events are merged with Docker events with a limited buffer with current setting of 256. This may need to be configurable if task events are enabled. --since, --until is done thru linear search, a large buffer may increase response time.
2. Task state is a combination of desired state, and reported status. Each can be any of the current 13 states https://github.com/docker/swarmkit/blob/master/api/types.proto#L415. While only some combinations are reachable, the number of changes from a task is still large.
3. Current implementation is to map each change in cluster store to an event. Some task events are implementation details which may change over time. It may not be easy to maintain backward compatibility.

Jan 24 19:42:29 ip-172-19-241-146 dockerd[7258]: time="2017-01-24T19:42:29.070046953Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="PREPARING->READY" task.id=padqpqv4olv8khsp9g3xhwxbk

1. The event may not provide enough info for external system to react. A common pattern is to trigger reaction. With the number of task events, it may trigger a lot of requests to the system. (One bad scenario could be an amplified feedback loop which could halt your system.)

[kaikuchn]

For my use case there's two task states I'm interested in:

1. shutdown - I'd like to know when a task has disappeared so I can unregister it inside another service since it cannot handle requests any longer
2. running - I'd like to know when a task is up and ready to accept work, so I can register it

A little background on my use case:
I'm running CitusDB, which is in essence a distributed Postgres. It has one coordinator and several worker nodes. I need to (de)register CitusDB workers (each task is a worker) with my CitusDB Coordinator (another service) since it looks them up in an internal table via domain name and port. I therefore cannot use the routing mesh and have to handle that myself. There is overlap in my workers responsibilities (shards) such that a few can be down without any queries failing. Therefore I can do rolling updates if I know when a task is taken down and another is up.

[dongluochen]

@kaikuchn Thanks for providing your use case. Task level events should meet your requirement. It just carries a lot of unrelated events.

shutdown - I'd like to know when a task has disappeared so I can unregister it inside another service since it cannot handle requests any longer

A task status beyond RUNNING should be treated as service instance termination https://github.com/docker/swarmkit/blob/master/api/types.proto#L428.

[aaronlehmann]

Implemented in #2034.

[datacarl]

Any docs on how to use this now?

[thaJeztah]

@datacarl this is just the changes in SwarmKit; for docker, these events will show up in the events-stream in docker events