Containers-roadmap: Add support for plugins

[ejholmes]

Now that Go 1.8 is released, and has support for plugins, it could be handy to expose some hooks in the ECS agent as a method for people to extend it, without forking.

As an example, in Empire, we'd like to eventually switch to the RunTask api for running "attached" containers, however, #462 is a blocker for that. If the ECS agent had support for a plugin, where we could manipulate the task before it was started, then we could add the OpenStdin and Tty options to CreateContainer before starting (e.g. if it had a docker label set). Without this, our only option is to fork the agent, which is something we'd like to avoid.

[ejholmes]

Thinking about this a bit more, it might be nice to also pluginize the engine.TaskEngine interface, which could make it easy for consumers to customize the default DockerTaskEngine, or build their own (e.g. a ContainerDTaskEngine).

[samuelkarp]

Hi @ejholmes, haven't heard from you in a while! :smile:

I think we'd be open to the idea of Go 1.8 plugins. We've been talking about extensibility internally for a while and haven't come up with anything that we're really happy with; Go 1.8 plugins probably are the closest to what we'd eventually want. However, there are a few limitations with the plugins as they exist today that are going to present usability issues; you can see them in this thread. Even so, they look more promising to me than other options right now. We'll need to be very clear about these limitations and how they impact plugin authors/distributors.

If we're going to allow plugins, we need to get really crisp on the interface(s) and interaction model(s) that we expose. Before writing any implementation code, we'll want to really know what you want out of an interface. If you're looking to contribute, the interface would be the right thing to propose first.

Speaking of interfaces, I'd probably oppose making engine.TaskEngine as a plugin interface. While engine.TaskEngine has served us reasonably well in production, it really has seemed to get in the way in our day-to-day development, troubleshooting, and maintenance. I'd prefer not to have that particular contract set in stone and used as a plugin interface.

[ejholmes]

haven't heard from you in a while

Ah yes, having twins will do that ;P

If we're going to allow plugins, we need to get really crisp on the interface(s) and interaction model(s) that we expose. Before writing any implementation code, we'll want to really know what you want out of an interface. If you're looking to contribute, the interface would be the right thing to propose first.

Yep, totally agree. I think the most logical separation would be separating out the ECS "plumbing" (state management, interacting with ACS, etc) and actually spinning up and managing containers. Something akin to runtime's in the Docker engine.

I think (with a very shallow understanding of the internals of the ECS agent), I would expect an interface something like:

type Runtime interface {
        StartTask(*Task, chan *StateChange) error
        StopTask(id string) error
        Capabilities() []string
}

While my immediate goal is to just be able to extend the Docker runtime slightly, a long term goal for us would be to reduce/remove our dependency on Docker and instead run containers in a more "lightweight" manner using something like containerd, runc, or systemd. ECS does have some slight coupling to Docker (dockerLabels, dockerSecurityOpts, etc), but for the most part, concepts are abstracted away in TaskDefinitions, so I think this would be possible.

The official Docker runtime could still be statically compiled in, so the interface could be fleshed out before actually exposing it with the ability to load a Go plugin.

[samuelkarp]

Ah yes, having twins will do that ;P

Congratulations!

Yep, totally agree. I think the most logical separation would be separating out the ECS "plumbing" (state management, interacting with ACS, etc) and actually spinning up and managing containers. Something akin to runtime's in the Docker engine.

[M]y immediate goal is to just be able to extend the Docker runtime slightly [...]

I think that might be closer to just allowing replaceable runtimes than a straight-up plugin model. I think my biggest concern with something like that is that is that I think your immediate goal is potentially more useful to a larger number of people than completely abstracting the runtime away; I think more people are interested in providing small extensions either to how ECS is invoking Docker or around the lifecycle of containers/tasks than in implementing a runtime.

[A] long term goal for us would be to reduce/remove our dependency on Docker and instead run containers in a more "lightweight" manner using something like containerd, runc, or systemd.

I've started looking at containerd as a potential alternative to (or supplement to) Docker as used by ECS. I'll link to some code here when I have that ready.

[ejholmes]

I've started looking at containerd as a potential alternative to (or supplement to) Docker as used by ECS. I'll link to some code here when I have that ready.

❤️

[samuelkarp]

@ejholmes You can follow along with my prototyping for containerd here.

[iyesin]

Any updates here?

[samuelkarp]

@iyesin Are you looking for an update on plugins or containerd? If plugins, what kinds of things would you want to do with them? Are you looking to override specific settings in Docker's Config or HostConfig API structures, hook into lifecycle events, or something else?

[StyleT]

@samuelkarp hook into lifecycle events please :) For example we use Hashicorp Vault for secrets and it would be beneficial to write a plugin to pulls them from the Vault before container startup and inject as Env variables or container secrets.

[hridyeshpant]

@StyleT you can deploy vault functionality as sidecar container in ECS task def and share the mounted volume to the application container. This will make sure vault container will start first and fetch the secrete and put that secret in share volume which application container can read at the startup.

[iyesin]

Greetings, @samuelkarp.

Actually I'm looking for a way to hook-up to ecs-agent updates. I'm OK with anything that will allow me to emit event (http request, command) outside ECS. So plugin, hooks… something that will allow to signal about task/service state change via push model.