Swarmkit: Support for executing into a task

[aluzzardi]

Similar to docker exec, swarmkit should provide a way to execute commands inside a task, mostly for debugging purposes.

[stevvooe]

I'm not sure I understand the use case. I could see the point of attach but exec'ing other processes seems ripe for abuse. Is it just cause docker exec exists?

[hairyhenderson]

@stevvooe for me, the use case is exactly the same as docker exec - for debugging and troubleshooting. It can be difficult to access a swarm node where a specific task is running to do a local docker exec, and downright impossible if the container only runs for a few seconds and then dies, only to be rescheduled to a different worker. Imagine doing this on a swarm with dozens or hundreds of workers...

IMO it'd be very useful to be able to docker service exec myservice.1 foo or something similar.

[stevvooe]

@hairyhenderson You can already docker exec into a task container, even after exiting. The only challenge is getting to that node. Short-lived containers will be around until the manager instructs nodes to clean them up (i think we keep five for service instance).

I am not sure if this works, but it would be good if we could get to something like this:

ssh $(docker service ps <task-id> | awk '{print $4}' | grep -v NODE) docker exec <task-container-id> 

Ideally, we'd like to automate this, but if you need to debug, this may help as a workaround, as long as the node names will resolve with ssh.

[hairyhenderson]

The only challenge is getting to that node.

That's exactly my point 🙂

The problem with your discover-node-then-ssh approach (which I do use on occasion) is that it presumes the host is reachable via SSH. There's no strict requirement for SSH to be available on swarm workers otherwise, so that's where some sort of docker service exec command would be useful.

I have no _pressing_ requirement for this _right now_, but I hope this helps convince you that this is a legitimate use-case 🙂

[stevvooe]

The problem with your discover-node-then-ssh approach (which I do use on occasion) is that it presumes the host is reachable via SSH. There's no strict requirement for SSH to be available on swarm workers otherwise, so that's where some sort of docker service exec command would be useful.

This is just a suggested possible workaround and basically what needs to be built it.

[sylvainmouquet]

+1 docker service exec

[pnickolov]

+1 docker service exec
Not sure if this is one or two different features: one is to be able to run the command on a specific task; the other one is to run it on all tasks of a service. Having at least the first one will be great!

[pnickolov]

I would love to see this working, especially now that 17.05.0-ce has service logs (I also think service signal, pause/unpause and top may provide a good closure, too).

In the meantime, check out this swarm-exec tool on github or use the ready container.

Essentially, it provides the equivalent of the desired docker service exec <task> <cmd> [<args>...] using the following command executed on the swarm manager node:

docker run -v /var/run/docker.sock:/var/run/docker.sock
    datagridsys/skopos-plugin-swarm-exec \
    task-exec <taskID> <command> [<arguments>...]

For details, how it works and current limitations, see the project on github

[duro]

I too have a use case for this. I have a swarm (via Docker Cloud) that I use to deploy feature branches too using a CI server. I need a way to trigger some cleanup tasks that can only be ran from inside the container when a branch is deleted.

My flow looks like this:

Delete Branch -> Github Webhook -> Cleanup CI Job Executed -> docker service exec '.bin/perform-cleanup.sh'

This is just my use case, but I'm certain there will be increasingly more of solid use cases for this feature.

[rdxmb]

As a multiple tasks/containers can belong to one service, it would be more explicit to have a
docker task exec

With having
docker service exec
which task/container will be used?

That doesn't matter? Randomly? Ok, if it will defined and designed like this, no problem.

//edited:
When we talk about use cases, this can also be Backups (like database-dumps) triggered by cron on the Docker Hosts, as there is usually no cron in the official docker images.

[RehanSaeed]

I'd be happy with docker service exec entering a random container. Getting into a short lived container for trouble shooting is a nightmare at the moment.

[duro]

I tend to agree that a random container would be fine. I need this to run things like crons and cleanup scripts before teardown, so any of the containers would suffice.

Alternatively, you could pass one of the IDs of the running processes that show up when running docker service ps <service_name>

[rdxmb]

just wrote a workaround through ssh and bash for that. Maybe it is helpful for somebody.

https://github.com/rdxmb/docker_swarm_helpers

[cypx]

Thanks @rdxmb

I make a less sophisticated but very similar one a few days ago ;-)
https://gist.github.com/cypx/a2b4765fc1ccb497c99f77c78d0d4933

[westfood]

Hmm... I would like to see

docker task exec to run in defined task, ideally with multiple task ids (if it does not break expectations regarding cli behavior)

and
docker service exec to run across all service tasks

[anthony-o]

As I said here, the simpliest command I found to docker exec into a swarm node (with a swarm manager at $SWARM_MANAGER_HOST) running the service $SERVICE_NAME (for example mystack_myservice) is the following:

SERVICE_JSON=$(ssh $SWARM_MANAGER_HOST "docker service ps $SERVICE_NAME --no-trunc --format '{{ json . }}' -f desired-state=running")
ssh -t $(echo $SERVICE_JSON | jq -r '.Node') "docker exec -it $(echo $SERVICE_JSON | jq -r '.Name').$(echo $SERVICE_JSON | jq -r '.ID') bash"

This asserts that you have ssh access to $SWARM_MANAGER_HOST as well as the swarm node currently running the service task.

This also asserts that you have jq installed (apt install jq), but if you can't or don't want to install it and you have python installed you can create the following alias (based on this answer):

alias jq="python3 -c 'import sys, json; print(json.load(sys.stdin)[sys.argv[2].partition(\".\")[-1]])'"