Swarmkit: Run a single task (batch job)

[rayjohnson]

In the old version of swarm we had configured a Rundeck server to launch batch jobs on the swarm cluster. It works great. However, running a single job no longer seems to work in the new docker swarm (1.12).

docker run seems to only run on the server you are connected to (unlike the old swarm).

docker service will restart a batch job which is not what is needed.

Are there plans to have docker run work with swarm? Or some other mechanism within swarm?

We are trying to work around the problem now by querying the swarm for all nodes to kind a machine to do docker run against. The "scheduling" mechanism in this hack is rand(). This is not ideal as it does not take into account used resources or labels.

Will swarm add back the functionality it ripped out? Or do we need to develop a different (more sophisticated than rand()) scheduler outside of swarm?

[aaronlehmann]

It is possible to set up a service that does not restart on completion. The service can be created or updated with --restart-condition none.

We're looking at adding a new service type geared specifically towards batch jobs in a future release.

[xiaods]

batch job is coo.

[AkihiroSuda]

linking https://github.com/docker/docker/issues/23880 to this issue

[feature request] Swarm mode should support batch/cron jobs in addition to persistent services

[stevvooe]

@rayjohnson It really sounds like your confusing standalone swarm with swarmkit and swarm mode integrated in docker 1.12. If you need docker run to be scheduled across a cluster, I'd recommend that you continue to use https://github.com/docker/swarm. SwarmKit simply does not do this. It focuses on a service model, which may or may not hit your use case.

However, running a single job no longer seems to work in the new docker swarm (1.12).
docker run seems to only run on the server you are connected to (unlike the old swarm).
Are there plans to have docker run work with swarm? Or some other mechanism within swarm?

docker run still works with swarm. You can still run a standalone swarm on top of a cluster of docker machines. Swarm-mode (or SwarmKit) provides a different functionality, based on services.

We are trying to work around the problem now by querying the swarm for all nodes to kind a machine to do docker run against. The "scheduling" mechanism in this hack is rand(). This is not ideal as it does not take into account used resources or labels.
Will swarm add back the functionality it ripped out? Or do we need to develop a different (more sophisticated than rand()) scheduler outside of swarm?

No functionality was removed or "ripped" out from swarm. Just continue running Swarm standalone on your existing cluster (even with 1.12) and things will continue to work, as normal.

[hekaldama]

@stevvooe can we run standalone and swarmkit in the same cluster? I would imagine the answer is yes, and I thought I would ask before trying to get it setup.

[stevvooe]

@hekaldama Absolutely!

We didn't want anyone's use cases to be interrupted by the addition of services. Everything should continue working more or less the same.

[rayjohnson]

So the recommended solution is to run both the old swarm and the new swarm together? Will it be maintained going forward? Why the heck did you name the same thing if they are meant to solve different things and both going to be supported going forward?

Bottom line is I'd love to use the new functionality of swarm kit for our long running jobs. At night I'd like to schedule additional batch jobs on the same cluster.

For this to work well, I'd think you would need a single scheduler that understood the resources across the cluster. No?

[stevvooe]

@rayjohnson

So the recommended solution is to run both the old swarm and the new swarm together?

Yes, if that meets your use case.

Will it be maintained going forward?

Swarm standalone is still under active development and we don't plan on deprecating unless that functionality is covered by something else.

Why the heck did you name the same thing if they are meant to solve different things and both going to be supported going forward?

They don't have the same name. We use the term "Swarm" to mean a cluster. There is Standalone Swarm and Docker Swarm-mode, which is SwarmKit as integrated into docker.

Bottom line is I'd love to use the new functionality of swarm kit for our long running jobs. At night I'd like to schedule additional batch jobs on the same cluster.
For this to work well, I'd think you would need a single scheduler that understood the resources across the cluster. No?

Ideally, yes, they would be on a scheduler that had the same visibility.

In practice, these workloads are very different, so you might want to have separate banks of machines for running services and batch jobs. You can use --pause/--drain to control which nodes take your service loads from docker node update.

[rayjohnson]

Thanks for all the feedback! I had no idea both could run together - we will try that out. I think this may work for our immediate needs.

Looking forward to see how this evolves. :)

[stevvooe]

@rayjohnson Let us know if you have any issues.

[dperny]

I'm gonna close this as answered. Feel free to reopen if you're not satisfied.

[oskca]

It's sad to see this issue closed! (Update: see the feature request here:https://github.com/moby/moby/issues/23880)

K8s is good which support batch jobs, but it's really hard to get installed, and the docker-mode is great to bring a native and easy to use cluster to the general public.

While batch job is another general computation model compared to the long running services, it's needed for some amazing project,ie Pachyderm, it would be good to have batch job supported in swarm-mode to make docker much more useful and much more great! :smile:

[benjamir]

I agree with @oskca and disagree that batch jobs and services should run on different machines -- nice if you can, but who is paying the bill? And why should I pack my VMs/containers tightly and risk loosing a lot of infrastructure at once, when I can distribute them and use the extra resources for peak utilization?
We currently use SLURM (HPC, batch) and Ganeti VMs (KVM+DRBD) on our whole cluster -- both are scheduling unaware of each other. (SLURM simply doesn't know about all phys. resources... a hack)
I'm looking into alternatives to that, K8s seems scary for a single Op responsible for a department (agree again with @oskca ) and while I thought about a custom scheduler there to replace SLURM (we don't need much functionality) I thought docker-swarm-whatever-gives-me-a-cluster-view-of-containers would be an easy to support solution for services and batch jobs.
Am I missing something?

[alexanderkjeldaas]

I agree with @benjamir. Services and batch jobs are usually tightly integrated.

Deploying a new version of a service might involve several steps that are global in nature. For example populating volumes with the correct data. Collecting and preparing data in general.

Doing this in the service is wrong - it should be a separate job to keep the service small, and to make the service fail fast - i.e. the service should be able to have strict invariants on which environments it will want to work in.

Batch jobs in the "process big data" sense might be something that is usually done in a separate cluster, but that's just a small subset of what batch jobs are used for.