Compose: Feature request: add scale parameter in yml

[jmmills]

As a user of compose (formally fig), I would like to be able to specify the number of nodes started for any given definition (a.k.a scale) from inside the manifest (yaml configuration file), so that I can ship my cluster definition with my service orchestration.

E.g. syntax:

worker:
    build: rqworker
    scale: 5
    links:
       - redis
    command: rqworker -u tcp://redis 

[aanm]

@jmmills Can't you start your containers such as: "docker-compose scale worker=5" and that service would start with 5?

[jmmills]

@aanm Yes, but I think that functionality should be mirrored as a default in the service definition. Maybe I have a minimal set of workers that should always be running and I want that to be clearly declared as a default.

[prologic]

@@aanm Do you expect that a docker-compose up should take the scale parameter into account? The only problem with this that I see is that we're introducing parameters into the declarative configuration that are NOT compatible with the underlying Docker / Docker API concepts but very specific to Docker Compose.

If we were to do this going forward; I'd suggest something like:

#!yaml worker: build: rqworker $scale: 5 links: - redis command: rqworker -u tcp://redis

Where $<parameter> denotes a Docker Compose specific thing.

[aanand]

We've gone back and forth on whether scale belongs in the YAML; there's been a PR to add it before (#630). I'm still of the opinion that most people shouldn't put scale numbers in their config, because it makes it less portable, but I understand the use cases for doing so.

Now that we've got a rudimentary way to augment Compose files with extends (and hopefully better ones soon - #1380, #758), the concerns I raised in https://github.com/docker/compose/pull/630#issuecomment-69210279 are perhaps less of an issue.

[jamshid]

I'd like to set scale=0 in my yml for test-related services that I don't normally want started. I only want to create those services explicitly with docker-compose run or an explicit scale .

[jmmills]

@jamshid I've often wanted that, a definition that sets up an environment but doesn't run by default. I've been relegated to creating a base image (which a zero/no-op scale would also help with) in which I run my unit tests out of (via docker run), and then my container composition consumes the base image.

Something like this seems pretty useful for dev configurations

myproject:
    build: .
    command: nosetests
    scale: 0
    links:
       - redis
redis:
    image: redis
apiserver:
    image: myproject
    command: plackup
    links:
       - redis
workerserver:
    image: myproject
    command: rqworker
    links:
        - redis

[prologic]

@jamshid @jmmills What about a enabled parameter/key in the YAML file per service? Such that you can disable/enable a service?

[jmmills]

@prologic Why do that when the a "scale" parameter would solve both needs?
If you want to imagine a running process/container as an instance of a class, one could even name it instances

[prologic]

@jmmills I'm just trying to find a _solution_ to your use-case that doesn't involve breaking the current docker-compose as such. I do tend to think scale=0 doesn't seem that fitting and I'm in two minds about whether scale=X should even be part of Compose itself.

[jmmills]

In my opinion scale (or number of copies) is part of the composition of a service, thus should be included in compose.

[prologic]

Well I think we either have a scale=0 or disabled key.

[JeanMertz]

:+1: on having the capability of setting a default scale size for an instance. And I agree, once scale is in, there is no need for a disabled key, as you'd simply set scale to 0.

[Filirom1]

+1

[jmmills]

Also, another use case: What if I want to scale the number of containers but don't want to background all of the services, or have to jump over to another terminal (or process) and set my scale numbers...
e.g:

$ docker-compose up && docker scale thing=4

Doesn't work because up doesn't exit.
But if my composition file sets the scale of my containers...

$ docker-compose up

Becomes DWIM.

[prologic]

I'm not sure that I really like this; all of a sudden up takes on two capabilities:

• Bring any containers up that don't have scale parameter.
• Bring any containers up that have scale=0.

We're now abusing the "up" command really. "Scale" also takes on new meaning in that it now does two things:

• scale containers up/down
• Disable containers/services by way of `scale=0

[jmmills]

Why would up bring up containers with a scale=0?
Build would build images with a scale=0, thus facilitating a base image need.

[prologic]

I _could be wrong_ but reading your last comment it kind of implied that :)

[jmmills]

Let me elaborate:

base_thing:
    build: .
    scale: 0

thing:
    image: base_thing
    # scale: 1 implied by default

workers_for_thing:
    image: some_other_image
    scale: 4
    links:
      - thing

test_harness:
    image: base_thing
    command: nosetests --where=my_test_code_dir --all-modules
    scale: 0

Now, expected behavior: docker-compose build builds any containers with "build" (does compose pull in external images on build? don't remember), docker-compose up would run anything with a positive scale (default is 1), docker-compose run test_harness would build "base_thing" if needed and run my one command. Savvy? :)

Edit: docker-compose up would run 1 "thing" and 4 "workers_for_thing"

[prologic]

Okay :) Thanks; your example makes a bi more sense and a bit clearer as to the intention of scale=0

I think docker-compose "pulls" images on up / run.

[Miuler]

I need create recipe indicate number of de intances (scale), for test, production, qa, etc.

[mclate]

+1 for scale=X. This would be very helpful.
And +1 for @jmmills comment with configuration description and expected results.

[verges-io]

Yay! for scale=x. Initializing a set of containers would definately help to identify potential race conditions when setting up cluster configurations.

[michaelcontento]

+1 for scale=x (including scale=0 to disable services for the initial docker-compose up)

[gnoymmij]

+1 for scale=x.

[aanm]

x is NaN, I would propose -1 instead.

[samuelololol]

+1 for scale=x.

[sutyrin]

+1

[PaulPetring]

+1

[petarmaric]

How about we stop with the +1's, please?

[shofetim]

+1'ing is useful to see the level of interest for a feature.

[petarmaric]

@shofetim I know a better way to do just that: implement the feature in question and send out a pull request...

[coconitro]

+1ing is also a good way to see people agree on a proposed solution. Its pretty common behavior across github. Clicking the unsubscribe button from notifications will turn these off if thats a problem.

[jmmills]

Well, it looks like people like this. There is a similar item in the compose backlog (left over from Fig), I'm pretty sure I made a comment on it at some point. I'll try and follow up with it later tonight.
I'm at PuppetCon most of this week, so hopefully that affords me some hack time - I'll see if I can write this.

[wkonkel]

Here is a work around for the "scale=0" use case:

app:
  build: .
  environment:
    - DATABASE_URL=postgres://user:password@host:5432/dbname
  command: sleep 999999999999

app_web:
  extends:
    service: app
  ports:
    - "3000:3000"
  command:
    # intentionally blank to use Dockerfile's default RUN command

app_worker:
  extends:
    service: app
  command:
    rake jobs:work

[jmmills]

@wkonkel yeah, I've done similar things in the past.

[jmmills]

I'm currently working on familiarizing myself with the compose codebase, once I know where all the things are for compose, I'll hack up a PR for a scale configuration parameter.
It doesn't seem like it's going to be too hard, there is a scale method for a project which is the backend for the cli interface, so all I should really have to do is add "scale" to the field schema, make sure that if it's present to call it after container creation... then make sure it doesn't run a container if it's set to zero.

[dnephin]

There is actually a really old PR open for this: #630.

The problem is that scale is an operational concern, it's not part of the application definition, so it doesn't really fit with the compose file.

It would be nice to support a configuration for a default scale level, but I don't think the compose file is the right place for it.

---

The case of scale: 0 should already be addressed by #1754. A "no-op" container can just have a command that exits right away (echo, true, etc). The cases for wanting scale: 0 are usually one of two: data-volume containers, or "adhoc/admin tasks".

Pretty soon we shouldn't need data volume containers because volumes are getting new API endpoints, and we'll be able to define volumes without the need for a container.

Administrative tasks are better handled with #2051. You can define admin.yml which extends the core docker-compose.yml and allows you to link your administrative tasks to the "core composition" without muddying the definition of each.

Both of these changes are in master now and will be available in the 1.5.0 release (which is right around the corner).

So that only leaves the case of wanting to scale a service to > 1 by default. It's already pretty easy to script something like this, but being able to put it in a config file would still be nice. I think we're going to explore the idea of a separate config in #745. It is my opinion that this new config would be a good place for things that are not part of the application definition (project name, default network name, default scale, etc).

[jmmills]

Respectfully I disagree with scale being only an operational concern. Applications can care about minimum count of services running.
As far as no-op container, it feels kludgey to actually run a container when the purpose of that container is trigger a base image to be built in which other containers use for their image field.

[dnephin]

Applications can care about minimum count of services running.

Could you give an example of this case?

As far as no-op container, it feels kludgey to actually run a container when the purpose of that container is trigger a base image to be built in which other containers use for their image field

Is there a reason the base image needs to be part of the same build? As I call out in #1455 compose is not primarily a "docker build" tool. It's goal is to provide a composition of containers at runtime. Trying to support every possible build scenario greatly increases the scope of compose and brings the focus away from container composition. It would be difficult for even a tool designed around building images to support every one of these requests. I think a better direction is to keep as much of the build complexity out of compose, and let users swap in the appropriate build tool in place of docker-compose build.

[wkonkel]

The use case I care about is scale=0 (perhaps abstract=true would be a better descriptor). I want to share images and environment variables amongst different commands. Specifically I want one web server running and one background jobs server running, both with the same code and both with the same environment variables.

[dnephin]

@wkonkel using your example, I guess this would also work?

app_web:
  build: .
  ports:
    - "3000:3000"
  environment:
    - DATABASE_URL=postgres://user:password@host:5432/dbname

app_worker:
  extends:
    service: app_web
  command: rake jobs:work
  ports: []

You trade having an abstract service with a no-op override command for no abstract with a no-op override on ports. Does that sound right?

1988 is related to abstract services.

[wkonkel]

@dnephin Yes, that does work in my case.

[wkonkel]

Actually, I take that back... I just tried it with docker-compose-1.4.2 and it seems that "ports: []" doesn't override the parent, so the app_worker fails to start with "port is already allocated".

[jmmills]

@dnephin The thread has maybe better explanation, but I'll attempt to articulate it here.

The first that comes to mind is job systems where separate containers running the same code could be modeled like a parent->fork()->child pool type of service in which the default application configuration wants a minimum number of workers for concurrency.

My inspiration for this came from an app that uses RQ workers attached to different queues that shared a base image that contained my python packages (worker code), but had multiple instances running for each queue. Minimum availability of concurrency was a requirement of the application because of long running jobs.

I just think that having a no-op command seems like a waste of resources just to get a shared base image built in the same way as the rest of ones application stack. You end up with a while/sleep loop just for a base image, which is a cool work-around, but doesn't seem like an intuitive way to accomplish this. Not to mention leaves an item in our process tree with no runtime function.

If docker-compose is to truly not cross-over to the domain of codifying image build relationships, then maybe build option should go away, and some other build definition systems should be created so that I can define a build target to build a base image, and then other images that consume that base with a few of the modified artifacts/configuration, and in the correct order?

Or maybe, I'm just being too opinionated and should wrap my docker compose commands in shell scripts to start up my services with scale, and define all my images builds as make targets with dependancies.

[dnephin]

I just think that having a no-op command seems like a waste of resources just to get a shared base image built in the same way as the rest of ones application stack.

With #1754 (in the next release) that isn't necessary anymore. A service can exit and it won't stop the rest of the services. So you can have the base just exit and not worry about it.

[jmmills]

@dnephin Cool, so would you then provide a link to that base/intermediate container in order to make sure it gets built first?

[pwaller]

@dnephin: I have a CI runner which does the equivalent of docker compose up. I want my test environment to run with multiple versions of a service (i.e, scale). I could copy the whole configuration block but this would involve repeating myself. In this case, it isn't "just an operational concern", it is something I need in my development environment while I develop a clustered application, which is currently fully described a compose file. At the moment I would have to have some out-of-band scale configuration and then somehow invoke docker-compose scale, I suppose, but this doesn't seem ideal and introduces further opportunities for failure and racing.

[bitgandtter]

In production cases you may want to star your services with a minimum scale. Let's say for example you are migrating from one cluster to another, and lets keep some hard parts of the migration as copy data out for the simplicity of the example; and you really need to start dealing with some traffic from the start so you need to deploy with docker-compose up at least (n) instances of some service lets say web.

Having scale under the service on the config file will really handle that. It's also an usability use case since if the use case in fact expose the requirement of having at least (n) instances of some service runnign and not just one at start point.

From my point of view scale is in fact a defining parameter of a topology and composing.

[nazar-pc]

@dnephin

Could you give an example of this case?

Consul, MariaDB Master-Master or any other distributed app on Swarm really needs to have at least 3 nodes in cluster to be reliable. There are definitely use cases for number of services being available in config file, I do not understand why are you so against it. Big :+1: from me here.

[dnephin]

I'm not against a way to configure scale, I just don't think it belongs in the compose file because it is state that changes independently from the compose file.

Take this example which assumes we add some scale config to the compose file:

1. I set an initial scale of 3 for a service
2. I run docker-compose up -d, my service scales to 3
3. I run docker-compose scale service=4
4. I make some changes and redeploy with docker-compose up -d.. What happens? Does it down-scale to 3 again? Does it ignore scale entirely now?

Neither of these scenarios sound good or appropriate, and this is just a trivial example which ignores instance failures (which makes it even more complicated).

If it were to get added to the compose file, I think we'd want to remove the scale command, so they can't conflict.

The example you give is an example of an operational requirement. You don't need multiple masters to run the application, you need it for reliability (operation) of the service. And you can still accomplish that with scale db=x.

[bitgandtter]

Using the example given above by @dnephin:

First i think the scale command is also a naive one, since you cannot tell if your going to scale up or down a service, having two diff command to explain the intention of scale-up or scale-down by just telling how many services do you want to create or remove respectively, will be much better.

The answer to the question raised by @dnephin is that i expect to be as many services/containers running as before the modification.

Compose is an stateless tool it doesn't know or monitor the services/containers it use the docker api to help Ops to orchestrate an there its where i think the issue lie, compose was designed to be a help tool not a full orchestration service, and now we need that one, we have engine that runs just nice, we have machine to provision and we have swarm to join all of them into a cluster but we leak a real orchestration service/tool a one that gave us the flexibility to configure and to manage the deployed services/containers something like k8s does. Right now compose is the thing that is much alike to that but if it is not in the future of this project to evolve into something bigger the best think will be from the official developers and maintainers to tell about it so we can figure it out another tool that can do it.

Personally i think it will be better to evolve compose since its a great tool and well known for all of us.

[nazar-pc]

I'll be happy to see this: docker-compose > docker compose.
Not sure about the rest of tools, but for this it would be really nice to have stateful integration with engine. Sorry for a bit off-topic comment.

[dnephin]

I've created #2496 for my proposal of removing the scale command and replacing it with the scale option (from the post above). Feedback on this proposal would be great.

[bitgandtter]

@dnephin with the #2496 compose lose the ability to scale up or down easily, we will need to reconfigure the compose file and then run compose up again just when what we just need was scale one instance down or up, i think this is pretty complicated.

Again we are losing the point that all this scenarios come from the point that compose is an stateless tool and from that it cannot control the state of the services and how many of them are at some point.

The scenario you mention in the new proposal will be easily solved by an state-full service/tool new-compose.

[npetzall]

+1

[AdamAJeng]

+1

[kubaracek]

+1

[jmmills]

@dnephin Wouldn't the behavior be the same if you do:

$ docker-compose up -d && docker-compose scale node=4 && docker-compose up -d

Isn't that more of an issue with how the compose persists scale internally.

[dnephin]

That's the thing, "Compose the application" is stateless. The only state is in 1) the compose file, and 2) the docker container labels managed by the engine.

The compose file is only edited by a human, not by Compose. It would be a mess to try and write out yaml with the same comments, order and structure. It's not supported by pyyaml, and it's not something we'd really want to do anyway.

The docker engine isn't aware of scale, so it can't store that state. It's possible with much larger architectural changes, but that's somewhat outside the scope of this issue.

For now our options are basically to either just keep scale as a command line options, or move it into the compose file, but as I describe in #2496 I think having both is confusing and leads to incorrect behaviour.

up && scale && up actually does the right thing now. up will recreate all containers, and since there is no scale value in the config, there is no confusing about what the desired scale should be. It was already set by the scale command, and tracked by container labels.

[jmmills]

@dnephin I think I agree with what you are saying, [I might be wrong in this assumption] that what you are really questioning is _if_ number of instances of a component is actually the concern of the composition of a service (a grouping of containers running different component containers). I would say that it is currently addressing that concern, just with the caveat of lack parity between the cli and the composition definition (yaml).

If the scope of responsibility is determined that scale isn't the concern of composition then scale should be removed as a cli option (probably angering a lot of users), or if it is determined that scale is the concern of the composition of a service that there should be parity between the cli and yaml with the additional support of a minimum instances to account for clustered instances that require N+1.

[ffissore]

Agree with @jmmills Especially when running clusters of data containers, when availability and replication go hand in hand, a minimum scale is part of the application: without proper scale, it may not even work

[Jasperswaagman]

+1

[lutskanu]

+1

[bugcy013]

+1

[caioquirino]

Currently i need to declare things like that:

selenium-chrome1:
...
selenium-chrome2:
...

It would be nice:
selenium-chrome:
scale: 2

+1

[BrianAdams]

exactly what @caioquirino said. +1

docker-compose up should bring up a working environment without requiring other work. The only way to bring up services that require multiple nodes is to use the pattern @caioquirino mentions. Ignoring the DRY violations here, the pattern smells wrong when you then try to use the scale command to add another node as it is not clear which 'service' should be scaled.

Scale in the yml file fixes this.

[affixalex]

Big +1, I'd also like to use this to define services that have scale set to 0 initially.

Along the lines of...

consul_bootstrap:
  build: ./consul
consul_master:
  build: ./consul_master
  scale: 0

The idea is that you could use the same docker-compose.yml and seamlessly transition from a development environment to production. The consul_master instances would automatically join the raft and establish a quorum by doing docker-compose scale consul_master=3. This would be pretty cool.

[affixalex]

To respond to @dnephin...

Take this example which assumes we add some scale config to the compose file:

1. I set an initial scale of 3 for a service
2. I run docker-compose up -d, my service scales to 3
3. I run docker-compose scale service=4
4. I make some changes and redeploy with docker-compose up -d.. What happens? Does it down-scale to 3 again? Does it ignore scale entirely now?
5. Love it
6. Alright
7. I'm still with you
8. I think there should be some differentiation between an existing deployment and a new deployment so that continuity of scale can be maintained for existing deployments that are receiving ongoing updates. An individual deployment would include the DOCKER_HOST env etc, along with the scale of each component, image IDs, and a history of redeploys. This could be a good place to hook into upstack mechanisms like continuous deployment or blue green deployments? I am imagining a slightly more production ready workflow, so you can simply connect to a different DOCKER_HOST and do a docker-compose up -d and then provide hooks so that upstack tools can manage things like CI and blue/green rollouts. Maybe add something like DOCKER_COMPOSE_ENV={"testing " || "production" || "dev"}?

[ffissore]

IMHO, having "hardcoded" scale, say to 3, it should start with 3 containers

[ceejatec]

I think a lot of the confusion about a "scale" parameter vs. the "scale" command would be alleviated by naming the YAML parameter "initial_scale" or "default_scale". That makes it pretty clear that it's just a value that will be used if there isn't some kind of override. It also (IMHO at least) seems reasonable that this "initial_scale" parameter would only be referenced when docker-compose brings up the service for the first time, and not when running docker-compose up again with some instances of the service already running.

[honsiorovskyi]

+1 for "initial_scale", also it will explicitly describe this parameter as compose-specific, because docker itself doesn't have such parameter.

[bitgandtter]

There is an interesting thing happens, in the blog post https://blog.docker.com/2016/02/video-containers-as-a-service-caas/ at video "48.56 shows that on the new released docker datacenter you can at stack creation point actually setup how many instances of a container you want to run. So this idea its not new to the docker team, hope it gets to be included in next compose releases

[migueleliasweb]

+1

[jeffersonsouza]

+1

[gochev]

Millions of issues and requests scale/initial_count and so on .. and still not planned.

Sad seeing otherwise good project but peoples only finding excuses and creating new and duplicate issues, closing issues and "doing" work.

[melya]

+1

[EricMeallier]

+1

[AndreyUtka]

+1

[AndrianBdn]

+1

[pierrre]

PLEASE STOP!
DON'T ADD +1 !
JUST SUBSCRIBE TO THE ISSUE!

[draev]

+1 (@pierrre sorry)

[aka-toxa]

+1 its very important (@pierrre, sorry)

[pierrre]

I agree, but I don't want to receive 100000 notifications for your "+1"!

[jmmills]

/me forks docker compose and starts learning the codebase.

[sergeyklay]

[aperepel]

Many valid points above, it looks like having a default/inital scale parameter in the config file + a scale cli command will address both. To add use cases in support of an initial scale option, consider clustered services where a quorum consists of an X number of nodes and which shouldn't be operational otherwise (e.g. no less than 3). The point of docker-compose is to have a fully-contained descriptor of such a system.

[jasonf20]

My use case is:

I have extended a service with some common settings to allow for less duplication. The base service is however created (when running without explicitly specifying service names) and then I shut it down manually. I would like the base service to have an initial scale of 0.

[mattgiles]

I am in general agreement. Though to my mind, this issue and the linked/related issues boil down to a slightly different problem: up and scale ultimately want to be the same command.

They sometimes are already:

• docker-compose up service
• docker-compose scale service=1.

And, if scale exposed all the cli options (e.g. --force-recreate) that up does, or if up knew about counting, then they basically would be.

If, as this issue proposes, an "initial scale" directive is added to the docker-compose.yml, then up will have to learn to count. At which point, shouldn't up just support the counting syntax docker-compose up service=10 (which could overwrite any scale defined in the yaml)?

There's a distinction being drawn between "initial scale" and "scale", which could be the domain of different commands. But because up is idempotent and designed to be run repeatedly without necessarily changing state, I don't think this distinction is strict. I think we should instead consider having up cannibalize scale outright.

[dnephin]

@mattgiles I agree, that's where I was going with #2496, however I hadn't considered that the service=count could be added to up. I think that might handle the situation. The only problem I see is that currently up web means only bring up the web service and it's dependencies. Trying to do up web=3 would have to continue to mean the same thing. Which means that there is no way to up everything and override the scale count at once, but that's probably ok.

I'll update the proposal in #2496

[mattgiles]

@dnephin I missed the more recent proposal. Awesome!

fwiw, I think it is totally intuitive for docker-compose up web=3 to bring up three containers for the "web" service, as well as any dependencies... in the ratios defined in the yaml (per something like a scale directive). That way the compose file continues to govern unless overridden by the command line.

Subsequent invocations of up could continue, as they do now, to leave existing containers alone. Only if the current scale is smaller than that defined in the yaml would it then change counts to be in line with the proscribed scale. No scale defined would continue to imply a scale of 1.

Things like --force-recreate should also probably leave counts alone if higher than defined in the yaml, but still recreate all containers to be in line with other attributes.

Would also make sense to me to be able to kill containers, as one currently can with scale, by calling something like docker-compose up web=2 etc etc.

[OferE]

+1

[foxx]

+1 (just to annoy @pierrre ;) )

[artemkaint]

+1

[ramsrib]

+1 for scale=x.

[bfosberry]

Related to this I would love a way to describe Singletons. scale=1

[lavvy]

Any success here with the scale option

[astewart-twist]

+1

[jmmills]

I haven't seen anything new on this from upstream , personally I haven't had a chance to see if I can hack up a patch for it.

Thanks,
Jason Mills

• sent from mobile.

On Aug 8, 2016, at 5:19 PM, lavvy [email protected] wrote:

Any success here with the scale option

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[drew-r]

+1

[pablolibo]

+1

[alirezaDavid]

+1

[cgarciae]

+1

[foxx]

+1 (to annoy @pierrre again)

[darkdefender27]

+1

[alvinr]

I would like to re-iterate the case

scale=0
Useful for test, debug. probe etc. services that you don't normally want to bring up, but you want defined - because they are usually used in context of the other services in the file.

scale=1
Because I want one-and-only-one of these - _ever_. In an MariaDB cluster, there are case where I may only ever want a single node configured with a specific storage engine (e.g. Spider, ColumnStore etc.).

scale=n
Because there are times when I know I need 'n' services running, for example, a MariaDB cluster where I will always have one Primary (with its configuration) and two Secondaries (with their different, from Primary, configuration).

[boyvinall]

Makes perfect sense to me @alvinr. The default should prob be 1.

[westlakem]

DISCLAMER: I am not a representative of Marathon or Mesosphere

This is yet another example of a fairly simple feature request that has been abandoned / pushed off by the docker support team for over a year. https://github.com/docker/docker/pull/12749 is another example. Kubernetes and Marathon have had these options for quite some time now ( "instances": 3 in the marathon.json file) and even implement auto scaling. It's the unwillingness of the docker support group that has pushed me away from docker data center & swarm for some time now.

[matti]

Kontena has this also as instances: 3 (https://www.kontena.io/docs/references/kontena-yml)

[hugochinchilla]

Another way we could do this is to add a scale section on the compose file version 2.

It wouldn't mix with the docker-only options used in services.

An example would be:

version: "2"

services:
  worker:
    image: something
    command: work

scale:
  worker: 2    

[drew-r]

This with the ability to specify scale: 0 would potentially kill two birds with one stone, the other being https://github.com/docker/compose/issues/1896

[mschwartz]

This seems like a no brainer.

I want to deploy nginx+nodejs+mongodb with 3 or 4 nodejs instances. Every time I start up the app.

If you have a command line option to do it, why not in the yml file?

docker-compose up scale nodejs=4

Would do.

[westlakem]

It's simple. They want to go the route of "someone else build it and we
will try to support it". Other than base docker, they haven't created
anything. Compose was a different product they consumed and now they have
no idea how to support/extend it, which is why products like marathon and
kubernetes are more popular then swarm.

On Oct 16, 2016 9:36 PM, "Michael Schwartz" [email protected]
wrote:

This seems like a no brainer.

I want to deploy nginx+nodejs+mongodb with 3 or 4 nodejs instances. Every
time I start up the app.

If you have a command line option to do it, why not in the yml file?

docker-compose up scale nodejs=4

Would do.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/1661#issuecomment-254093296,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AES98nZt1uIejnIU9iajVt54QbzdlVK1ks5q0tETgaJpZM4FS8ZQ
.

[jeremysmitherman]

I'm honestly a little astounded that there's _any_ pushback on this as it seems so obvious to have...

[foxx]

@westlakem Lets hope that Docker for MacOS doesn't suffer the same fate, hopefully the Unikernel guys stick around.

[hholst80]

It would be nice to bring up a complete stack at its functional size in one go. In our current use case the service stack needs to have "workers=16" to work properly.

This is just awful:

docker-compose up -d
docker-compose scale workers=16
docker-compose down
docker-compose up --abort-on-container-exit

which clearly could be replaced with something such as:

docker-compose up --abort-on-container-exit --scale workers=16

Edit: I see that "Rancher" has a docker-compose like syntax and support an initial scale parameter: https://paypertrail.com/blog/tech/docker-rancher-and-laravel-easy-and-safe-scalability

[maxigit]

A workaround is to use YAML anchor and merge capabilities and duplicate the service within the docker-compose file.

services:
  toscale: &my_service
     ... # all paramteres
  # second instance
  toscale2: 
     << : *my_service
     ports:
       - 81:80 # override port if needed

etc ...

[hugodes]

This would be super useful !

[aphillipo]

This is super weird this doesn't exist...

I might end up having to create duplicate services because I'm that lazy.

[bitgandtter]

This is already on the work on composer file v3 https://github.com/aanand/compose-file/blob/master/schema/data/config_schema_v3.0.json that will be and its already supported on docker-compose v1.10.0-rc1

[westlakem]

Their "Services" offerings have the parameter built in. Very sad that as a
subscriber to this issue no one from the development team informed us that
it was coming in the Services product, and someone had to read the RC notes
for 1.10 for any insight. Where are the docker reps?

On Fri, Jan 6, 2017 at 6:30 AM, Yasmany Cubela Medina <
[email protected]> wrote:

This is already on the work on composer file v3 https://github.com/aanand/
compose-file/blob/master/schema/data/config_schema_v3.0.json that will be
and its already supported on docker-compose v1.10.0-rc1

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/1661#issuecomment-270886347,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AES98nOAW-h0BVaR8D9uQfLpMCQRfdyfks5rPiXEgaJpZM4FS8ZQ
.

[charneykaye]

+1

[mantisbayne]

+1

[jjqq2013]

Hi, i have a related question.
I was ever wondering that docker-compose up can remember the scale i ever set.

For example, i run docker-compose scale nodejs_web=4 then docker-compose down,
then I start again docker-compose up, it will start 4 nodejs_web.

I want to know where does the scale command save the number 4?

The docker-compose.yml can be:

version: '2'
services:
  nodejs_web:
    image: node
    command: node

[alwaysastudent]

I support @dnephin proposal on #2496
Please give us this basic feature.

[westlakem]

Services in the new version of Docker does offer a 'replicas' feature.

On Mon, Mar 20, 2017 at 3:28 PM, alwaysastudent [e-mail]
wrote:

I support @dnephin https://github.com/dnephin proposal on #2496
https://github.com/docker/compose/issues/2496
Please give us this basic feature.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/1661#issuecomment-287870998,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AES98uWGEnbyCFyxNSSlv7QkXy5oBek1ks5rntNBgaJpZM4FS8ZQ
.

[alwaysastudent]

@westlakem - that feature is for swarm. It is not for the regular service scaling.

[cgarciae]

So many pros, few cons. What is the matter with this issue?

[matti]

There is a more recent proposal about completely removing the scale command in https://github.com/docker/compose/issues/2496

I really dont understand why or why there is no initial_scale: 3 option in compose file (I do understand why it can not be just scale: 3)

[shin-]

Implemented back in 1.13.0. Thank you everyone for your feedback and patience.

[greenscar]

@shin- Why is this considered closed? In managing large scale apps where EVERYTHING is automated & in version control, I now must modify my Ansible which executes Docker to define how many instances of a service will be running.

Ansible should be setting up everything so Docker can run. Docker should be defining how many instances of a service the app needs. Now it's Docker's responsibility to know the inventory of services but Ansible's responsibility to call Docker so it knows how many containers to spin up?

[shin-]

@greenscar Sorry, I'm not exactly sure what the problem is - are you unhappy about the way the feature has been implemented? If so, in what way can we make it better?

[cgarciae]

Why is this not present in version 3? Extreamly confusing since "deploy.replicas" and "up" are not equivalente, you could say this feature is missing in v3.

[shin-]

@cgarciae deploy.replicas serves the same purpose as scale. Why are they not equivalent in your view?

[cgarciae]

@shin- If I dont want to create a swarm and only use docker-compose up -d, docker compose tells me it will ignore the "deploy" settings.

[shin-]

@cgarciae Why are you using a v3 file if you don't want to create a Swarm?

[aphillipo]

I think everyone using docker expects the scale parameter to work for either docker 2 or 3 files. There is no reason for it not to and it could even set the default value for deploy.replicas as well. If deploy.replicas is set it therefore overrides scale in swarm. What is wrong with this picture apart from it being what everyone expects to happen (and of course if you don't like it no need to use it in your docker-compose.yml)?

[boscowitch]

The missing scale parameter made a servie fail, since i forgot to scale it up like it was before since the last redeloy -.- why is this not a thing it would save service and maybe even lifes depending on what its running

[tsloughter]

Just hit this issue as well. Seems there is no way to specify how many of a service to start from within the docker compose config? It is even more confusing that the docker compose documentation has a lot of documentation for things that don't work with docker-compose...

[tsloughter]

Oh wait, so if i change my docker compose config to version: "2.2" then I can use scale: 2 but if one were to want to go to a later version there is no longer an option to do this?

[tsloughter]

Ok, I see this issue of "version" not really meaning versions has already been brought up, https://github.com/docker/compose/issues/4693

[freesh]

I use v3.6 because i need some configs that does not exist in 2.2. For local development i use no swarm, but i want to set scale=0 to some containers in my docker-compose.overrides.yml. Because some of them are just for buildprocesses like the frontendbuild container. This container shares some volumes with other running containers, but should not started together with these other containers on docker-compose up. So the buildcontainer just runs with the run command. Yes i can set the --scale param on calling docker-compose up, but i think it is better to write it directly to the configuration. :)

[samrocketman]

Thanks for the scale parameter. I was able to use it in a non-production docker-compose.yml so that anybody could use a HA configuration of Consul and Vault. The scale parameter made provisioning a local HA configuration easy. https://github.com/samrocketman/docker-compose-ha-consul-vault-ui

In this case, only intended for somebody to try out Consul and Vault in a HA configuration and use Consul DNS.

[jjsaunier]

Which version of docker compose are you running on ? Since v3 this feature has been dropped (By drop, I mean, never implemented), so it's ignored. See https://docs.docker.com/compose/reference/scale/

[samrocketman]

I don’t ever use v3 format due to missing features. I tend to use the lowest 2.1 or 2.2 format depending on what features I’m using.

Edit: not to say I avoid v3 format. Just that when I go to write a compose file it is usually missing what I want to use.

[jjsaunier]

Ok, I guessed you speak about docker compose with swarm stack, that's why. I never tried, but it's weird to talk about HA with docker compose without orchestrator and mutli node. Is it for "poc" or testing purpose ? Anyway, I never tried in this context

[samrocketman]

In this case, the HA simply refers to a consul cluster and a vault cluster. It is meant to serve as a proof of concept available for experimentation and ease of bootstrap. The HA does not refer to docker HA or even multi-machine HA. It's only HA in that the person experimenting can kill one of the cluster containers and see that the consul and/or vault service is not affected.