Rexray: Are there any robust way to scale containers with persistent volumes in Docker Swarm mode?

[kitgary]

Hi,

I know this question may not specific to rexray but I want to get some advices from you guys.

I am running a Hadoop cluster in docker swarm mode, I defined 2 data nodes in the docker-compose file, and each one attached to a rexray volume, I think this approach is not scalable and I want to use docker swarm's built-in features to automatically scale containers

eg.
docker swarm service scale datanode=5

But this won't work since the new created containers will share the same rexray volume, I want to know if there are good solutions to handle this? I think this is a common use cases for persistent container like databases.

Thanks

[jpe42]

@kitgary I'm barking up the same tree with this issue: https://github.com/docker/docker/issues/31671

I originally started here as well: https://github.com/codedellemc/rexray/issues/765

[ghost]

You can use docker service placeholders for the Go template, to specify name for each container.
Ex.

volumes:
  data:
    name: '{{.Service.Name}}-{{.Task.Slot}}-vol'
    driver: cloudstor:aws

[tschwaller]

Unfortunately I get a parsing error.

ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./docker-compose.yaml", line 43, column 5
expected <block end>, but found '<scalar>'
  in "./docker-compose.yaml", line 43, column 28

[clintkitson]

Have you tried yamlint.com?

On Thu, Apr 5, 2018 at 12:54 PM, Tom Schwaller [e-mail]
wrote:

Unfortunately I got a parsing error.

ERROR: yaml.parser.ParserError: while parsing a block mapping
in "./docker-compose.yaml", line 43, column 5
expected , but found ''
in "./docker-compose.yaml", line 43, column 28

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rexray/rexray/issues/804#issuecomment-379057206, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABVMMVNhw2nzNCJy29AMN2rxKRmN_lDoks5tlnaSgaJpZM4M5gkO
.

[clintkitson]

Sorry that is, yamllint.com

On Thu, Apr 5, 2018 at 12:55 PM, Clinton Kitson [e-mail]
wrote:

Have you tried yamlint.com?

On Thu, Apr 5, 2018 at 12:54 PM, Tom Schwaller [e-mail]
wrote:

Unfortunately I got a parsing error.

ERROR: yaml.parser.ParserError: while parsing a block mapping
in "./docker-compose.yaml", line 43, column 5
expected , but found ''
in "./docker-compose.yaml", line 43, column 28

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rexray/rexray/issues/804#issuecomment-379057206, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABVMMVNhw2nzNCJy29AMN2rxKRmN_lDoks5tlnaSgaJpZM4M5gkO
.

[ghost]

String quote is missed because of special charactors, I've corrected them.
Edit: In docker swarm mode, skipping string quote won't cause the problem.

[ghost]

@tschwaller Did you run it with docker-compose? Go language's template is not supported in docker-compose in Python. Try docker swarm mode instead.