Rexray: Use rex-ray in docker swarm with EBS in a Multi-AZ setup.

[anirbanroydas]

Summary

I have been successful in using rex-ray in docker swarm mode with service scheduling constraint to a single AZ in AWS using EBS. And it works. But how to put it to use to work without any constraints. How can I allow services to get scheduled in any AZ (The point of fault-tolerant behaviour is useless if I am unable to do this).

New Feature

What is the possibility that rex-ray once creating a volume in a particular AZ, starts taking snapshots of the EBS volume and whenever the service gets scheduled to a different AZ where the EBS volume is not available, then rex-ray would create a new EBS volume using the snapshot and attach it to the new node. And keep taking snapshots of this new volume. And whenever there is a AZ change, the volume creation from the snapshot to be done first and only then the mounting of the volume to the node should take place.

What is the roadmap for this? Or is it not possible at all?

[kacole2]

EC2 Instance access to EBS volumes is a limitation from AWS. What you put forth is a possible solution but the engineering efforts behind it aren't prioritized. You can see the roadmap for future releases under the milestones.

[koensayr]

Possibly related to #771

[kacole2]

@koensayr EFS in #771 vs EBS here.

EBS volumes are ONLY available to EC2 instances in the same AZ. So an EBS volume created in us-east-1b is only accessible to EC2 instances in us-east-1b. It's not accessible by any EC2 instances us-east-1c, us-east-1d, us-west-x, or other parts of the world.

Docker Swarm (and other container schedulers) can have hosts spread across AZs and Regions for deploying containers. So if a host fails (or an entire AZ) the containers could be restarted in a different AZ or region entirely.

To make this possible, REX-Ray would have to replicate the volumes between AZs. If this is done when the host fails, then it's constrained by the amount of data there is to replicate. If this is done all the time, then you have to worry about losing an entire AZ and EBS not being available and the data is only as good as the replication timestamp.

This is a feature that would be beyond the scope of what REX-Ray can offer. However, it could be an idea for a new project/product for someone to tackle.

[codenrhoden]

This is more related to #561. @cduchesne explains would have to happen in https://github.com/codedellemc/rexray/issues/561#issuecomment-252442711

It's not clear (to me) whether this is a workflow we would or could take on

[nunofernandes]

This would be a great feature! +1

[kinghuang]

I'm currently looking at using REX-Ray instead of Docker's Cloudstor volume plugin for EBS volumes in Docker Swarm. But, based on experimentation and this issue, it seems REX-Ray does not automatically snapshot and move volumes between availability zones within a region like Cloudstor. It would be super useful if it did so, in order to support, for example, Docker Swarm clusters that span multiple zones.