Machine: Persistent volumes with OpenStack

[glikson]

Is there an easy way to have persistent docker volumes with docker-machine running on OpenStack? For example, would it make sense to have docker-machine configure & use a block storage volume backed by OpenStack Cinder? Anyone working on this?

[dmp42]

cc @ggiamarchi

[ggiamarchi]

@glikson This is not handle at the moment. Just to be sure what exactly you want to do, can you confirm the following scenario match your use case ?

1. I create a docker host using docker-machine with a cinder volume attached and mounted on /data
2. I spawn a container on this host with a docker volume mounted from /data/vol1
3. I destroy the docker host
4. I repeat steps 1. & 2. with the same cinder volume, and thus my container retrieve its data

To achieve that, i think we can add a flag --openstack-volume-mapping or something like that to allow docker-machine to attach a cinder volume to the instance but i IMO the rest of the logic should be handled outside docker-machine. I mean :

• The cinder volume should exist prior to the docker host creation
• The cinder volume filesystem mount on linux should not be done by docker-machine. It can be done using either a custom glance image or cloud-init (if #1405 get merged).

@glikson WDYT ?

[glikson]

@ggiamarchi -- yes, this is the scenario we had in mind, more or less (cc @KennethNagin).
Optional creation of the cinder volume as part of the flow might be very helpful, given that docker-machine aims at minimizing user's direct interaction with OpenStack (as far as I understand).
With cloud-init support, seems that several things which are currently in docker-machine can potentially move there. Indeed, sounds like a good direction for post-creation customization (maybe can even 'borrow' some code from Magnum templates?). However, it might make sense (at least in some cases) to consider using cloud-init under the covers by docker-machine itself, rather than expecting the docker-machine user to manage it.

[kacole2]

@glikson @ggiamarchi I took a stab at this already. Take a look at #1765 and #1881. There is a Docker Volume Driver called REX-Ray that supports Open Stack volumes. If you can use 3rd party extensions like my proposal and PR would allow, then this could easily be achieved. Let me know what you think.

[KennethNagin]

We have experimented with adding storage volume support to the openstack driver.
In brief the openstack driver already leverages gophercloud to interact with OpenStack.
We enhanced the driver so that it could also create and attach a volume to the new docker host.
We introduced three new flags:
o openstack-volume-size: When the size is greater than zero the docker machine openstack driver will create a new storage volume according to the size. The size is in gigabytes;
o openstack-volume-name: When the docker machine openstack driver creates a new storage volume it will assign this name to the storage volume. It defaults to “docker-machine volume”;
o openstack-volume-id: The volume id is assigned to the block storage by openstack when it is created. When openstack-volume-id is specified alone, without the openstack-volume-size, it will attach the volume identified by the id to the host machine;

Do you think this would be of interest to the community?

--Ken

[kacole2]

@KennethNagin i think you are being too specific and not thinking about it in terms of a framework. If you think about EC2, GCE, vSphere, etc there should be a common command that works amongst all of them. Again, there are already Docker Volume Driver Plug-ins available and those should probably be leveraged instead of re-creating the wheel from scratch.

The Docker Volume Plug-ins allow multiple commands for volume interaction such as create/delete, mount/unmount, etc. I think adding the ability to only mount volumes to one driver is too narrow. Hence the reason why I looked at implementing 3rd party drivers into Docker Machine in #1881. It would allow you to add the Docker Volume Driver of your choosing, and then perform commands to create, attach, and even run a specific container all in a single .json file.

[glikson]

@kacole2 -- I think @KennethNagin is targeting the default volume driver, allocating volumes from a local filesystem on the host. It is just the underlying infrastructure piece -- i.e., making this local filesystem backed by a "really" persistent storage - which is missing. The exact way to achieve this would be different on different platforms, depending on their block storage APIs etc. It makes perfect sense to support automated configuration of different volume drivers (potentially also leveraging Cinder underneath) too -- but it is a slightly different problem.

[avkitex]

There is another problem with persistent volumes+docker+openstack. Some docker containers are storing temp files. Some of them could be quite big. Openstack supports boot-from-volumes flavors. Their size is always small (50Gb ssd).
I had problems with free space on booted volume. So I've decided to use persistent volumes (200Gb f.e.).
But by default docker installs on system-boot disk (50Gb).
Here I have 2 alternatives:
1) Move docker to new persistent volume (200Gb) +modify DOCKER_OPTS
2) Mount the volume. This option requires my container to store all temp files in one specified directory, which is not very flexible.

I think it could be usefull to have some options about docker intalation on persistent volume in addition to volume creation/attaching/formatting/deleting.

[fsoppelsa]

@avkitex Yes, Docker installs in /, does Guillaume's proposal to create a /data mountpoint on a Cinder volume on the host sound reasonable?

Then, that containers write "many" (how many? Which containers?) "temp files" (where? /tmp?) is something that shouldn't have to do with dm provisioning. WDYT?

[avkitex]

@fsoppelsa Different programs may create temp files in different places. For example in /root (if docker user is root, or /home/user otherwise) or in /tmp or some other user-specified directory.
The problem is that each time creating docker container I don't know where is it location -> i can't mount cinder volume somewhere in container and be sure that the program is going to write temp files there.

Thus all temp files increase my container's size that leads to less free space on my system disc.

Mounting cinder volumes may solve this problem. But in some cases - not

[fsoppelsa]

@avkitex I think it's impossible to have an universal rules for containers. We're here talking about a machine's driver feature, rather than how containers will use it.

Having a /data mounted on the Docker host will let you to run containers with -v /data/smth:/tmp.

We must decide if, in this case, move Docker itself in block to /data, or only keep the /data as a (big) external utility volume to use with Docker's -v (I am for the latter).

[avkitex]

@fsoppelsa Here i wanted to say that it could be useful to have docker already preinstalled on persistent big volume instead of small system disc after docker-machine initialization process.

[fsoppelsa]

I don't know if our gurus will be in favor of this, better to ask for a consult.

Idea is, if persistent big volume is enabled, installed and provisioned, to install Docker on this instead than on /: WDYT? @nathanleclaire @ggiamarchi @jeanlaurent @ehazlett

[avkitex]

@fsoppelsa, are there any updates?

[fsoppelsa]

@avkitex mh no I am waiting for a direction by machine experts, than I can implement...

[LipingMao]

I just notice a project called "fuxi" recently add in openstack which target to do this.

Basic Introduce:
http://www.slideshare.net/nkzhangni/fuxi-introduction-63895002
Code:
https://github.com/openstack/fuxi