Rexray: Create plugin for Ceph RBD

[clintkitson]

Summary

Create a Docker managed volume plugin for Red Hat Ceph from existing rbd storage driver.

[wzrdtales]

Has work on this one already started? Or nothing yet?

[clintkitson]

@wzrdtales Nothing yet.

[clintkitson]

We had a discussion internally about this implementation. It is perfectly acceptable to assume with this plugin that host level dependencies are in place to make this plugin function as expected. For example, it is not in scope to for REX-Ray to handle kernel module loads.

Due to Docker's handling of paths, it is very difficult to create a plugin today that works across multiple OS types and Ceph versions due to paths. These distributions have discrepancies between paths and required dependencies. Unfortunately, these specific paths if specified to Docker and through plugins, will cause the plugin to not load. It is perfectly fine in the short term to map / of the host to /rootfs (RO) inside REX's container to give the plugin access to all paths. At that point the Ceph storage driver would have to make sense of the appropriate paths.

[wzrdtales]

@clintkitson Agreeing on this, especially having the right kernel version in place is the job of the host, but there are problems with CEPH in general on somewhat "older" kernels like the 4.4 one, which does not work properly for some features, so this shouldn't be something new to CEPH users. Anyway you might want to put this bit of information into the docs to avoid possible confusion.

[codenrhoden]

I'm working on this currently. I have a working prototype, but have a few things to explore still...

Currently our plugins are based on Alpine, but if I want to install the Ceph CLI tools in the container, those are not (yet) pre-packaged for Alpine so what I've done so far is build this off of a CentOS 7 image. I've got this working, but have only tested when the host OS is also CentOS 7. I need to test if this works on other distros still.

The other thing worth testing is if instead of installing the Ceph CLI tools, I can map them into the container image instead. And if so, can I make this worth with Alpine? Or do I still need to switch distros?

more details to come. But what I have so far is at https://github.com/codenrhoden/rexray/commits/feature/rbd_plugin.

An extremely unstable plugin is also available at https://hub.docker.com/r/codenrhoden/rbdplugin/, such that you you can reference codenrhoden/rbdplugin:testing. This is my personal testing image, so it changes often. It will also disappear once we release a real one.

[codenrhoden]

Another update.

It does not appear that I will be able to map in the binaries/executables to the container, which would have reduced image size considerably. While I can produce a working plugin via this method, it is not portable across distros. It's not the executables (we need /usr/lib/{rados,rbd,ceph-conf}, but it's the dynamic libraries that are a problem. Libraries like librados, librbd and others are needed, and their location is different between CentOS (/lib64) and Ubuntu (/usr/lib).

Since you have to give the docker plugin a list of bind mounts, and if any of these are missing the plugin fails, I can't produce a single plugin that works on both CentOS and Ubuntu. That's disappointing.

On the flip side, the plugin I made that is based on CentOS 7 seems to have no issues working on CentOS 7, Ubuntu 16.04, and even CoreOS. The CoreOS testing wasn't completely valid, as the latest stable release includes a Docker version that doesn't have Docker managed plugin support (no docker plugin) command, but if I run the plugin as a regular container, it works fine, which is promising.

There was one hiccup I ran into. Even though the plugins are launched as privileged containers, the /sys directory within the container was showing up as ro instead of rw, which prevents rbd map from succeeding. I found a few references to bugs within Docker and this issue, but no smoking gun I could point to. Ultimately I worked around the problem by having our REX-Ray helper script fix this before starting the service. Not ideal, but the best I could come up with for now.

The plugin testing image has been updated. Anybody who feels comfortable with it please give it a spin. I'll be opening a PR for it soon.