Rexray: EBS Filesystem Type

[dl00]

Summary

How do I specify the use of an XFS filesystem type for an EBS volume?

I have tried creating the volume like so:

docker volume create -d ebs --name testvol --opt=fsType=xfs

docker volume create -d ebs --name testvol --opt=newFsType=xfs

REXRAY_FSTYPE=xfs docker volume create -d ebs --name testvol

Each command results in an ext4 filesystem upon mount.

I expect/desire for it to create an XFS filesystem somehow.

Version

I am using rexray/ebs via docker plugin. I do not know how to present the version...

The docker version output is:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Mar 24 00:00:50 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Mar 24 00:00:50 2017
 OS/Arch:      linux/amd64
 Experimental: false

[fmaturel]

Same issue here, as xfs is the recommended fs type for mongodb, I tried that one:

docker service create \
  --name mongo \
  --mount type=volume,src=vol-mongo-xfs,dst=/data/db,volume-driver=rexray,volume-opt=size=8,volume-opt=volumeType=gp2,volume-opt=fsType=xfs \
  mongo

It works but creates an ext4 fs:

$ df -T | awk '{print $1,$2,$NF}' | grep "^/dev"
...
/dev/xvdf ext4 /var/lib/libstorage/volumes/vol-mongo-xfs

Versions:

$ rexray version
REX-Ray
-------
Binary: /opt/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: 2373541479478b817a8b143629e552f404f75226
Formed: Fri, 09 Jun 2017 19:23:38 UTC

libStorage
----------
SemVer: 0.6.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: fd26f0ec72b077ffa7c82160fbd12a276e12c2ad
Formed: Fri, 09 Jun 2017 19:23:05 UTC

$ docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.6
 Git commit:   89658be
 Built:        Wed Jun  7 23:07:34 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.6
 Git commit:   89658be
 Built:        Wed Jun  7 23:07:34 2017
 OS/Arch:      linux/amd64
 Experimental: false

[clintkitson]

@codenrhoden Can you update this issue with the issue relating to the Docker volume map and milestone? @fmaturel I believe this is occurring because of a bug in the volume options translation for Docker. Currently only one option can be specified. Can you try this an update the thread?

[fmaturel]

if I use only one volume-opt:

docker service create \
  --name mongo \
  --mount type=volume,src=vol-mongo-xfs,dst=/data/db,volume-driver=rexray,volume-opt=fsType=xfs  \   
  mongo

I still get the same issue:

$ df -T | awk '{print $1,$2,$NF}' | grep "^/dev"
...
/dev/xvdf ext4 /var/lib/libstorage/volumes/vol-mongo-xfs

[codenrhoden]

i know what's going on here. give me some time to write it up properly. Several issues are being conflated.

[codenrhoden]

@clintkitson The issue that you were referring to is https://github.com/codedellemc/rexray/issues/707. Please see my final comment in there for why it is not a bug, and docker create does accept multiple options just fine. That is not what is going on here.

TL;DR you'll never be able to pass the fs type via docker create because that never mounts the filesystem -- only creates a new block storage object on the back end. The filesystem is not involved at that step.

Let's take a step back and start with just REX-Ray, not Docker...

REX-Ray lists the following options for volume create:

Flags:
      --amount                    Attach and mount the new volume
      --attach                    Attach the new volume
      --availabilityZone string
      --continueOnError           Continue processing a collection upon error
      --encrypted                 A flag that requests the storage platform create an encrypted volume. Specifying true doesn't guarantee encryption; it's up the storage driver and platform to implement this feature.
      --encryptionKey string      The key used to encrypt the volume.
      --force
      --fsType string
      --iops int
      --overwriteFS
      --size int
      --snapshotID string
      --volumeType string

So it does indeed show that --fsType is an option. However, it does not make clear that this does nothing unless you are also specifying --amount, which will attach then mount the volume after you create it. We can only affect the filesystem when we've been asked to mount it. Examples:

# rexray volume create --size 1 --fsType xfs fstest
ID                     Name    Status     Size
vol-07a050137396702f6  fstest  available  1
# rexray volume mount fstest
ID                     Name    Status    Size  Path
vol-07a050137396702f6  fstest  attached  1     /var/lib/libstorage/volumes/fstest/data
# mount | grep fstest
/dev/xvdf on /var/lib/libstorage/volumes/fstest type ext4 (rw,relatime,data=ordered)

Notice that asking for a specific fs had no effect.

Now try it with --amount:

# rexray volume create --size 1 --fsType xfs --amount fstest
ID                     Name    Status    Size  Path
vol-021e6cf5a2e309a63  fstest  attached  1     /var/lib/libstorage/volumes/fstest/data
root@ip-172-31-15-28:/etc/rexray# mount | grep fstest
/dev/xvdf on /var/lib/libstorage/volumes/fstest type xfs (rw,relatime,nouuid,attr2,inode64,noquota)

Note that we correctly got XFS this time.

You can (and when using REX-Ray, might find easier to) specify the fs at mount time:

# rexray volume create --size 1 fstest
ID                     Name    Status     Size
vol-082d55b1ba827677e  fstest  available  1
# rexray volume mount --fsType xfs fstest
ID                     Name    Status    Size  Path
vol-082d55b1ba827677e  fstest  attached  1     /var/lib/libstorage/volumes/fstest/data
# mount | grep fstest
/dev/xvdf on /var/lib/libstorage/volumes/fstest type xfs (rw,relatime,nouuid,attr2,inode64,noquota)

You can even change it (destroying old data!):

# mount | grep fstest
/dev/xvdf on /var/lib/libstorage/volumes/fstest type xfs (rw,relatime,nouuid,attr2,inode64,noquota)
# rexray volume unmount fstest
ID                     Name    Status     Size
vol-082d55b1ba827677e  fstest  available  1
# rexray volume mount --fsType ext4 --overwriteFS fstest
ID                     Name    Status    Size  Path
vol-082d55b1ba827677e  fstest  attached  1     /var/lib/libstorage/volumes/fstest/data
# mount | grep fstest
/dev/xvdf on /var/lib/libstorage/volumes/fstest type ext4 (rw,relatime,data=ordered)

So that's our base functionality, on top of which we can now introduce Docker...

When doing a docker volume create, Docker makes a /VolumeDriver.Create request to our plugin, which in turn goes to our integration driver's create method. The only options it is looking for are availabilityZone, iops, size, type, and encryption flags. There is no opportunity to influence the filesystem at this stage.

When doing a docker run -v .... that uses our driver, that makes a /VolumeDriver.Mount request to our plugin. Our plugin will first check if the volume exists, and if it does not, will (by default) create it implicitly. That's why you can do things like docker run --volume-driver rexray -v data:/data ... and have things work, but today you can not pass options that influence the create call (like size, iops, etc). Not dynamically like this. The only thing you can do is set values in a config file that change the defaults globally. You can't even change these defaults via ENV var because that would be in the docker process space, but won't transfer to the rexray process space.

Unfortunately, Docker does not provide a method to pass volume options when doing docker run. You can also see that in our plugin, we don't look for opts, and always pass a set of empty mount options to the integration driver. So there really is currently no way to, on a per-volume basis, change the fstype.

You could change it globally by setting the following in the config file on the client:

libstorage:
  integration:
    volume:
      operations:
        create:
          default:
            fsType: xfs

But that's not super helpful if you wanted to have a mix of ext4 and xfs.

The best thing I can offer there is to use REX-Ray modules, and create two different services/plugins with different defaults:

rexray:
  modules:
    default-docker:
      type: docker
      desc: The default docker module.
      host: unix:///run/docker/plugins/ebsext4.sock
      libstorage:
        service: ebs
        integration:
          volume:
            operations:
              create:
                default:
                  size: 1
                  fsType: ext4
    xfs-module:
      type: docker
      desc: The second docker module.
      host: unix:///run/docker/plugins/ebsxfs.sock
      libstorage:
        service: ebs
        integration:
          volume:
            operations:
              create:
                default:
                  size: 10
                  fsType: xfs
libstorage:
  service: ebs
ebs:
  accessKey: ...
  secretKey: ...

When REX-Ray starts, this will create two sock files in /var/run/docker/plugins, named ebsxfs and ebsext4, and then these can be accessed as the volume driver names instead of the default rexray.

Example:

# rexray start

# ls -al /var/run/docker/plugins/
total 0
drwx------ 3 root root 100 Jun 15 17:35 .
drwx------ 7 root root 140 Jun 14 16:54 ..
drwx------ 2 root root  60 Jun 14 16:46 0c08bc2d85b3356d7ecc5543a1c125397123495f0dc8e0f9e1cb75029de69ac5
srwxr-xr-x 1 root root   0 Jun 15 17:35 ebsext4.sock
srwxr-xr-x 1 root root   0 Jun 15 17:35 ebsxfs.sock

# docker run --volume-driver ebsext4 -v ext4vol:/data busybox mount | grep \/data
/dev/xvdg on /data type ext4 (rw,relatime,data=ordered)

# docker run --volume-driver ebsxfs -v xfsvol:/data busybox mount | grep \/data
/dev/xvdg on /data type xfs (rw,relatime,nouuid,attr2,inode64,noquota)

That the best path for how REX-Ray functions today. The same method should work with Swarm services, where @fmaturel would instead need to use:

docker service create \
  --name mongo \
  --mount type=volume,src=vol-mongo-xfs,dst=/data/db,volume-driver=ebsxfs,volume-opt=size=8,volume-opt=volumeType=gp2 \
  mongo

Hope this helps!

[codenrhoden]

Everything above is about how things work today. Whether we should make a change, such that you can specify --fsType during create, whereby we mount the disk and format it first, is a separate matter.