Rexray: [EBS VOLUME TYPE CONFIGURATION] - How to specify the type of EBS volume type?

[anirbanroydas]

Summary

I have a docker swarm setup in AWS in a multi-AZ setup, I am using rex-ray to create persistent storage for docker containers via docker service create command. Now back in my aws ec2 web console, I see the volume is being created but the type of volume created is standard. So what is the configuration that I have to add to change the volume type of my choosing, gp2, iops, etc.

TL;DR
Everything works fine, expected behavior and actual behavior are same, although the logs show some errors.
So Basically I have 2 questions:

1. First one is regarding the EBS Volume Type that is getting created. How to define my own type?
2. Since everything works fine, why do I still have those error logs?

If you need to know the entire workflow, if it may help, I have furnished more and verbose info below.

MORE INFO
I have 5 node cluster -
3 Managers (nightswatch-01, lannisters-02, starks-03),
2 Workers (johnsnow-04, tyrion-05).
Region - ap-south-1
AZ - ap-south-1a and ap-sount-1b
2 Manager in AZ a : (nightswatch-01, starks-03),
1 Manager in AZ b : (starks-03).
1 Worker in AZ a : (johnsnow-04),
1 Worker in AZ b : (tyrion-05).

NOTE : The above info is not needed, just to make it easy if required.

Version

Please paste the output of rexray version. For example:

$ rexray version
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.7.0
OsArch: Linux-x86_64
Branch: v0.7.0
Commit: a20a838ca70838a914b632637398824fcb10d0db
Formed: Mon, 23 Jan 2017 15:14:32 UTC

libStorage
----------
SemVer: 0.4.0
OsArch: Linux-x86_64
Branch: v0.7.0
Commit: a1103d3f215117f7b9f51dae2b24f852c9c54995
Formed: Mon, 23 Jan 2017 15:14:12 UTC

NOTE: Additionally I am also giving my docker version report.

$ docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:50:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:50:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

Expected Behavior

1. After I create a service with mount as volume with volume-driver as rexray with a constraint on the service to restrict the task scheduling to a particular AZ (since cross AZ EBS support not possible)
2. Able to add data to the redis service in one node, shutting the node down, so as the auto scheduling of task(container) to a different node satisfying the AZ constraint.
3. Able to enter the container to check the data persisted.
4. Also to be able to check the volume appearing in the aws ec2 web console with attached instance, first to be the original node instance-id, and after its shut down, then re-attachment to the new node instance (belonging to the same AZ)

Actual Behavior

All of the above expected behaviours are satisfied.

Screenshots to show the steps.

1. Creation of service from a manager node in AZ a (nightswatch-01). The service gets scheduled according to the constraint to one node in AZ b. In this case gets scheduled to (lannisters-02)
   

2. Data added successfully, no screenshot for this. Node Shut Down to mimic node failure, Below Screenshot shows Node (lannisters-02) becomes Down and Manager status as Unreachable
   

Also the task(container) gets rescheduled to a new node with satisfying constraint of AZ b

1. Able to enter the container to check if data persisted in the volume.
   

2. Able to see the volume creation. Attached instances not shown, but it worked perfectly.
   

NOTE this is my main question. The above volume does get created and everything works as expected but how to change the volume type (or is the standard the only possible type)

Steps To Reproduce

Please list the steps to reproduce the issue in this section.

NOTE Debug mode enabled.

1. Create Swarm of 5 nodes according the initial info. Add labels to node so as to use those labels during service creation to put constraint to service scheduling to particular AZ. Install rex-ray via the curl comand to all nodes, add config.yml to all nodes in required directory, and restart rexray service in all nodes.

2. Step 1: Create Service from any manager node. I have put the constraint to schedule task to a node in AZ b

$ docker service create --name redis --mount type=volume,source=redis_data,target=/data,volume-driver=rexray --constraint 'node.labels.io.aroyd.swarm.cluster.node.aws.az == b' redis:alpine redis-server --appendonly yes

1. Find out where the service task got scheduled and note down the node name.

$ docker service ps redis

1. SSH in to the scheduled node and enter into the container and add data to redis

$ docker exec -it $CONTAINER_ID sh
# redis-cli
>set foo bar
>lpush mylist a b 123 456
>exit
#^P^Q

1. Shut down above node to mimic node failure

$ docker-machine stop <node_name_of_the_service_scheduled>

1. Check current node where task got scheduled from any present manager node

$ docker service ps redis

7 Go to the new node, enter into the container and check if data persisted

$ docker exec -it $CONTAINER_ID sh
# redis-cli
>get foo
bar
>lrange mylist 0 -1
> <result>
> exit
# ^P^Q

Configuration Files

Please paste any related configuration files, such as /etc/rexray/config.yml
in this section. Please use the appropriate formatting when pasting YAML.
content. For example:

NOTE I used the rex-ray configuration generator to generate this config.

rexray:
  logLevel: debug
libstorage:
  logging:
    level: debug
    httpRequests:  true
    httpResponses: true
  service: ebs
  integration:
    volume:
      operations:
        create:
          default:
            size: 8
        mount:
          preempt: true
        unmount:
          ignoreUsedCount: false
        path:
          cache:
            enabled: true
            async: true
ebs:
  accessKey: <my_access_key>
  secretKey: <my_secret_key>
  region: ap-south-1

Logs

It is very important when filing an issue to include associated logs. There are
two different logs about which to be concerned: the service log (if REX-Ray is
running as a service) and the client log.

Service Log

NOTE Adding logs here since its not that verbose, just a few lines.

1. Logs on the first node where the service gets scheduled first. In my case (lannisters-02):

No error logs here.

1. After rescheduling of service to a new node, in my case (tyrion-05) according to the constraint of AZ b. The logs on this node are:

SemVer: 0.7.0                                                                                                                                                                 [0/141]
OsArch: Linux-x86_64
Branch: v0.7.0
Commit: a20a838ca70838a914b632637398824fcb10d0db
Formed: Mon, 23 Jan 2017 15:14:32 UTC

libStorage
----------
SemVer: 0.4.0
OsArch: Linux-x86_64
Branch: v0.7.0
Commit: a1103d3f215117f7b9f51dae2b24f852c9c54995
Formed: Mon, 23 Jan 2017 15:14:12 UTC

time="2017-02-16T09:41:28Z" level=error msg="resource not found" host="unix:///var/run/libstorage/711720440.sock" instanceID="ebs=i-<tyrion-0f_instance_id>,availabilityZone=ap-south-1b&region=ap-south-1" resourceID=vol-<redis_data_volume_id> route=volumeInspect server=jet-singer-gb service=ebs storageDriver=ebs task=0 time=1487238088942 tls=false txCR=1487238088 txID=1d5a9aa0-bd1b-42cd-44c0-673575353773
time="2017-02-16T09:41:28Z" level=error msg="resource not found" host="unix:///var/run/libstorage/711720440.sock" resourceID=vol-<redis_data_volume_id> route=volumeInspect server=jet-singer-gb time=1487238088942 tls=false txCR=1487238088 txID=1d5a9aa0-bd1b-42cd-44c0-673575353773
time="2017-02-16T09:41:28Z" level=error msg="/VolumeDriver.Path: error returning path" error.inner.resourceID=vol-<redis_data_volume_id> error.status=404 host="unix:///var/run/libstorage
/711720440.sock" time=1487238088944

[kacole2]

@anirbanroydas you need to pass the --opt flag and pre-create the volume before deploying the container

this will show you all the options you can use and would be the best way for re-creation

rexray volume create --help

      --amount[=false]: Attach and mount the new volume
      --attach[=false]: Attach the new volume
      --availabilityZone="":
      --continueOnError[=false]: Continue processing a collection upon error
      --encrypted[=false]: A flag that requests the storage platform create an encrypted volume. Specifying true doesn't guarantee encryption; it's up the storage driver and platform to implement this feature.
      --encryptionKey="": The key used to encrypt the volume.
      --force[=false]:
      --fsType="":
      --iops=0:
      --overwriteFS[=false]:
      --size=0:
      --snapshotID="":
      --volumeType="":

sudo rexray volume create testvol01 --size=20 --volumeType=gp2
sudo rexray volume create testvol02 --size=100 --volumeType=io1 --iops=2500

or you can use docker volume cli and pass the opt flag:

sudo docker volume create -d rexray --name testvol03 --opt=size=20 --opt=volumeType=gp2

[anirbanroydas]

@kacole2 Thanks. Thats the solution for the volume type problem.

Can you also shed some light on the error logs?

Also, I would request to do some refactoring of the docs. Its a little overwhelming when you first try to look at it. Plus you have to refer the libstorage docs at the same time. And the fact the all the options and stuffs are not mentioned clearly. It would be great if the docs get easier to grasp. Right now there are a lot of things which are not clear at first go.

[akutz]

Hi @anirbanroydas,

I just wanted to step in and comment on how amazed and impressed I am by your attention to detail and willingness to adhere to the issue template's requested guidelines. Thank you so much. This type of information truly does make it easier for us to help you. Thank you again.

[anirbanroydas]

@akutz Thanks. Its indeed is a good practice when creating new issues.

[kacole2]

@anirbanroydas sorry for being a little late to the reply here. you've made @akutz very happy :)

as for the docs, yes. they are intimidating and overwhelming when first looking at them. We are currently underway of adding in new sections, as well a quick "get started" splash page that should ease a lot of the burden.

the service logs...

time="2017-02-16T09:41:28Z" level=error msg="resource not found" host="unix:///var/run/libstorage/711720440.sock" instanceID="ebs=i-<tyrion-0f_instance_id>,availabilityZone=ap-south-1b&region=ap-south-1" resourceID=vol-<redis_data_volume_id> route=volumeInspect server=jet-singer-gb service=ebs storageDriver=ebs task=0 time=1487238088942 tls=false txCR=1487238088 txID=1d5a9aa0-bd1b-42cd-44c0-673575353773
time="2017-02-16T09:41:28Z" level=error msg="resource not found" host="unix:///var/run/libstorage/711720440.sock" resourceID=vol-<redis_data_volume_id> route=volumeInspect server=jet-singer-gb time=1487238088942 tls=false txCR=1487238088 txID=1d5a9aa0-bd1b-42cd-44c0-673575353773
time="2017-02-16T09:41:28Z" level=error msg="/VolumeDriver.Path: error returning path" error.inner.resourceID=vol-<redis_data_volume_id> error.status=404 host="unix:///var/run/libstorage
/711720440.sock" time=1487238088944

it's saying that it can't find vol-<redis_data_volume_id>. This is probably because of a scripting error. I don't believe < or > are valid usable characters. I've never tested this myself though.

I can see the preempt is set to true, which will take care of the failover in a single AZ. You can probably clean up your config.yml a bit by using http://rexrayconfig.codedellemc.com/.

[akutz]

Hi @kacole2,

His config file is actually fairly concise for what he's trying to do, but I do recommend your generator. Just probably not for this case.

[anirbanroydas]

@kacole2 Actually, I have used the generator itself (I mentioned it as a NOTE in the issue template). And those <, > are not what I have used. I generally avoid any leaking of personal info via logs when submitting issues. So I just replace them with humanly-names. And for what is worth, the actual vol-ids were the same as what rex-ray created and which appeared on the ec2 web console. Thats why it feels strange, since everything worked fine and still those weird error logs.

Also, what I did with the config was, I added those path cache and ignore keys via the generator just to have a fair idea of what its doing and later made the flags as true/false accordingly.

@akutz What additional configuration should I add? The docs, as I mentioned is intimidating and took me a while to get around things and hence I cheated and used the configuration generator at first place to make things easier. Let me know where to go next.

[anirbanroydas]

@kacole2 Hey, any follow ups on the above?

[kacole2]

@anirbanroydas better question... have you had any more errors? I'll have to defer on to others about the logs because from what I can tell is that it's looking for a volume that doesn't exist.

[codenrhoden]

@anirbanroydas I'm looking through old issues, and wanted to check if you were still having any issues along these lines, or if this can be closed.

Thanks.