Rexray: Can't start containers with Rexray 0.3.3 / EBS / Docker 1.11

[derfloh]

Trying to start a container with the combination above results in a error looking like "docker: Error response from daemon: Container command 'sh' not found or does not exist“"

1) Starting on a fresh Amazon Linux EC2 (amzn-ami-hvm-2016.03.3.x86_64-gp2 )
2) yum install docker (-> yields version 1.11.2) && /etc/init.d/docker start
3) curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -s stable 0.3.3
4) putting the following config.yml into /etc/rexray

rexray:
  storageDrivers:
  - ec2

(the ec2 instance has a role with EC2FullAccess, so no keys/secrets here)
5) /etc/init.d/rexray start
6) docker volume create --driver rexray --opt size=10 --name hellopersistence
7) docker run -ti --rm --volume-driver=rexray -v hellopersistence:/mystore busybox ls
docker: Error response from daemon: Container command 'ls' not found or does not exist..

Additional Info:
from /var/log/docker:
time="2016-07-19T14:53:21Z" level=error msg="containerd: start container" error="oci runtime error: could not synchronise with container process: stat /var/lib/rexray/volumes/hellopersistence/data: no such file or directory" id=3aa8643fc30c8e6ada5fbf20f48c7a29c010cbfe997cd9277013b339fc671c0a
But when mounting the volume manually with "rexray volume mount --volumename hellopersistence" the directory mentioned above does exist.

It seems like a timing issue.. I’ve tried the following:
1) hellopersistence volume is _not_ mounted
2) create a /var/lib/rexray/volumes/hellopersistence/data directory (which is now on the local disk)
3) Now this works: "docker run -ti --rm --volume-driver=rexray -v hellopersistence:/mystore busy box ls"
4) Start a busy box container "docker run -ti --rm --volume-driver=rexray -v hellopersistence:/mystore busybox"
5) Do a "touch /mystore/hello" in the container
6) Outside of the container: ls /var/lib/rexray/volumes/hellopersistence/data/ yields empty directory
7) Stop busybox container & hellopersistence volume gets unmounted
8) ls /var/lib/rexray/volumes/hellopersistence/data/ yields "hello"

From my point of view, this looks like docker mounted the volume at a stage where rexray hasn't mounted the volume yet. So in the container every IO operation goes to the local disk "data" directory. On the host system the /var/lib/rexray/volumes/hellopersistence/data does reflect the correct state, so it reads stuff from EBS

But it seems to work with docker v1.10: Since there’s no 1.10 in the amazon repository, I’ve just stopped the v1.11 daemon, and started a 1.10 docker daemon from binaries / used the docker 1.10 cli for the commands above on the same host… and everything’s fine :/

[kacole2]

@derfloh we specifically haven't called out support for Amazon Linux in our docs. Can you try using Ubunutu or CentOS?

I'll dive a bit deeper and see if there is something inhibiting with Amazon Linux

[derfloh]

Seems like you had a reason to not officially support Amazon Linux ;)
I did the same tests with

• "Ubuntu 14.04.4 LTS"
• "CentOS Linux release 7.2.1511 (Core) "

both with Docker v1.11.2 and rexray 0.3.3
And it works on both systems :/ Still wondering what's going on with Amazon Linux _hm_

[kacole2]

@derfloh honestly, I wasn't aware of an issue with Amazon Linux but the thought process was if we supported all the major distros then we know it will work on-prem and in any cloud. Amazon Linux is really a corner case for us.

If your issue is solved can you please close it out.

If you would like to see Amazon Linux supported please open a new issue under the libStorage project

[derfloh]

I wanted to do a little bit more investigating before closing, so here's some more information:

Rexray seems to run fine on Amazon Linux, even with the docker version that is shipped by amazon.... _BUT_ not if the docker daemon is started via "/etc/init.d/docker start", if it's started on the command line via "docker daemon ..." it runs as expected :) Still trying to find where's the important little difference..

[derfloh]

Final solution/reason for this bug is that the initscript for docker from the amazon repository is starting docking with an "unshare -m" (From Manpage: "Mounting and unmounting filesystems will not affect the rest of the system ....")

Modifying /etc/init.d/docker from
"$unshare" -m -- nohup $exec daemon ${OPTIONS} ${DOCKER_STORAGE_OPTIONS} &>> $logfile &
to
nohup $exec daemon ${OPTIONS} ${DOCKER_STORAGE_OPTIONS} &>> $logfile &
resolved it.

[kacole2]

Nice @derfloh! Glad you found the culprit. Will note this down and see if it's worth officially supporting in the future.