Rexray: Unable to mount S3 bucket with Docker volume plugin on CoreOS
Summary
I have rexray/s3fs installed in Docker on CoreOS as a Docker volume plugin. The EC2 instance currently has an IAM role allowing all S3 actions. I'm able to create S3 buckets, but not mount them.
Bug Reports
The rexray/s3fs plugin was installed with the following.
$ docker plugin install --alias rexray/s3fs --grant-all-permissions \
rexray/s3fs:edge S3FS_REGION=us-west-2
$ docker volume create -d rexray/s3fs s3fs-test.transzap.com
Nov 02 20:02:40 ip-10-52-11-82 env[1066]: time="2018-11-02T20:02:40Z" level=error msg="time=\"2018-11-02T20:02:40Z\" level=error msg=\"resource not found\" host=\"unix:///var/run/rexray/701694972.sock\" instanceID=\"s3fs=ip-10-52-11-82\" resourceID=s3fs-test.transzap.com route=volumeInspect server=hot-healer-ci service=s3fs storageDriver=s3fs task=0 time=1541188960620 tls=false txCR=1541188960 txID=bc203d36-e3c6-443e-6fdc-e709abf6ded0 " plugin=745d1c4668b6b4de18029d6f120dfde3ce96409e88d8d78968eb6f9e01973c05
Nov 02 20:02:40 ip-10-52-11-82 env[1066]: time="2018-11-02T20:02:40Z" level=error msg="time=\"2018-11-02T20:02:40Z\" level=error msg=\"error: api call failed\" error.resourceID=s3fs-test.transzap.com host=\"unix:///var/run/rexray/701694972.sock\" route=volumeInspect server=hot-healer-ci time=1541188960621 tls=false txCR=1541188960 txID=bc203d36-e3c6-443e-6fdc-e709abf6ded0 " plugin=745d1c4668b6b4de18029d6f120dfde3ce96409e88d8d78968eb6f9e01973c05
Nov 02 20:02:40 ip-10-52-11-82 env[1066]: time="2018-11-02T20:02:40Z" level=error msg="time=\"2018-11-02T20:02:40Z\" level=error msg=\"docker-legacy: Get: s3fs-test.transzap.com: failed: resource not found\" host=\"unix:///var/run/rexray/701694972.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1541188960621 " plugin=745d1c4668b6b4de18029d6f120dfde3ce96409e88d8d78968eb6f9e01973c05
$ docker container run -it --rm -v s3fs-test.transzap.com:/s3fs alpine:3.8 ash
/run/torcx/bin/docker: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: s3fs-test.transzap.com: failed: error mounting s3fs bucket.
Nov 02 20:04:56 ip-10-52-11-82 env[1066]: time="2018-11-02T20:04:56Z" level=error msg="time=\"2018-11-02T20:04:56Z\" level=error msg=\"docker-legacy: Mount: s3fs-test.transzap.com: failed: error mounting s3fs bucket\" host=\"unix:///var/run/rexray/701694972.sock\" integrationDriver=linux osDriver=linux service=s3fs storageDriver=libstorage time=1541189096416 " plugin=745d1c4668b6b4de18029d6f120dfde3ce96409e88d8d78968eb6f9e01973c05
Nov 02 20:04:56 ip-10-52-11-82 env[1066]: time="2018-11-02T20:04:56.480132534Z" level=error msg="93f79f15805ba36d98e3dc810328649b286fa021db17f238467f49bb7e63fabf cleanup: failed to delete container from containerd: no such container"
Nov 02 20:04:56 ip-10-52-11-82 env[1066]: time="2018-11-02T20:04:56.486523608Z" level=error msg="Handler for POST /v1.38/containers/93f79f15805ba36d98e3dc810328649b286fa021db17f238467f49bb7e63fabf/start returned error: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: s3fs-test.transzap.com: failed: error mounting s3fs bucket"
$ docker plugin ls
ID NAME DESCRIPTION ENABLED
3775e16b3498 rexray/efs:latest REX-Ray for Amazon EFS true
e08a39ea7a14 rexray/ebs:latest REX-Ray for Amazon EBS true
745d1c4668b6 rexray/s3fs:latest REX-Ray FUSE Driver for Amazon Simple Storag… true
$ docker plugin inspect rexray/s3fs
[
{
"Config": {
"Args": {
"Description": "",
"Name": "",
"Settable": null,
"Value": null
},
"Description": "REX-Ray FUSE Driver for Amazon Simple Storage Service (S3FS)",
"DockerVersion": "18.06.1-ce",
"Documentation": "https://github.com/thecodeteam/rexray/.docker/plugins/s3fs",
"Entrypoint": [
"/rexray.sh",
"rexray",
"start",
"-f",
"--nopid"
],
"Env": [
{
"Description": "",
"Name": "REXRAY_LOGLEVEL",
"Settable": [
"value"
],
"Value": "warn"
},
{
"Description": "",
"Name": "LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_ROOTPATH",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LINUX_VOLUME_ROOTPATH",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LINUX_VOLUME_FILEMODE",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "HTTP_PROXY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_ENDPOINT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_ACCESSKEY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_DISABLEPATHSTYLE",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_MAXRETRIES",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_OPTIONS",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_REGION",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_SECRETKEY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "CSI_ENDPOINT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "The name of the CSI plug-in used by the CSI module.",
"Name": "X_CSI_DRIVER",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "A flag that disables the CSI to libStorage bridge.",
"Name": "X_CSI_NATIVE",
"Settable": [
"value"
],
"Value": "false"
}
],
"Interface": {
"Socket": "rexray.sock",
"Types": [
"docker.volumedriver/1.0"
]
},
"IpcHost": false,
"Linux": {
"AllowAllDevices": true,
"Capabilities": [
"CAP_SYS_ADMIN"
],
"Devices": null
},
"Mounts": [
{
"Description": "",
"Destination": "/dev",
"Name": "",
"Options": [
"rbind"
],
"Settable": null,
"Source": "/dev",
"Type": "bind"
}
],
"Network": {
"Type": "host"
},
"PidHost": false,
"PropagatedMount": "/var/lib/rexray",
"User": {},
"WorkDir": "",
"rootfs": {
"diff_ids": [
"sha256:d620848ffd1a5676055decee1cb9e1a95f9dfcfa8b62cf498d3254b4af2ede44"
],
"type": "layers"
}
},
"Enabled": true,
"Id": "745d1c4668b6b4de18029d6f120dfde3ce96409e88d8d78968eb6f9e01973c05",
"Name": "rexray/s3fs:latest",
"PluginReference": "docker.io/rexray/s3fs:edge",
"Settings": {
"Args": [],
"Devices": [],
"Env": [
"REXRAY_LOGLEVEL=warn",
"LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_ROOTPATH=",
"LINUX_VOLUME_ROOTPATH=",
"LINUX_VOLUME_FILEMODE=",
"HTTP_PROXY=",
"S3FS_ENDPOINT=",
"S3FS_ACCESSKEY=",
"S3FS_DISABLEPATHSTYLE=",
"S3FS_MAXRETRIES=",
"S3FS_OPTIONS=",
"S3FS_REGION=us-west-2",
"S3FS_SECRETKEY=",
"CSI_ENDPOINT=",
"X_CSI_DRIVER=",
"X_CSI_NATIVE=false"
],
"Mounts": [
{
"Description": "",
"Destination": "/dev",
"Name": "",
"Options": [
"rbind"
],
"Settable": null,
"Source": "/dev",
"Type": "bind"
}
]
}
}
]
Expected Behavior
REX-Ray should mount the specified S3 bucket.
Actual Behavior
An error occurs mounting the bucket.
core@ip-10-52-11-82 ~ $ docker container run -it --rm -v s3fs-test.transzap.com:/s3fs alpine:3.8 ash
/run/torcx/bin/docker: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: s3fs-test.transzap.com: failed: error mounting s3fs bucket.
Logs
Debug level logs when mounting bucket.
https://gist.github.com/kinghuang/7bee5aa11e14c433c86fe83bc8ba5136
kinghuang
All 3 comments
I've entered the plugin container in an attempt to debug this.
sudo docker-runc --root /var/run/docker/plugins/runtime-root/plugins.moby exec -t 5e358de… ash
Using the rexray CLI, I can list the available S3 buckets and mount them.
/ # rexray volume ls
ID Name Status Size
rexray-test.transzap.com rexray-test.transzap.com attached 0
/ # rexray volume mount rexray-test.transzap.com
ID Name Status Size Path
rexray-test.transzap.com rexray-test.transzap.com attached 0 /var/lib/rexray/volumes/rexray-test.transzap.com/data
/ # cd /var/lib/rexray/volumes/rexray-test.transzap.com/data
/var/lib/rexray/volumes/rexray-test.transzap.com/data # ls
/var/lib/rexray/volumes/rexray-test.transzap.com/data #
But, the mounted volume is empty. I put some a couple of test objects in the bucket using a S3 client.
If I unmount the bucket, the directory gets left behind, causing errors on subsequent mounts.
/ # rexray volume unmount rexray-test.transzap.com
ID Name Status Size
rexray-test.transzap.com rexray-test.transzap.com available 0
/ # ls -al /var/lib/rexray/volumes
total 24
drwxr-xr-x 3 root root 4096 Nov 3 22:18 .
drwxr-xr-x 3 root root 4096 Nov 3 22:13 ..
drwxr-xr-x 3 root root 4096 Nov 3 22:18 rexray-test.transzap.com
/ # rexray volume mount rexray-test.transzap.com
ID Name Status Size Path
FATA[0000] error mounting volume error.isAWSAuthEnvVars=true error.output="s3fs: MOUNTPOINT directory /var/lib/rexray/volumes/rexray-test.transzap.com is not empty. if you are sure this is safe, can use the 'nonempty' mount option.\n" error.inner.Stderr=[] error.bucket=rexray-test.transzap.com error.mountPoint="/var/lib/rexray/volumes/rexray-test.transzap.com" error.cmd=s3fs error.args=[rexray-test.transzap.com /var/lib/rexray/volumes/rexray-test.transzap.com] volume=rexray-test.transzap.com
I'm not sure how to resolve these problems. It seems like the mounted S3 buckets aren't connected to the bucket contents. And, unmounting leaves directories behind.
kinghuang
on 3 Nov 2018
Explicitly setting the URL to the S3 endpoint in my region seems to be the key in my case.
S3FS_OPTIONS="url=https://s3-us-west-2.amazonaws.com"
I also have buckets with dots in the name (i.e., domain names). The full set of options I'm now specifying on the rexray/s3fs driver is:
S3FS_OPTIONS="allow_other,nonempty,use_path_request_style,url=https://s3-us-west-2.amazonaws.com"
👍4
@kinghuang - this was my answer after a few days of debugging. Thanks.
Can anyone elaborate on why explicitly setting the url is needed? Why wouldn't this be picked up by rexray's S3FS_REGION variable?
jedateach
on 16 May 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
gonzalomarcote
·
10Comments
dl00
·
6Comments
frank-montyne
·
8Comments
derfloh
·
6Comments
clintkitson
·
9Comments
Most helpful comment
Explicitly setting the URL to the S3 endpoint in my region seems to be the key in my case.
I also have buckets with dots in the name (i.e., domain names). The full set of options I'm now specifying on the rexray/s3fs driver is: