Rexray: Error: rexray/s3fs as docker plugin, can not find the right rexray.socket

[neoitec]

Summary

I was trying to mount a postgresql database volume to S3 wasabi storage by using rexray/s3fs

I can install the plugin using following instruction without any problem.

docker plugin install rexray/s3fs LINUX_VOLUME_FILEMODE=0777 REXRAY_LOGLEVEL=debug S3FS_MAXRETRIES=10 S3FS_ACCESSKEY=access_key S3FS_SECRETKEY=access_secret S3FS_ENDPOINT=https://s3.wasabisys.com S3FS_OPTIONS=url=https://s3.wasabisys.com,use_path_request_style,allow_other,nonempty

ID NAME DESCRIPTION ENABLED
4e68aa2b2996 rexray/s3fs:latest REX-Ray FUSE Driver for Amazon Simple Storag… true

and after if I do docker volume ls. every buckets in S3.wasabi show as expected.

DRIVER VOLUME NAME
rexray/s3fs:latest pg-data

so far everything is fine. then I try to mount a container - postgresql with it in my docker swarm.

docker service create \
--mount type=volume,source=pg-data,destination=/var/lib/postgresql/data,volume-driver=rexray/s3fs
...

then this error happen happens.

"Post http://%2Frun%2Fdocker%2Fplugins%2Fd5a23b54fa533d9d4199bb733857b882c8cee6ab1c3386b2941f42777fd1d0b0%2Frexray.sock/VolumeDriver.Mount: dial unix /run/docker/plugins/d5a23b54fa533d9d4199bb733857b882c8cee6ab1c3386b2941f42777fd1d0b0/rexray.sock: connect: no such file or directory"

It was obviously that the rexray.sock could not be found. So I went to /var/docker/plugins. I see this.

root@usa-main:/var/lib/docker/plugins# ls
4e68aa2b2996f5f75f2f6d5f4b8556bbf7a819e3abc6a9016617b8368eab3de5 storage tmp

"4e68aa2b2996f5f75f2f6d5f4b8556bbf7a819e3abc6a9016617b8368eab3de5" is the plugin id and this folder has the rexray.sock in it. So the rexray plugin is searching the wrong location ("d5a23b54fa533d9d4199bb733857b882c8cee6ab1c3386b2941f42777fd1d0b0") for the socket. I don't know if this is a bug.

I will try with with rexray standalone version to see if this error reproduces.

Version

[
{
"Config": {
"Args": {
"Description": "",
"Name": "",
"Settable": null,
"Value": null
},
"Description": "REX-Ray FUSE Driver for Amazon Simple Storage Service (S3FS)",
"DockerVersion": "18.02.0-ce",
"Documentation": "https://github.com/thecodeteam/rexray/.docker/plugins/s3fs",
"Entrypoint": [
"/rexray.sh",
"rexray",
"start",
"-f",
"--nopid"
],
"Env": [
{
"Description": "",
"Name": "REXRAY_LOGLEVEL",
"Settable": [
"value"
],
"Value": "warn"
},
{
"Description": "",
"Name": "LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_ROOTPATH",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LINUX_VOLUME_ROOTPATH",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LINUX_VOLUME_FILEMODE",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "HTTP_PROXY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_ENDPOINT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_ACCESSKEY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_DISABLEPATHSTYLE",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_MAXRETRIES",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_OPTIONS",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_REGION",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "S3FS_SECRETKEY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "CSI_ENDPOINT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "The name of the CSI plug-in used by the CSI module.",
"Name": "X_CSI_DRIVER",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "A flag that disables the CSI to libStorage bridge.",
"Name": "X_CSI_NATIVE",
"Settable": [
"value"
],
"Value": "false"
}
],
"Interface": {
"Socket": "rexray.sock",
"Types": [
"docker.volumedriver/1.0"
]
},
"IpcHost": false,
"Linux": {
"AllowAllDevices": true,
"Capabilities": [
"CAP_SYS_ADMIN"
],
"Devices": null
},
"Mounts": [
{
"Description": "",
"Destination": "/dev",
"Name": "",
"Options": [
"rbind"
],
"Settable": null,
"Source": "/dev",
"Type": "bind"
}
],
"Network": {
"Type": "host"
},
"PidHost": false,
"PropagatedMount": "/var/lib/rexray",
"User": {},
"WorkDir": "",
"rootfs": {
"diff_ids": [
"sha256:eebe6ffed8f4c85b884ec7c227cff7ed9a9c189f8d28949aa20c395dd121869f"
],
"type": "layers"
}
},
"Enabled": true,
"Id": "4e68aa2b2996f5f75f2f6d5f4b8556bbf7a819e3abc6a9016617b8368eab3de5",
"Name": "rexray/s3fs:latest",
"PluginReference": "docker.io/rexray/s3fs:latest",
"Settings": {
"Args": [],
"Devices": [],
"Env": [
"REXRAY_LOGLEVEL=debug",
"LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_ROOTPATH=",
"LINUX_VOLUME_ROOTPATH=",
"LINUX_VOLUME_FILEMODE=0777",
"HTTP_PROXY=",
"S3FS_ENDPOINT=https://s3.wasabisys.com",
"S3FS_ACCESSKEY=",
"S3FS_DISABLEPATHSTYLE=",
"S3FS_MAXRETRIES=10",
"S3FS_OPTIONS=url=https://s3.wasabisys.com,use_path_request_style,allow_other,nonempty",
"S3FS_REGION=",
"S3FS_SECRETKEY=",
"CSI_ENDPOINT=",
"X_CSI_DRIVER=",
"X_CSI_NATIVE=false"
],
"Mounts": [
{
"Description": "",
"Destination": "/dev",
"Name": "",
"Options": [
"rbind"
],
"Settable": null,
"Source": "/dev",
"Type": "bind"
}
]
}
}
]

Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:10:01 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:08:31 2018
OS/Arch: linux/amd64
Experimental: false

[moodymob]

Same issue , just after an upgrade from "docker-ce :17.12.1~ce-0~debian" to "18.03.0~ce-0~debian".
with rexray/rbd:latest :

/var/lib/docker# docker plugin install --grant-all-permissions rexray/rbd
latest: Pulling from rexray/rbd
20d5e3d9f105: Download complete
Digest: sha256:4ea5ae92c7bfec0d7f7b227deddf6520019b9fb764663fd24782ba56bc9d4052
Status: Downloaded newer image for rexray/rbd:latest
Error response from daemon: dial unix /run/docker/plugins/6ccfd5a3d01d6327bb8c83c08bb0437478cb345cc38494f6f28463a53cd3f706/rexray.sock: connect: no such file or director

[mmraz]

This issue has occurred for me when a volume was created with the rexray/s3fs plugin then the plugin was reinstalled to either upgrade the version or change install configuration environment variables. The newer plugin has a different socket path than the one associated with the volume.

The only way I have been able to correct the problem is to archive the contents of the bucket and run a 'docker volume rm' on the broken volume. Then I recreate the volume using the rexray/s3fs driver and repopulate the contents of the bucket. Mounting it is then successful.

[dcvan24]

I had the same issue and restarting Docker daemon resolved my problem.

[kladiv]

+1
Same issue on first installation on 18.03

[frjaraur]

I think this issue is related to https protocol. I have same problem when self signing my Minio S3 backend and exporting service as https. Without TLS certificates, plugin install worked without any problem, but when install the plugin with same endpoint but with TLS, It doesn't work I get issue with rexray socket.
I tryied Rexray S3FS service without plugin install, configuring my S3 service on /etc/rexray/config.yml and it worked (of course after trusting my self signed certificates).
So I think that the issue is with HTTPS endpoints (in my case with self signed certificates, with docker engine trusting my backend certificates).

Thanks,
Javier R.

[jamshid]

Just curious have you found s3fs works reliably for database files? https://github.com/s3fs-fuse/s3fs-fuse#limitations lists eventual consistency and random writes as limitations so I'm surprised rexray/s3fs works for database volumes. Maybe it doesn't do any live sync'ing?

[kolbma]

For me this error has been a bucket/permission problem.
In the sys journal I could read that authentication failed in libstorage errors.
After enabling REXRAY_LOGLEVEL=debug during plugin install, I saw that I had specified a non-existing bucket in the urls.
But same problem could also be if the user is not allowed for ListBucket or ListAllMyBuckets action.