rexray/ebs on AWS and docker swarm, problem with device discovery
Summary
When creating a container with an EBS volume docker returns with an error "problem with device discovery".
Bug Reports
Version
rexray/ebs:latest
Expected Behavior
- EBS Volume Created in AWS
- Volume attaches to instance.
- Docker mounts volume into container.
- Container starts.
Actual Behavior
- EBS Volume Created in AWS
- Volume attaches to instance
- Docker errors out with error
docker: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: test: failed: problem with device discovery.
Steps To Reproduce
- Create AWS instances using latest Amazon Linux AMI
- Install docker
- Set up swarm
- Install rexray/ebs following guide on each instance
- Create test container using rexray/ebs
See below for a full console output. Interestingly, in my troubleshooting the volume will not delete either until I manually detach it from the instance in AWS.
[ec2-user@xxxxxxx]$ sudo docker plugin install rexray/ebs REXRAY_PREEMPT=true EBS_REGION=ap-southeast-2 EBS_ACCESSKEY=AKIAJJEZLCOGWWU2S26Q EBS_SECRETKEY=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
Plugin "rexray/ebs" is requesting the following privileges:
- network: [host]
- mount: [/dev]
- allow-all-devices: [true]
- capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from rexray/ebs
d726edfb6c8d: Download complete
Digest: sha256:b5b0680b6dd53a4c61bac21942738a69edfe3d6a6516acc13045d257889c9213
Status: Downloaded newer image for rexray/ebs:latest
Installed plugin rexray/ebs
[ec2-user@xxxxxxx]$ docker run -ti --volume-driver=rexray/ebs -v test:/test busybox
docker: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: docker-legacy: Mount: test: failed: problem with device discovery.
[ec2-user@xxxxxxx]$ docker run -ti --volume-driver=rexray/ebs -v test:/test busybox
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/1b5875fe11fd6f46eebb647c4b7986039a7d2955f341f69561e0080ae9f2fa80/rootfs': VolumeDriver.Mount: docker-legacy: Mount: test: failed: no device name returned.
[ec2-user@xxxxxxx]$ docker volume inspect test
[
{
"CreatedAt": "0001-01-01T00:00:00Z",
"Driver": "rexray/ebs:latest",
"Labels": null,
"Mountpoint": "",
"Name": "test",
"Options": null,
"Scope": "global",
"Status": {
"availabilityZone": "ap-southeast-2c",
"fields": null,
"iops": 0,
"name": "test",
"server": "ebs",
"service": "ebs",
"size": 16,
"type": "standard"
}
}
]
[ec2-user@xxxxxxx]$ docker volume rm test
Error response from daemon: remove test: volume is in use - [314b5a64875451191fb9dd52bca19d073a858a567d244a733e222f5c7763c208, c319647db5a63e139be2c0a1481054765a730aa025c8955d6d7194e28a3d2d88]
[ec2-user@xxxxxxx]$ docker container rm 314b5a64875451191fb9dd52bca19d073a858a567d244a733e222f5c7763c208 c319647db5a63e139be2c0a1481054765a730aa025c8955d6d7194e28a3d2d88
314b5a64875451191fb9dd52bca19d073a858a567d244a733e222f5c7763c208
c319647db5a63e139be2c0a1481054765a730aa025c8955d6d7194e28a3d2d88
[ec2-user@xxxxxxx]$ docker volume rm test
Error response from daemon: remove test: VolumeDriver.Remove: docker-legacy: Remove: test: failed: error deleting volume
here's where I detach the volume in the EC2 console
[ec2-user@xxxxxxx]$ docker volume rm test
test
Configuration Files
No applicable, using the docker plugin. But here's the output of docker plugin inspect
[
{
"Config": {
"Args": {
"Description": "",
"Name": "",
"Settable": null,
"Value": null
},
"Description": "REX-Ray for Amazon EBS",
"DockerVersion": "18.05.0-ce",
"Documentation": "https://github.com/thecodeteam/rexray/.docker/plugins/ebs",
"Entrypoint": [
"/rexray.sh",
"rexray",
"start",
"-f",
"--nopid"
],
"Env": [
{
"Description": "",
"Name": "DOCKER_LEGACY",
"Settable": [
"value"
],
"Value": "true"
},
{
"Description": "",
"Name": "REXRAY_FSTYPE",
"Settable": [
"value"
],
"Value": "ext4"
},
{
"Description": "",
"Name": "REXRAY_LOGLEVEL",
"Settable": [
"value"
],
"Value": "warn"
},
{
"Description": "",
"Name": "REXRAY_PREEMPT",
"Settable": [
"value"
],
"Value": "false"
},
{
"Description": "",
"Name": "HTTP_PROXY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_ROOTPATH",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LINUX_VOLUME_ROOTPATH",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "LINUX_VOLUME_FILEMODE",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_ACCESSKEY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_ENDPOINT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_KMSKEYID",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_MAXRETRIES",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_REGION",
"Settable": [
"value"
],
"Value": "us-east-1"
},
{
"Description": "",
"Name": "EBS_SECRETKEY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_STATUSINITIALDELAY",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_STATUSMAXATTEMPTS",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_STATUSTIMEOUT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "EBS_USELARGEDEVICERANGE",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "",
"Name": "CSI_ENDPOINT",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "The name of the CSI plug-in used by the CSI module.",
"Name": "X_CSI_DRIVER",
"Settable": [
"value"
],
"Value": ""
},
{
"Description": "A flag that disables the CSI to libStorage bridge.",
"Name": "X_CSI_NATIVE",
"Settable": [
"value"
],
"Value": ""
}
],
"Interface": {
"Socket": "rexray.sock",
"Types": [
"docker.volumedriver/1.0"
]
},
"IpcHost": false,
"Linux": {
"AllowAllDevices": true,
"Capabilities": [
"CAP_SYS_ADMIN"
],
"Devices": null
},
"Mounts": [
{
"Description": "",
"Destination": "/dev",
"Name": "",
"Options": [
"rbind"
],
"Settable": null,
"Source": "/dev",
"Type": "bind"
}
],
"Network": {
"Type": "host"
},
"PidHost": false,
"PropagatedMount": "/var/lib/rexray",
"User": {},
"WorkDir": "",
"rootfs": {
"diff_ids": [
"sha256:3e6c5310eb0c1fcbe49312bae86e363bf7ef7be4dd9e0dcd2206200b265a0693"
],
"type": "layers"
}
},
"Enabled": true,
"Id": "1b5875fe11fd6f46eebb647c4b7986039a7d2955f341f69561e0080ae9f2fa80",
"Name": "rexray/ebs:latest",
"PluginReference": "docker.io/rexray/ebs:latest",
"Settings": {
"Args": [],
"Devices": [],
"Env": [
"DOCKER_LEGACY=true",
"REXRAY_FSTYPE=ext4",
"REXRAY_LOGLEVEL=warn",
"REXRAY_PREEMPT=true",
"HTTP_PROXY=",
"LIBSTORAGE_INTEGRATION_VOLUME_OPERATIONS_MOUNT_ROOTPATH=",
"LINUX_VOLUME_ROOTPATH=",
"LINUX_VOLUME_FILEMODE=",
"EBS_ACCESSKEY=ABC",
"EBS_ENDPOINT=",
"EBS_KMSKEYID=",
"EBS_MAXRETRIES=",
"EBS_REGION=ap-southeast-2",
"EBS_SECRETKEY=123",
"EBS_STATUSINITIALDELAY=",
"EBS_STATUSMAXATTEMPTS=",
"EBS_STATUSTIMEOUT=",
"EBS_USELARGEDEVICERANGE=",
"CSI_ENDPOINT=",
"X_CSI_DRIVER=",
"X_CSI_NATIVE="
],
"Mounts": [
{
"Description": "",
"Destination": "/dev",
"Name": "",
"Options": [
"rbind"
],
"Settable": null,
"Source": "/dev",
"Type": "bind"
}
]
}
}
]
Logs
I'd love to provide this, but I can't for the life of me find any logs output by the docker plugin, happy to run commands though.
Edit: Formatting
voney
👍5
All 7 comments
I'm having the exact same problem as @voney , but using EBS (no swarm).
@voney, when installing rexray as a docker plugin, the logs will appear on /var/log/docker.
Maybe my logs could help? https://gist.github.com/fabiob/a318bc24b6e96f9ae34c28af53409f61
fabiob
on 16 Dec 2018
I too have this issue with 0.11.3. I'm starting to wonder if anyone is successfully using the EBS plugin on a swarm deployment...
brewsteropsdev
on 20 Dec 2018
👍3
I've fixed the problem by switching out of the t3.* instances. Looks like their storage drivers aren't compatible with rexray's.
fabiob
on 22 Dec 2018
👍1
Another me too.
Switching from M5 to M4 instances solved this for me. No errors in the log apart from the ones listed above.
The EBS device is attached to the instance correctly, but rexray can't see the attachment for some reason.
james-masson
on 2 Jan 2019
Switching from a t3.* to t2.* instance type resolved this for me too.
Linked to this it seems: https://github.com/rexray/rexray/issues/1104
karlatkinson
on 4 Jan 2019
Switching from a t3.* to t2.* instance type resolved this for me too.
Must have wasted two evenings till coming to this conclusion also.
st3xupery
on 27 Aug 2019
I've wasted like a week XD, I was using t3a.*, moved to t2, and it's working
eddygarros
on 6 May 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
kinghuang
路
3Comments
frank-montyne
路
8Comments
carl-utter
路
10Comments
pacoxu
路
3Comments
anirbanroydas
路
6Comments
Most helpful comment
I too have this issue with
0.11.3. I'm starting to wonder if anyone is successfully using the EBS plugin on a swarm deployment...