Rexray: Ceph RBD - error initializing instance ID cache
Summary
Unable to use Ceph RBD with Rexray.
Version
$ rexray version
INFO[0000] updated log level logLevel=debug
DEBU[0000] os.args time=1493094295213 val=[rexray version]
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.8.2
OsArch: Linux-x86_64
Branch: v0.8.2
Commit: 7b6be36e0c3b9d69e59f87ebb0098d56a74467db
Formed: Wed, 29 Mar 2017 03:55:54 +06
libStorage
----------
SemVer: 0.5.2
OsArch: Linux-x86_64
Branch: v0.8.2
Commit: 09b3df4425ca4a46382403ecab5b1fad032a279d
Formed: Wed, 29 Mar 2017 03:54:27 +06
DEBU[0000] completed cli execution time=1493094295213
INFO[0000] exiting process time=1493094295213
DEBU[0000] completed onExit at end of program time=1493094295213
$ ceph -v
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
$ uname -a
Linux docker-w1 4.4.0-70-generic #91-Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Expected Behavior
Working Ceph RBD volumes with Rexray.
Actual Behavior
Rexray fails with error: error initializing instance ID cache
Steps To Reproduce
- Install Ceph binaries:
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
echo "deb https://download.ceph.com/debian-jewel/ xenial main" | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update && sudo apt-get install ceph-base ceph-common
- Copy ceph.conf & ceph.client.admin.keyring
$ ls -l /etc/ceph
total 12
-rw------- 1 root root 129 Apr 24 17:36 ceph.client.admin.keyring
-rw-r--r-- 1 root root 1173 Apr 24 17:36 ceph.conf
-rw-r--r-- 1 root root 92 Apr 10 18:21 rbdmap
-rw------- 1 root root 0 Apr 24 17:36 tmpLAXO86
$ cat /etc/ceph/ceph.conf
[global]
fsid = 3359d1d8-5936-40c0-be56-be0a87aa7b36
mon_initial_members = ceph-virt-mon1-int, ceph-virt-mon2-int, ceph-virt-mon3-int
mon_host = 10.32.7.31, 10.32.7.32, 10.32.7.33
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
cluster_network = 10.32.6.0/24
public network = 10.32.7.0/24
[client]
rbd cache = true
rbd cache size = 67108864
rbd cache max dirty = 50331648
[osd]
osd journal size = 60000
journal dio = true
journal aio = true
osd mount options xfs = rw,noexec,noatime,inode64,nodiratime,nobarrier
osd op threads = 12
osd disk threads = 12
filestore op threads = 12
[osd.0]
public addr = 10.32.7.41
[osd.1]
public addr = 10.32.7.41
[osd.2]
public addr = 10.32.7.41
[osd.3]
public addr = 10.32.7.41
[osd.4]
public addr = 10.32.7.42
[osd.5]
public addr = 10.32.7.42
[osd.6]
public addr = 10.32.7.42
[osd.7]
public addr = 10.32.7.42
[osd.8]
public addr = 10.32.7.43
[osd.9]
public addr = 10.32.7.43
[osd.10]
public addr = 10.32.7.43
[osd.11]
public addr = 10.32.7.43
[osd.12]
public addr = 10.32.7.44
[osd.13]
public addr = 10.32.7.44
[osd.14]
public addr = 10.32.7.44
[osd.15]
public addr = 10.32.7.44
- Check for ceph connection:
$ sudo ceph -s
cluster 3359d1d8-5936-40c0-be56-be0a87aa7b36
health HEALTH_OK
monmap e21: 3 mons at {ceph-virt-mon1-int=10.32.7.31:6789/0,ceph-virt-mon2-int=10.32.7.32:6789/0,ceph-virt-mon3-int=10.32.7.33:6789/0}
election epoch 362, quorum 0,1,2 ceph-virt-mon1-int,ceph-virt-mon2-int,ceph-virt-mon3-int
osdmap e4974: 16 osds: 16 up, 16 in
flags sortbitwise,require_jewel_osds
pgmap v1601207: 1280 pgs, 2 pools, 567 GB data, 142 kobjects
1703 GB used, 57883 GB / 59587 GB avail
1280 active+clean
client io 77974 B/s wr, 0 op/s rd, 6 op/s wr
- Create Ceph pool:
$ sudo ceph osd pool create docker1 256
pool 'docker1' created
- Make simple rexray config with http://rexrayconfig.codedellemc.com, debug added:
$ cat /etc/rexray/config.yml
rexray:
logLevel: debug
libstorage:
logging:
level: debug
httpRequests: true
httpResponses: true
libstorage:
service: rbd
rbd:
defaultPool: docker1
- Start rexray:
$ sudo rexray start
INFO[0000] updated log level logLevel=debug
DEBU[0000] os.args time=1493093596262 val=[rexray start]
● rexray.service - rexray
Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2017-04-25 10:13:16 +06; 5ms ago
Main PID: 25093 (rexray)
Tasks: 6
Memory: 3.2M
CPU: 4ms
CGroup: /system.slice/rexray.service
└─25093 /usr/bin/rexray start -f
Apr 25 10:13:16 docker-w1 systemd[1]: Started rexray.
DEBU[0000] completed cli execution time=1493093596292
INFO[0000] exiting process time=1493093596292
DEBU[0000] completed onExit at end of program time=1493093596292
- Try to list volumes:
$ sudo rexray volume ls
INFO[0000] updated log level logLevel=debug
DEBU[0000] os.args time=1493093690464 val=[rexray volume ls]
DEBU[0000] activating libStorage cmd=ls time=1493093690464
DEBU[0000] read spec file host=unix:///var/run/libstorage/874911042.sock path=/var/run/rexray/rexray.spec time=1493093690466
DEBU[0000] is local unix server active sock=/var/run/libstorage/874911042.sock time=1493093690466
INFO[0000] removed invalid sock file sockFile=/var/run/libstorage/874911042.sock time=1493093690467
INFO[0000] removed invalid spec file specFile=/var/run/rexray/rexray.spec time=1493093690467
DEBU[0000] host is empty; initiliazing default services time=1493093690467
INFO[0000] libStorage auto service mode enabled driver=rbd time=1493093690468
DEBU[0000] starting embedded libStorage server time=1493093690469
DEBU[0000] created scoped scope new=libstorage.server parentScopes=rexray,
INFO[0000] configured logging libstorage.logging.httpRequests=false libstorage.logging.httpResponses=false libstorage.logging.level=debug server=mud-hound-ps time=1493093690495
INFO[0000] initializing server server=mud-hound-ps time=1493093690495
INFO[0000] initializing default endpoint autoEndpointMode=unix server=mud-hound-ps time=1493093690500
INFO[0000] initializing auto unix endpoint endpoint=libstorage.server.endpoints.localhost server=mud-hound-ps time=1493093690503
DEBU[0000] endpoint info address=unix:///var/run/libstorage/365777883.sock endpoint=libstorage.server.endpoints.localhost server=mud-hound-ps time=1493093690503
DEBU[0000] created scoped scope new=libstorage.server.endpoints.localhost parentScopes=libstorage.server,rexray,
DEBU[0000] parsing tls config server=mud-hound-ps time=1493093690503
INFO[0000] tls not configured server=mud-hound-ps time=1493093690514
INFO[0000] configured endpoint address=unix:///var/run/libstorage/365777883.sock endpoint=localhost server=mud-hound-ps time=1493093690514
INFO[0000] server created server=mud-hound-ps time=1493093690514
INFO[0000] initialized endpoints server=mud-hound-ps time=1493093690514
INFO[0000] initializing server services server=mud-hound-ps time=1493093690514
DEBU[0000] configured result schema validation enabled=false server=mud-hound-ps time=1493093690516
DEBU[0000] got services map count=1 server=mud-hound-ps time=1493093690517
DEBU[0000] processing service config server=mud-hound-ps service=rbd time=1493093690517
DEBU[0000] getting scoped config for service scope=libstorage.server.services.rbd server=mud-hound-ps service=rbd time=1493093690517
DEBU[0000] created scoped scope new=libstorage.server.services.rbd parentScopes=libstorage.server,rexray,
DEBU[0000] got driver name driverName=rbd server=mud-hound-ps service=rbd time=1493093690520
INFO[0000] storage driver initialized server=mud-hound-ps service=rbd storageDriver=rbd time=1493093690520
INFO[0000] created new service server=mud-hound-ps service=rbd time=1493093690521
INFO[0000] initialized services server=mud-hound-ps time=1493093690521
INFO[0000] initialized router len(routes)=3 router=executor-router
INFO[0000] initialized router len(routes)=4 router=help-router
INFO[0000] initialized router len(routes)=1 router=root-router
INFO[0000] initialized router len(routes)=2 router=service-router
INFO[0000] initialized router len(routes)=11 router=volume-router
INFO[0000] initialized router len(routes)=6 router=snapshot-router
INFO[0000] initialized router len(routes)=2 router=tasks-router
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/executors queries=[] route=executors server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/executors/{executor} queries=[] route=executorInspect server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=HEAD path=/executors/{executor} queries=[] route=executorHead server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/help queries=[] route=version server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/help/config queries=[] route=version server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/help/env queries=[] route=version server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/help/version queries=[] route=version server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/ queries=[] route=root server=mud-hound-ps time=1493093690521 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/services queries=[] route=services server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/services/{service} queries=[] route=serviceInspect server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/volumes queries=[] route=volumes server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/volumes/{service} queries=[] route=volumesForService server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/volumes/{service}/{volumeID} queries=[] route=volumeInspect server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/volumes/{service} queries=[detach ] route=volumesDetachForService server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=POST path=/volumes/{service} queries=[] route=volumeCreate server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/volumes/{service}/{volumeID} queries=[copy ] route=volumeCopy server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/volumes/{service}/{volumeID} queries=[snapshot ] route=volumeSnapshot server=mud-hound-ps time=1493093690522 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/volumes/{service}/{volumeID} queries=[attach ] route=volumeAttach server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/volumes queries=[detach ] route=volumesDetachAll server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/volumes/{service}/{volumeID} queries=[detach ] route=volumeDetach server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=DELETE path=/volumes/{service}/{volumeID} queries=[] route=volumeRemove server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/snapshots queries=[] route=snapshots server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/snapshots/{service} queries=[] route=snapshotsForService server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/snapshots/{service}/{snapshotID} queries=[] route=snapshotInspect server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/snapshots/{service}/{snapshotID} queries=[create ] route=snapshotCreate server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=2 method=POST path=/snapshots/{service}/{snapshotID} queries=[copy ] route=snapshotCopy server=mud-hound-ps time=1493093690523 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=DELETE path=/snapshots/{service}/{snapshotID} queries=[] route=snapshotRemove server=mud-hound-ps time=1493093690524 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/tasks queries=[] route=tasks server=mud-hound-ps time=1493093690524 tls=false
DEBU[0000] registered route host=unix:///var/run/libstorage/365777883.sock len(queries)=0 method=GET path=/tasks/{taskID} queries=[] route=taskInspect server=mud-hound-ps time=1493093690524 tls=false
INFO[0000] waiting for err or close signal server=mud-hound-ps time=1493093690524
INFO[0000] api listening host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps time=1493093690524 tls=false
INFO[0001] server started server=mud-hound-ps time=1493093691524
DEBU[0001] got host from new server address host=unix:///var/run/libstorage/365777883.sock time=1493093691527
DEBU[0001] set host in context host=unix:///var/run/libstorage/365777883.sock time=1493093691527
DEBU[0001] set host in config host=unix:///var/run/libstorage/365777883.sock time=1493093691527
DEBU[0001] created spec file host=unix:///var/run/libstorage/365777883.sock path=/var/run/rexray/rexray.spec time=1493093691527
DEBU[0001] creating libStorage client cmd=ls host=unix:///var/run/libstorage/365777883.sock time=1493093691527
DEBU[0001] created scoped scope new=libstorage.client parentScopes=rexray,
INFO[0001] configured logging host=unix:///var/run/libstorage/365777883.sock libstorage.logging.httpRequests=false libstorage.logging.httpResponses=false libstorage.logging.level=debug time=1493093691576
DEBU[0001] got configured host address host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691581
DEBU[0001] parsing tls config host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691581
INFO[0001] tls not configured host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691585
INFO[0001] created libStorage client clientType=integration disableKeepAlive=false enableInstanceIDHeaders=true enableLocalDevicesHeaders=true host=unix:///var/run/libstorage/365777883.sock logRequests=false logResponses=false lsxPath=/var/lib/libstorage/lsx-linux service=rbd storageDriver=libstorage time=1493093691595
INFO[0001] lsx lock file path host=unix:///var/run/libstorage/365777883.sock path=/var/run/libstorage/lsx.lock service=rbd storageDriver=libstorage time=1493093691595
INFO[0001] http request host=unix:///var/run/libstorage/365777883.sock route=services server=mud-hound-ps time=1493093691596 tls=false
DEBU[0001] added route middleware host=unix:///var/run/libstorage/365777883.sock middleware=schema-validator route=services server=mud-hound-ps time=1493093691596 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=transaction-handler route=services server=mud-hound-ps time=1493093691596 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=local-devices-handler route=services server=mud-hound-ps time=1493093691596 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=instanceIDs-handler route=services server=mud-hound-ps time=1493093691596 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=error-handler route=services server=mud-hound-ps time=1493093691596 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=transaction-handler route=services server=mud-hound-ps time=1493093691597 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=query-params-handler route=services server=mud-hound-ps time=1493093691597 tls=false
DEBU[0001] http header Libstorage-Tx=txID=8fb51982-9e75-41b3-609e-f6a4f5bcbb3d, txCR=1493093691 host=unix:///var/run/libstorage/365777883.sock route=services server=mud-hound-ps time=1493093691597 tls=false
DEBU[0001] http header Libstorage-Instanceid=[] host=unix:///var/run/libstorage/365777883.sock route=services server=mud-hound-ps time=1493093691597 tls=false txCR=1493093691 txID=8fb51982-9e75-41b3-609e-f6a4f5bcbb3d
DEBU[0001] http header Libstorage-Localdevices=[] host=unix:///var/run/libstorage/365777883.sock route=services server=mud-hound-ps time=1493093691597 tls=false txCR=1493093691 txID=8fb51982-9e75-41b3-609e-f6a4f5bcbb3d
INFO[0001] initializing executors cache host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691600
INFO[0001] http request host=unix:///var/run/libstorage/365777883.sock route=executors server=mud-hound-ps time=1493093691600 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=transaction-handler route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=local-devices-handler route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=instanceIDs-handler route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=error-handler route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=transaction-handler route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] added global middleware host=unix:///var/run/libstorage/365777883.sock middleware=query-params-handler route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] http header Libstorage-Tx=txID=af28fb8e-7f8f-425d-529b-5ace490478b9, txCR=1493093691 host=unix:///var/run/libstorage/365777883.sock route=executors server=mud-hound-ps time=1493093691601 tls=false
DEBU[0001] http header Libstorage-Instanceid=[] host=unix:///var/run/libstorage/365777883.sock route=executors server=mud-hound-ps time=1493093691601 tls=false txCR=1493093691 txID=af28fb8e-7f8f-425d-529b-5ace490478b9
DEBU[0001] http header Libstorage-Localdevices=[] host=unix:///var/run/libstorage/365777883.sock route=executors server=mud-hound-ps time=1493093691601 tls=false txCR=1493093691 txID=af28fb8e-7f8f-425d-529b-5ace490478b9
DEBU[0001] updating executor host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691601
DEBU[0001] waiting on executor lock host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691601
DEBU[0001] executor exists, getting local checksum host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691601
DEBU[0001] getting executor checksum host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691602
DEBU[0001] got local executor checksum host=unix:///var/run/libstorage/365777883.sock localChecksum=fc0c6bb68e5bb8f384784b54d2c978d4 service=rbd storageDriver=libstorage time=1493093691630
DEBU[0001] signalling executor lock host=unix:///var/run/libstorage/365777883.sock service=rbd storageDriver=libstorage time=1493093691630
INFO[0001] initializing supported cache host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691630
DEBU[0001] waiting on executor lock host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691630 txCR=1493093691 txID=d3e668f8-0023-457e-76af-67e6862b43fc
DEBU[0001] invoking executor cli args=[rbd supported] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691630 txCR=1493093691 txID=d3e668f8-0023-457e-76af-67e6862b43fc
DEBU[0001] signalling executor lock host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691700 txCR=1493093691 txID=d3e668f8-0023-457e-76af-67e6862b43fc
DEBU[0001] cached supported flag host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage supported=15 time=1493093691701 txCR=1493093691 txID=d3e668f8-0023-457e-76af-67e6862b43fc
INFO[0001] initializing instance ID cache host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691701
DEBU[0001] waiting on executor lock host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691701 txCR=1493093691 txID=8903752a-54bc-44a7-5395-78778cc7dfe2
DEBU[0001] invoking executor cli args=[rbd instanceID] cmd=/var/lib/libstorage/lsx-linux host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691701 txCR=1493093691 txID=8903752a-54bc-44a7-5395-78778cc7dfe2
DEBU[0001] signalling executor lock host=unix:///var/run/libstorage/365777883.sock server=mud-hound-ps service=rbd storageDriver=libstorage time=1493093691775 txCR=1493093691 txID=8903752a-54bc-44a7-5395-78778cc7dfe2
Oops, an error occured!
error initializing instance ID cache
To correct the error please review:
- Debug output by using the flag -l debug
- The REX-ray website at https://github.com/codedellemc/rexray
- The on↓ine he↓p be↓ow
List volumes
Usage:
rexray volume ls [flags]
Aliases:
ls, l, list, get, inspect
Examples:
rexray volume ls [OPTIONS] [VOLUME...]
Flags:
--attached[=false]: A flag that indicates only volumes attached to this host should be returned
--available[=false]: A flag that indicates only available volumes should be returned
--path[=false]: A flag that indicates only volumes attached to this host should be returned, along with their path info
Global Flags:
-c, --config="": The path to a custom REX-Ray configuration file
-n, --dryRun[=false]: Show what action(s) will occur, but do not execute them
-f, --format="tmpl": The output format (tmpl, json, jsonp)
-?, --help[=false]: Help for ls
-h, --host="": The libStorage host.
-l, --logLevel="warn": The log level (error, warn, info, debug)
-q, --quiet[=false]: Suppress table headers
-s, --service="": The libStorage service.
--template="": The Go template to use when --format is set to 'tmpl'
--templateTabs[=true]: Set to true to use a Go tab writer with the output template
-v, --verbose[=false]: Print verbose help information
DEBU[0001] exiting with default error code 1, r=&{}
Rexray service log
https://gist.github.com/ZigmunD/315fc9df86e1aedd9441c7545db2ed3c
bug
zigmund
All 7 comments
Hi @ZigmunD,
Thanks for the detailed report. From the service log that you pasted, I can see that the underlying error is actually error getting instanceID: lookup 10.32.7.32: invalid domain name. In your log this is encoded as a byte string still, but this has been fixed on master.
Can you please run the following command on the node where you were running your rexray command:
/var/lib/libstorage/bin/lsx-linux rbd instanceID
and paste the output?
For some reason it's having a hard time determine the local IP address that would be used as the source when communicating with the Ceph monitors. This IP is used as the "Instance ID" to identify the REX-Ray client.
codenrhoden
on 25 Apr 2017
Thanks for reply, @codenrhoden
$ /var/lib/libstorage/lsx-linux rbd instanceID
error: error getting instanceID: lookup 10.32.7.32: invalid domain name
This machine in same subnet with ceph mons.
$ ip a s ens19
474: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 9e:81:e8:5e:22:d9 brd ff:ff:ff:ff:ff:ff
inet 10.32.7.51/24 brd 10.32.7.255 scope global ens19
valid_lft forever preferred_lft forever
inet6 fe80::9c81:e8ff:fe5e:22d9/64 scope link
valid_lft forever preferred_lft forever
$ ip r g 10.32.7.0/24
broadcast 10.32.7.0 dev ens19 src 10.32.7.51
cache <local,brd>
$ ping 10.32.7.32 -c1
PING 10.32.7.32 (10.32.7.32) 56(84) bytes of data.
64 bytes from 10.32.7.32: icmp_seq=1 ttl=64 time=0.361 ms
--- 10.32.7.32 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.361/0.361/0.361/0.000 ms
Upgraded rexray to 0.9.0-1, still no luck:
$ /var/lib/libstorage/lsx-linux rbd instanceID
error: error getting instanceID: lookup 10.32.7.32: invalid domain name
$ host 10.32.7.32
32.7.32.10.in-addr.arpa domain name pointer ceph-virt-mon2-int.alanls.kz.prod.
$ host ceph-virt-mon2-int.alanls.kz.prod
ceph-virt-mon2-int.alanls.kz.prod has address 10.32.7.32
Remove the whitespace from your mon_hosts ip addresses, fixed it for me :)
mon_host = 10.32.7.31, 10.32.7.32, 10.32.7.33
->
mon_host = 10.32.7.31,10.32.7.32,10.32.7.33
DGollings
on 12 May 2017
👍2
Thanks, @DGollings, it works.
I think that parser should be fixed.
zigmund
on 16 May 2017
Thanks for the assist, @DGollings !
@ZigmunD yes, it should be fixed, and will be. Issue is here: https://github.com/codedellemc/libstorage/issues/551
codenrhoden
on 23 May 2017
This will be fixed with codedellemc/libstorage#558
codenrhoden
on 24 May 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
neoitec
·
7Comments
carl-utter
·
10Comments
anirbanroydas
·
6Comments
agilezebra
·
7Comments
ptemmer
·
6Comments
Most helpful comment
Remove the whitespace from your mon_hosts ip addresses, fixed it for me :)
mon_host = 10.32.7.31, 10.32.7.32, 10.32.7.33
->
mon_host = 10.32.7.31,10.32.7.32,10.32.7.33