Rexray: Latest rexray version 0.3.3 for Debian dies just after start it

Created on 22 Apr 2016 · 10Comments · Source: rexray/rexray

Latest version 0.3.3 dies after start it in debian stable (jessie) with:
# systemctl start rexray.service

I can only see this logs in rexrray logs:
...
Binary: /usr/bin/rexray
SemVer: 0.3.3
OsArch: Linux-x86_64
Branch: v0.3.3
Commit: b30fb870b5b94cd8368824460ad020bfcf20be3a
Formed: Thu, 21 Apr 2016 21:41:36 CEST

time="2016-04-22T16:08:53+02:00" level=error msg="default module(s) failed to start"
time="2016-04-22T16:08:53+02:00" level=error msg="service initialized failed"

Version 0.3.2 works fine
# dpkg -l rexray
ii rexray 0.3.2-1
# systemctl start rexray.service
# ps aux | grep rexray
root 1075 0.1 4.9 165600 25284 ? Ssl 16:16 0:00 /usr/bin/rexray start -f

question

Source

gonzalomarcote

All 10 comments

Hi @clintonskitson,

I wonder if this is related to the fix for #373.

akutz on 22 Apr 2016

Will investigate further this am.

On Friday, April 22, 2016, Schley Andrew Kutz [email protected]
wrote:

Hi @clintonskitson https://github.com/clintonskitson,

I wonder if this is related to the fix for #373
https://github.com/emccode/rexray/pull/373.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/emccode/rexray/issues/389#issuecomment-213452081

clintkitson on 22 Apr 2016

@gonzalomarcote I just tried and couldn't replicate what you saw.

Can you try starting the service with rexray start -f -l debug?

clintkitson on 22 Apr 2016

@gonzalomarcote Were you able to get this working properly?

clintkitson on 10 May 2016

Yes. I have tried it again with the latest rexray debian package (0.3.3-1) and daemon rejects to start:

# systemctl status rexray.service
● rexray.service - rexray
   Loaded: loaded (/etc/systemd/system/rexray.service; enabled)
   Active: inactive (dead)

May 10 10:02:33 rexray rexray[454]: ⌐▄Q▓▄Ç▓▄,▄_
May 10 10:02:33 rexray rexray[454]: Σ▄▓▓▓▓▓▓▓▓▓▓▄π
May 10 10:02:33 rexray rexray[454]: ╒▓▓▌▓▓▓▓▓▓▓▓▓▓▀▓▄▄.
May 10 10:02:33 rexray rexray[454]: ,_▄▀▓▓ ▓▓ ▓▓▓▓▓▓▓▓▓▓▓█
May 10 10:02:33 rexray rexray[454]: │▄▓▓ _▓▓▓▓▓▓▓▓▓┌▓▓▓▓▓█
May 10 10:02:33 rexray rexray[454]: _J┤▓▓▓▓▓▓▓▓▓▓▓▓▓├█▓█▓▀Γ
May 10 10:02:33 rexray rexray[454]: ,▄▓▓▓▓▓▓^██▓▓▓▓▓▓▓▓▓▓▓▓▄▀▄▄▓▓Ω▄
May 10 10:02:33 rexray rexray[454]: F▌▓▌█ⁿⁿⁿ  ⁿ└▀ⁿ██▓▀▀▀▀▀▀▀▀▀▀▌▓▓▓▌
May 10 10:02:33 rexray rexray[454]: 'ⁿ_  ,▄▄▄▄▄▄▄▄▄█_▄▄▄▄▄▄▄▄▄ⁿ▀~██
May 10 10:02:33 rexray rexray[454]: Γ  ├▓▓▓▓▓█▀ⁿ█▌▓Ω]█▓▓▓▓▓▓ ├▓
May 10 10:02:33 rexray rexray[454]: │  ├▓▓▓▓▓▌≡,__▄▓▓▓█▓▓▓▓▓ ╞█~   Y,┐
May 10 10:02:33 rexray rexray[454]: ╞  ├▓▓▓▓▓▄▄__^^▓▓▓▌▓▓▓▓▓  ▓   /▓▓▓
May 10 10:02:33 rexray rexray[454]: ├▓▓▓▓▓▓▓▄▄═▄▓▓▓▓▓▓▓▓▓  π ⌐▄▓▓█║n
May 10 10:02:33 rexray rexray[454]: _ ├▓▓▓▓▓▓▓▓▓~▓▓▓▓▓▓▓▓▓▓  ▄4▄▓▓▓██
May 10 10:02:33 rexray rexray[454]: µ ├▓▓▓▓█▀█▓▓_▓▓███▓▓▓▓▓  ▓▓▓▓▓Ω4
May 10 10:02:33 rexray rexray[454]: µ ├▓▀▀L   └ⁿ  ▀   ▀ ▓▓█w ▓▓▓▀ìⁿ
May 10 10:02:33 rexray rexray[454]: ⌐ ├_                τ▀▓  Σ⌐└
May 10 10:02:33 rexray rexray[454]: ~ ├▓▓  ▄  _     ╒  ┌▄▓▓  Γ
May 10 10:02:33 rexray rexray[454]: ├▓▓▓▌█═┴▓▄╒▀▄_▄▌═¢▓▓▓  ╚
May 10 10:02:33 rexray rexray[454]: ⌠  ├▓▓▓▓▓ⁿ▄▓▓▓▓▓▓▓┐▄▓▓▓▓  └
May 10 10:02:33 rexray rexray[454]: Ω_.└██▓▀ⁿÇⁿ▀▀▀▀▀▀█≡▀▀▀▀▀   µ
May 10 10:02:33 rexray rexray[454]: ⁿ  .▄▄▓▓▓▓▄▄┌ ╖__▓_▄▄▄▄▄*Oⁿ
May 10 10:02:33 rexray rexray[454]: û▌├▓█▓▓▓██ⁿ ¡▓▓▓▓▓▓▓▓█▓╪
May 10 10:02:33 rexray rexray[454]: ╙Ω▀█ ▓██ⁿ    └█▀██▀▓█├█Å
May 10 10:02:33 rexray rexray[454]: ⁿⁿ             ⁿ ⁿ^
May 10 10:02:33 rexray rexray[454]: :::::::..  .,::::::    .,::      .::::::::..    :::.  .-:.     ::-.
May 10 10:02:33 rexray rexray[454]: ;;;;'';;;; ;;;;''''    ';;;,  .,;; ;;;;'';;;;   ;;';;  ';;.   ;;;;'
May 10 10:02:33 rexray rexray[454]: [[[,/[[['  [[cccc       '[[,,[['   [[[,/[[['  ,[[ '[[,  '[[,[[['
May 10 10:02:33 rexray rexray[454]: $$$$$$c    $$""""        Y$$$Pcccc $$$$$$c   c$$$cc$$$c   c$$"
May 10 10:02:33 rexray rexray[454]: 888b "88bo,888oo,__    oP"''"Yo,   888b "88bo,888   888,,8P"'
May 10 10:02:33 rexray rexray[454]: MMMM   "W" """"YUMMM,m"       "Mm, MMMM   "W" YMM   ""'mM"
May 10 10:02:33 rexray rexray[454]: Binary: /usr/bin/rexray
May 10 10:02:33 rexray rexray[454]: SemVer: 0.3.3
May 10 10:02:34 rexray rexray[454]: OsArch: Linux-x86_64
May 10 10:02:34 rexray rexray[454]: Branch: v0.3.3
May 10 10:02:34 rexray rexray[454]: Commit: b30fb870b5b94cd8368824460ad020bfcf20be3a
May 10 10:02:34 rexray rexray[454]: Formed: Thu, 21 Apr 2016 21:41:36 CEST
May 10 10:02:36 rexray rexray[454]: time="2016-05-10T10:02:36+02:00" level=error msg="default module(s) failed to start"
May 10 10:02:36 rexray rexray[454]: time="2016-05-10T10:02:36+02:00" level=error msg="service initialized failed"

This is my config.yml file:

rexray:
 logLevel: warn
 storageDrivers:
 - openstack
 osDrivers:
 - linux
 volumeDrivers:
 - docker
openstack:
 authUrl: http://***:35357/v2.0/
 username: ***
 password: ***
 tenantName: admin
 regionName: regionOne

This is the end of the log starting the service with rexray start -f -l debug:

DEBU[0000] flattening env vars                           key=isilon.endpoint value=
INFO[0001] storage driver initialized                    availabilityZone=nova domainId= domainName= identityEndpoint=http://***:35357/v2.0/ instanceId=f3176409-d365-4f4e-8dfc-***** moduleName=default-docker password=****** provider=Openstack region=regionOne tenantId= tenantName=*** userId= userName=***
INFO[0001] docker volume driver initialized              availabilityZone= iops= moduleName=default-docker provider=docker size= volumeRootPath=/data volumeType=
INFO[0001] os driver initialized                         moduleName=default-docker provider=linux
DEBU[0001] checking volume path cache setting            pathCache=true
INFO[0001] vdm.List                                      driverName=docker moduleName=default-docker
INFO[0001] listing volumes                               driverName=docker moduleName=default-docker
ERRO[0003] default module(s) failed to start            
INFO[0003] service sent registered modules start signals 
ERRO[0003] service initialized failed                   
INFO[0003] timed out while monitoring module star

I can see one error about one ´default module(s) failed to start´

gonzalomarcote on 10 May 2016

@clintonskitson @gonzalomarcote
I have run into this behavior as well. In my case I am using AWS EBS as my storage driver. As I am running REX-ray in a container, I simply restart it a few times until it works. It seems to be caused by #364. When 0.4 is released the timeout will be configurable.

fritz-fritz on 14 Jun 2016

@gonzalomarcote Thanks for the reply here. The issue is actually related to the startup timeout and not the mount/unmount timeout. This is covered in 0.4.0 here https://github.com/emccode/rexray/pull/429

clintkitson on 14 Jun 2016

Thank you for the info. I will update to that version.

Enviado desde mi Android
On Jun 14, 2016 6:44 PM, "Clinton Kitson" [email protected] wrote:

@gonzalomarcote https://github.com/gonzalomarcote Thanks for the reply
here. The issue is actually related to the startup timeout and not the
mount/unmount timeout. This is covered in 0.4.0 here #429
https://github.com/emccode/rexray/pull/429

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/emccode/rexray/issues/389#issuecomment-225942153, or mute
the thread
https://github.com/notifications/unsubscribe/ADb1ZarES5gtdZsbhSGiic5-7SATWLVkks5qLtpPgaJpZM4INqB9
.

gonzalomarcote on 14 Jun 2016

Just heads up, I got 0.3.3 starting fine on boot with systemd by putting the following in /etc/systemd/system/rexray.service.d/override.conf

[Service]
Restart=always
StartLimitIntervalSec=100
StartLimitBurst=10

byrnedo on 5 Jul 2016

👍1

@byrnedo You saved my life!! I was having half of my rexray server not initialising while the other half was working just fine. addind your hack to the ansible deployment, and 100% up and running. Thanks a lot!!