Rexray: Rexray can not start without "-f"

[pacoxu]

I have to use "rexray service start -f" to start my service.

[root@scale158 ~]# rexray service start
● rexray.service - rexray
Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled)
Active: active (running) since 二 2016-05-10 20:30:27 CST; 38ms ago
Main PID: 32414 (systemd)
Memory: 0B
CGroup: /system.slice/rexray.service
└─32414 /usr/lib/systemd/systemd --switched-root --system --deserialize 21

May10 20:30:27 scale158 systemd[1]: Started rexray.
May10 20:30:27 scale158 systemd[1]: Starting rexray...
[root@scale158 ~]# rexray service status
● rexray.service - rexray
Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled)
Active: inactive (dead) since 二 2016-05-10 20:30:27 CST; 7s ago
Process: 32414 ExecStart=/usr/bin/rexray start -f (code=exited, status=0/SUCCESS)
Main PID: 32414 (code=exited, status=0/SUCCESS)

May10 20:30:27 scale158 rexray[32414]: 888b "88bo,888oo,__ oP"''"Yo, 888b "88bo,888 888,,8P"'
May10 20:30:27 scale158 rexray[32414]: MMMM "W" """"YUMMM,m" "Mm, MMMM "W" YMM ""'mM"
May10 20:30:27 scale158 rexray[32414]: Binary: /usr/bin/rexray
May10 20:30:27 scale158 rexray[32414]: SemVer: 0.3.3
May10 20:30:27 scale158 rexray[32414]: OsArch: Linux-x86_64
May10 20:30:27 scale158 rexray[32414]: Branch: v0.3.3
May10 20:30:27 scale158 rexray[32414]: Commit: b30fb870b5b94cd8368824460ad020bfcf20be3a
May10 20:30:27 scale158 rexray[32414]: Formed: Fri, 22 Apr 2016 03:41:36 CST
*_May10 20:30:27 scale158 rexray[32414]: time="2016-05-10T20:30:27+08:00" level=error msg="default module(s) failed to start"
May10 20:30:27 scale158 rexray[32414]: time="2016-05-10T20:30:27+08:00" level=error msg="service initialized failed"
*_[root@scale158 ~]#

[akutz]

Hi @pacoxu,

This is not a bug; it's by design. In fact, if you use the REX-Ray service installer actually creates a unit file for SystemD (which you appear to be using) that includes the -f flag. The excerpt below is the template that is used to create the unit file during REX-Ray's service installation:

const unitFileTemplate = `[Unit]
Description=rexray
Before=docker.service

[Service]
EnvironmentFile={{.EnvFile}}
ExecStart={{.RexrayBin}} start -f
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process

[Install]
WantedBy=docker.service
`

So why does must REX-Ray start itself in the foreground when run as a SystemD service? The answer is because Go does not support the traditional Linux fork(2) command. The go authors did visit the issue back during the development of Go 1.3, but decided against trying to implement what I'm sure they considered to be a flawed compromise for forking.

I've personally heard the reason against a native fork implementation is that the number of goroutines running in a standard go process makes forking nearly impossible to safely manage and implement.

Regardless of why it's not implemented, without it we must use the -f flag when starting a REX-Ray service under SystemD. This is because of the way SystemD handles the service start-up. There are a few configurable options:

| Option | Description |
| --- | --- |
| simple | The default value. The process started with ExecStart is the main process of the service. |
| forking | The process started with ExecStart spawns a child process that becomes the main process of the service. The parent process exits when the startup is complete. |
| oneshot | This type is similar to simple, but the process exits before starting consequent units. |
| dbus | This type is similar to simple, but consequent units are started only after the main process gains a D-Bus name. |
| notify | This type is similar to simple, but consequent units are started only after a notification message is sent via the sd_notify() function. |
| idle | This type is similar to simple, the actual execution of the service binary is delayed until all jobs are finished, which avoids mixing the status output with shell output of services. |

REX-Ray uses the simple option from above. Most services use simple or forking. I'd personally prefer to fork to a new process when starting a service, but as it's not supported by Go, we do a trick when starting the REX-Ray daemon via rexray start on the command line. We actually start a separate process and then the first process communicates with the second via an RPC call in order to block until the second process signals that it has completed startup (or failed).

Unfortunately our trick, while it works great on the command line or via SystemV, confuses SystemD. See, SystemD has no way to know that once our first process exits that there is a second process started by us that is the true service process, and so SystemD considers our service to have failed.

Anyway, I hope this is a reasonable and helpful explanation as to why REX-Ray must use rexray start -f when starting as a service under SystemD.

[clintkitson]

@pacoxu Can you close this issue if you are satisfied with the response?

[pacoxu]

@clintonskitson Thanks a lot.