Cht-core: `svc-restart gardener` runs 2 gardeners

[alxndrsn]

$ sudo ps aux | grep node
  692 gardener /srv/software/medic-core/v1.6.1/x64/bin/node /srv/software/gardener/v1.1.1/bin/gardener http://gardener@localhost:5984 --stdinpass=true --dashboard=dashboard --logdir=/srv/storage/gardener/logs
  937 gardener /srv/software/medic-core/v1.6.1/x64/bin/node /srv/storage/gardener/data/working_dir/xxy/node_modules/medic-api/server.js
 9953 gardener /srv/software/medic-core/v1.6.1/x64/bin/node /srv/software/gardener/v1.1.1/bin/gardener http://gardener@localhost:5984 --stdinpass=true --dashboard=dashboard --logdir=/srv/storage/gardener/logs
10013 gardener /srv/software/medic-core/v1.6.1/x64/bin/node /srv/storage/gardener/data/working_dir/xxx/node_modules/medic-sentinel/server.js
18354 gardener /srv/software/medic-core/v1.6.1/x64/bin/node /srv/storage/gardener/data/working_dir/xxy/node_modules/medic-api/server.js
28121 gardener /srv/software/medic-core/v1.6.1/x64/bin/node /srv/storage/gardener/data/working_dir/xxx/node_modules/medic-sentinel/server.js

And then you get lots of errors like:

2017-03-23T10:20:34.789Z - error: error: Error updating metaData Error: Document update conflict.
    at Request._callback (/srv/storage/gardener/data/working_dir/aHR0cDovL2xvY2FsaG9zdDo1OTg0L21lZGljL19kZXNpZ24vbWVkaWMvbWVkaWMtc2VudGluZWw=/node_modules/medic-sentinel/node_modules/nano/lib/nano.js:248:15)
    at Request.self.callback (/srv/storage/gardener/data/working_dir/aHR0cDovL2xvY2FsaG9zdDo1OTg0L21lZGljL19kZXNpZ24vbWVkaWMvbWVkaWMtc2VudGluZWw=/node_modules/medic-sentinel/node_modules/request/request.js:188:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/srv/storage/gardener/data/working_dir/aHR0cDovL2xvY2FsaG9zdDo1OTg0L21lZGljL19kZXNpZ24vbWVkaWMvbWVkaWMtc2VudGluZWw=/node_modules/medic-sentinel/node_modules/request/request.js:1171:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/srv/storage/gardener/data/working_dir/aHR0cDovL2xvY2FsaG9zdDo1OTg0L21lZGljL19kZXNpZ24vbWVkaWMvbWVkaWMtc2VudGluZWw=/node_modules/medic-sentinel/node_modules/request/request.js:1091:12)
    at IncomingMessage.g (events.js:291:16)

[alxndrsn]

If I run svc-restart gardener 10 times, I get 10 gardener instances.

[ghost]

I've tracked this down to a problem with the gardener init script. Specifically, it appears to leave an orphaned node process subtree when sent SIGTERM, which is then reparented to init. I've reproduced this using the init script itself (bypassing the supervisor system entirely).

Should have a solution soon.

[ghost]

Bug was introduced in https://github.com/medic/medic-webapp/issues/3131; exec behaves differently when there's a pipeline attached to it.

[ghost]

I'll rework the timestamp script tomorrow. Waiting on a pipeline with ash seems unusually difficult; we have a non-default bash in the latest OS master that I might punt to if this takes too much longer. In any case, that should fix this problem everywhere (including #3191).

[ghost]

@alxndrsn: You mind reviewing https://github.com/medic/medic-os/commit/ea8b6703eb0d09f0741e26d25c09b9084689c675?

[alxndrsn]

Comments in-line, which I'd appreciate you review, but do not effect the validity of the code.

[ngaruko]

Any tips how to AT this @alxndrsn ?

[alxndrsn]

1. check how many gardener instances are running
2. call svc-restart gardener
3. check how many gardener instances are runnning

You might want to do it first on an un-patched instance to make sure you're checking correctly.

[mandric]

FYI the possibly non-obvious workaround here is to use /boot/svc-disable && /boot/svc-enable or /boot/svc-down && /boot/svc-up. I didn't even know there was a /boot/svc-restart in medic-os. Might also be worth verifying the workaround is not broken also if you're doing AT on this.