Consul-template: Consul-template exits with 1 if template command fails

[ghost]

I'm using consul-template to configure NGINX with the upstream address/ports of docker containers scheduled with Nomad. However, I have an issue where consul-template exits if the command in the template fails. To be more precise this doesn't appear to be a panic, but an actual exit.

For example, should the systemd service file for NGINX fails to restart (because of a rate limit on restarts) consul-template will then exit. This can cause a deterministic, synchronized failure of what should be a highly available, redundant service.

May 24 12:35:02 frontend-eu-west-1 systemd[1]: nginx.service start request repeated too quickly, refusing to start.
May 24 12:35:02 frontend-eu-west-1 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
May 24 12:35:02 frontend-eu-west-1 systemd[1]: Unit nginx.service entered failed state.
May 24 12:35:02 frontend-eu-west-1 consul-template[12615]: Job for nginx.service failed. See 'systemctl status nginx.service' and 'journalctl -xn' for details.
May 24 12:35:02 frontend-eu-west-1 consul-template[12615]: 2016/05/24 12:35:02 [ERR] (runner) error running command: exit status 1
May 24 12:35:02 frontend-eu-west-1 consul-template[12615]: Consul Template returned errors:
May 24 12:35:02 frontend-eu-west-1 consul-template[12615]: 1 error(s) occurred:
May 24 12:35:02 frontend-eu-west-1 consul-template[12615]: * exit status 1
May 24 12:35:02 frontend-eu-west-1 systemd[1]: consul-template.service: main process exited, code=exited, status=14/n/a
May 24 12:35:02 frontend-eu-west-1 systemd[1]: Unit consul-template.service entered failed state.

Whilst the rate limiting needs to be removed from the systemd service for NGINX, its non-the-less worrying that consul template exits under such circumstances. I'd much prefer it if consul-template logged the error and continued running.

As a temporary fix I've added the following lines to the systemd file for consul-template:

Restart=on-failure
RestartSec=10s

consul-template v0.14.0

[Mongey]

This is documented in the README here

[ghost]

@Mongey whoops! I clearly completely missed this from the docs! Thanks for the link!

[shakisha]

@grobinson-blockchain
There is no reference on how to do that from a configuration file.

[sethvargo]

https://github.com/hashicorp/consul-template#termination-on-error

[shakisha]

@sethvargo i read that, but there isn't any way to make the "|| true" work from the configuration file

[sethvargo]

If you want Consul Template to continue watching for changes, even if the optional command argument fails, you can append || true to your command. Note that || is a "shell-ism", not a built-in function. You will also need to run your command under a shell:

$ consul-template \
  -template "in.ctmpl:out.file:/bin/bash -c 'service nginx restart || true'"

[shakisha]

from configuration file i mean, anyway you already answered me on another issue about this, so i think docs should be enhanced, i will make a merge request

[phamtai97]

Hi @sethvargo, @eikenb I have the same issue. I use command in file config:

 command = "nginx reload || true"

My command:

consul-template -config="./consul-template/consul-template-config.hcl" -log-level=info

But Consul-template still exits.