Consul-template: exec mode order

[mafonso]

Consul Template version

/ # consul-template -version
consul-template v0.16.0

Configuration

/ # cat /usr/local/consul-template/config/nginx.hcl
template {
  source      = "/usr/local/consul-template/templates/service.ctmpl"
  destination = "/usr/local/openresty/nginx/conf/nginx.conf"
  command =  "/usr/local/openresty/bin/openresty -s reload"
}

exec {
  command = "/usr/local/openresty/bin/openresty -g \"daemon off;\""
}

/ # cat /usr/local/consul-template/templates/service.ctmpl
worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    gzip  on;

    server {
      listen       80;
      server_name  localhost;

      location /status {
          stub_status on;
          access_log   off;
        }

       {{range services}}{{$service:=.Name}}{{range .Tags}}{{if eq . "public" }}
       location  /{{$service}}/ {
           proxy_pass http://{{$service}}/;
       }{{end}}{{end}}
       {{end}}
    } #end server

    #Upstreams

    {{range services}} {{$service:=.Name}} {{range .Tags}} {{if eq . "public" }}

    upstream {{$service}} { {{range service $service "passing" }}
      server {{.Address}}:{{.Port}};{{end}}
      server 127.0.0.1:65535;
    }{{end}}{{end}}{{end}}

} #end http

Command

#!/bin/sh
CONSUL_IP=$(ip route show 0.0.0.0/0 | grep -Eo 'via \S+' | awk '{ print $2 }')
consul-template -consul=$CONSUL_IP:8500  -config="/usr/local/consul-template/config/nginx.hcl"

This is the container entrypoint script

Debug output

https://gist.github.com/mafonso/2ed99d729e3ea840c5a954710ab42eac

Expected behavior

consul-template first renders the complete file and only then should start the exec process.

Actual behavior

consul-template seems to be starting the exec command before the template is fully rendered and flushed to disk. So the inicial execution fails and consul-template still tries to send a reload signal when the child process is already gone.

After the this first run, nginx won't start because the config is not valid yet. But if the file is inspected after the fail it looks correctly rendered. So running it once again will succeed this time.
i.e. only the first run fails.

[sethvargo]

Hi @mafonso

Thank you for opening an issue. I think Consul Template is behaving as intended, but I definitely agree this is confusing. The order of operations is this:

1. Pull dependencies from Consul/Vault
2. Render templates which have all dependencies satisfied
3. For rendered templates, run associated command _after all templates are rendered_
4. Run exec command after all rendered templates are rendered and all rendered template commands have executed successfully

Looking at the debug log, it looks like the command for the _template_ openresty -s reload is running, but fails to reload due to the lack of a pid file. It does not have a pid file because the exec command hasn't actually started openresty yet. Does that make sense?

In your case, I think you want to change your template command to verify openrestry is running before running the command.

Thanks and have a great day! 😄

[mafonso]

Hi @sethvargo

Thank you the explanation.

My interpretation of the docs was that the exec command would be executed on the first template set rendering to kick of the service, and then subsequent template renderings would only trigger the associated reload command. And this was a perfect use case for my interpretation.

What would be a use case for triggering associated reload commands before the service is running?
Does it assume the reload command should be non-failing?

[sethvargo]

Hi @mafonso

The "command" associated with the template is not a "reload command". There is no reload command. In exec mode, Consul Template will send a _signal_ to the child process telling it to reload. The command is an arbitrary command to run when the template is rendered. It is completely independent of exec mode (and predates exec mode).

[mafonso]

Thanks @sethvargo that makes total sense.
I was mixing the two modes in my head and inferring it was a reload command because that is the most common use for the arbitrary command.
So I just need to drop the associated command and select the signal to send to child in exec mode.