Consul-template: Build in support for "consul lock"

Created on 12 Mar 2015 · 21Comments · Source: hashicorp/consul-template

I know that i can use "consul lock" in command, but it would be more convenient to have a "lock" key in the JSON to lock on the command execution to make sure that there is only one service restarting at the given time.

If we could also have some kind of extra lock duration variable to be able to say that after the command exited we want to keep the lock for X seconds.

These 2 features would ease rolling updates.

enhancement

Source

sebest

Most helpful comment

Not sure if helpful but here is an example with Chef:

consul_template_config %|nginx_upstream_#{recipe_name}| do
  templates [{
    source: %|#{node['my-consul']['consul_template']['templates']}/nginx_upstream_#{recipe_name}.ctmpl|,
    destination: %|#{node['my-loadbalancer']['consul']['include']}/nginx_upstream_#{recipe_name}.conf|,
    command: 'consul lock -n=1 locks/#{node.chef_environment}/#{cookbook_name}/#{recipe_name} nginx -t && service nginx reload'
  }]
  notifies :restart, 'service[consul-template]', :immediately
  action node['my-loadbalancer']['consul']['enabled'] ? :create : :delete
end

This will allow only one node at a time per Chef environment to reload nginx, just leverage consul lock directly within the consul-template command.

scalp42 on 23 Mar 2017

👍4 🎉1

All 21 comments

This makes sense to me, marking as an enhancement. Thanks!

ryanuber on 12 Mar 2015

Hey @sebest

Thanks for suggesting this. As I mentioned in the other issue, I think there are many assumptions CT would have to make about the underlying template and its environment. For example, CT could be rendering an HTML file that is unrelated to any service registered with Consul. More importantly, the command can be arbitrary, so it would be incredibly difficult to guess the name of the service in Consul from the command the user supplied.

The restarting of a service is entirely controlled by the end user due to the extreme complexity and customization requirements. For this reason, I think the best solution is for the end-user to make these decisions inside the script CT executes. Does that make sense?

sethvargo on 17 Mar 2015

@sethvargo I think that we actually don't need to make any assumptions here. For example, consider the following consul-template config:

template {
  source = "/var/consul/templates/httpd.ctmpl"
  destination = "/etc/httpd.conf"
  command = "restart httpd"
  lock = "services/httpd_restart_lock"
  lock_concurrent = 3
}

The two new keys would function like this:

lock - specifies the key to use for consul lock
lock_concurrent - specifies the number of times the lock can be acquired to allow the semaphore to be used.

This would not affect the current behavior of consul-template, but would allow the end user to get easy rolling upgrades. There are probably a few edge cases, but I think having this as optional behavior would allow us to just avoid them.

Does that make sense? It's possible I am missing something.

ryanuber on 17 Mar 2015

👍1

Thinking about this more, the common use case would probably work by just using the consul lock command directly. Its syntax is easy enough to not require any wrapper scripts. Thats probably the way to go for now. Having it integrated might be nice but also would probably introduce more complexity than its worth.

ryanuber on 17 Mar 2015

@ryanuber yea, I see your point, but as a counterexample, I think this is much more readable IMO:

$ consul-template -template "/templates/config:/etc/config:consul lock -n 3 service config restart"

The only disadvantage I see is that -consul and -http-addr would need to be specified if not the default (and the confusion that they are actually the same):

$ consul-template \
  -consul demo.consul.io \
  -template "/templates/config:/etc/config:consul lock -http-addr demo.consul.io -n 3 service config restart"

Even with that tradeoff, I think it will be better if we leave this decision in the hands of the user instead of having a tightly-grained integration with the lock functionality. What do you think?

sethvargo on 17 Mar 2015

I agree. I think the command gets us 80% there without complicating consul-template. We can also use env vars to configure the consul lock address.

ryanuber on 17 Mar 2015

Ah yes. Envvars. I'm going to leave this open because I think we should add an example to the README, but I think that's about it :smile:

sethvargo on 17 Mar 2015

Sure, enhancing the documentation with tips is maybe the best way to go in the near future. thanx!

sebest on 18 Mar 2015

Whoops. I updated the README in https://github.com/hashicorp/consul-template/commit/662e7d108f76abe643628ad2feb0c07a677b7830, but I forgot to close this out.

@sebest I think this is the best path moving forward for now. We can definitely revisit this issue in the future if things change though :smile:.

sethvargo on 27 Mar 2015

@ryanuber It's not clear for me..does the locking feature exists in Consul Template ?

The documentation does not mention any kind of lock directive for template

nhuray on 22 Jan 2017

It's part of the consul command.

sethvargo on 23 Jan 2017

@sethvargo Would you consider integrating this into consul-template itself? Would be nice for Nomad to be able to limit the concurrent restarts.

Referencing: https://github.com/hashicorp/nomad/issues/2202

dadgar on 1 Feb 2017

@dadgar - not really. There's no point (imo) of copy-pasting code that's already available in consul core. I think if Nomad wants to limit concurrent restarts, it should wrap the consul lock logic.

sethvargo on 2 Feb 2017

@sethvargo Slightly disagree. You could take the sha's of the template create the lock key automatically and use Consul's API rather than CLI tool. It would be very nice!

As a library, consul-template wouldn't make implementing this that easy.

dadgar on 2 Feb 2017

👍3

This would be a great feature for us. Right now in GCP we are testing consul and consul-template. When a new service becomes available or unavailable it brings down our app because everything restarts at the same time. This makes service discovery, the main reason we want to use consul, really hard to do safely.

ryancurrah on 20 Mar 2017

@ryancurrah You should look at setting splay in the meantime!

dadgar on 20 Mar 2017

@dadgar I was looking for something like splay to. I guess I did not give the docs enough of a read. It will be helpful in the meantime. Thanks!

EDIT1: Looks like its only for exec mode not templates :(

EDIT2: Look like for templates what I may want is wait

ryancurrah on 20 Mar 2017

Not sure if helpful but here is an example with Chef:

consul_template_config %|nginx_upstream_#{recipe_name}| do
  templates [{
    source: %|#{node['my-consul']['consul_template']['templates']}/nginx_upstream_#{recipe_name}.ctmpl|,
    destination: %|#{node['my-loadbalancer']['consul']['include']}/nginx_upstream_#{recipe_name}.conf|,
    command: 'consul lock -n=1 locks/#{node.chef_environment}/#{cookbook_name}/#{recipe_name} nginx -t && service nginx reload'
  }]
  notifies :restart, 'service[consul-template]', :immediately
  action node['my-loadbalancer']['consul']['enabled'] ? :create : :delete
end

This will allow only one node at a time per Chef environment to reload nginx, just leverage consul lock directly within the consul-template command.

scalp42 on 23 Mar 2017

👍4 🎉1

@scalp42 thanks this is a nice example we will def be using it

ryancurrah on 23 Mar 2017

👍1

For those of us not versed in Chef, could someone help explain this usage of lock?

spanktar on 23 Jun 2017

Just to be clear on the example provided above for anyone googling this issue: if you're running multiple commands with a lock, wrap them in quotes, otherwise only the first command will run under the lock, ie:

consul lock mylocks 'curl localhost:443/status && sleep 10'