Consul-template: Feature request for skipping a failed datacenter

[sukujgrg]

Consul Template version

consul-template v0.18.1 (9c62737)

Configuration

I am passing config as arguments

Template

{{ range $dc:= datacenters }}
{{ . }}
    {{ range services "@" $dc }}
        {{ .Name }}: {{ .Tags | join "," }}{{ end }}
{{ end }}

Command

./consul-template -consul-addr=127.0.1.4:8500 -template config.tmpl:config -once

Debug output

consul.log.txt

Enhancement request

A filter to skip failed datacenters

Steps to reproduce

1. setup a datacenter named dc1

[root@localhost vagrant]# ./consul members  -rpc-addr=127.0.1.1:8400
Node                   Address         Status  Type    Build  Protocol  DC
consul-agent           127.0.1.4:8301  alive   client  0.7.2  2         dc1
consul-server_WAN_A_1  127.0.1.1:8301  alive   server  0.7.2  2         dc1
consul-server_WAN_A_2  127.0.1.2:8301  alive   server  0.7.2  2         dc1
consul-server_WAN_A_3  127.0.1.3:8301  alive   server  0.7.2  2         dc1

1. setup another datacenter named dc2

[root@localhost vagrant]# ./consul members  -rpc-addr=127.0.2.1:8400
Node                   Address         Status  Type    Build  Protocol  DC
consul-agent           127.0.2.4:8301  alive   client  0.7.2  2         dc2
consul-server_WAN_B_1  127.0.2.1:8301  alive   server  0.7.2  2         dc2
consul-server_WAN_B_2  127.0.2.2:8301  alive   server  0.7.2  2         dc2
consul-server_WAN_B_3  127.0.2.3:8301  alive   server  0.7.2  2         dc2

1. make sure -wan is able to list all server members in both DCs

[root@localhost vagrant]# ./consul members  -rpc-addr=127.0.1.1:8400 -wan
Node                       Address         Status  Type    Build  Protocol  DC
consul-server_WAN_A_1.dc1  127.0.1.1:8302  alive   server  0.7.2  2         dc1
consul-server_WAN_A_2.dc1  127.0.1.2:8302  alive   server  0.7.2  2         dc1
consul-server_WAN_A_3.dc1  127.0.1.3:8302  alive   server  0.7.2  2         dc1
consul-server_WAN_B_1.dc2  127.0.2.1:8302  alive   server  0.7.2  2         dc2
consul-server_WAN_B_2.dc2  127.0.2.2:8302  alive   server  0.7.2  2         dc2
consul-server_WAN_B_3.dc2  127.0.2.3:8302  alive   server  0.7.2  2         dc2

1. Register two services, one on each dc

[root@localhost vagrant]# curl -X PUT -d '{"Datacenter": "dc1", "Node": "service1", "Address": "cff", "Service": {"Servie": "cff", "Port": 80}}' http://127.0.1.1:8500/v1/catalog/register
true

[root@localhost vagrant]# curl -X PUT -d '{"Datacenter": "dc2", "Node": "service2", "Address": "crr", "Service": {"Service": "crr", "Port": 80}}' http://127.0.2.1:8500/v1/catalog/register
true

[root@localhost vagrant]# curl http://127.0.1.3:8500/v1/catalog/services?dc=dc1
{"crr":[],"consul":[]}

[root@localhost vagrant]# curl http://127.0.1.3:8500/v1/catalog/services?dc=dc2
{"consul":[],"cff":[]}

1. Kill two consul instances in DC2

[root@localhost vagrant]# ./consul members  -rpc-addr=127.0.2.2:8400
Node                   Address         Status  Type    Build  Protocol  DC
consul-agent           127.0.2.4:8301  alive   client  0.7.2  2         dc2
consul-server_WAN_B_1  127.0.2.1:8301  failed  server  0.7.2  2         dc2
consul-server_WAN_B_2  127.0.2.2:8301  alive   server  0.7.2  2         dc2
consul-server_WAN_B_3  127.0.2.3:8301  failed  server  0.7.2  2         dc2

1. Create a template

[root@localhost vagrant]# cat config.tmpl
{{ range $dc:= datacenters }}
{{ . }}
    {{ range services "@" $dc }}
        {{ .Name }}: {{ .Tags | join "," }}{{ end }}
{{ end }}

1. And execute consul template against above template

[root@localhost vagrant]# ./consul-template -consul-addr=127.0.1.4:8500 -template config.tmpl:config -once
2017/03/09 06:06:35.471611 [WARN] (view) catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader) (retry attempt 1 after "250ms")
2017/03/09 06:06:42.762007 [WARN] (view) catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader) (retry attempt 2 after "500ms")
2017/03/09 06:06:50.324379 [WARN] (view) catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader) (retry attempt 3 after "1s")
2017/03/09 06:06:58.461451 [WARN] (view) catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader) (retry attempt 4 after "2s")
2017/03/09 06:07:07.760496 [WARN] (view) catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader) (retry attempt 5 after "4s")
2017/03/09 06:07:18.879185 [ERR] (view) catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader) (exceeded maximum retries)
2017/03/09 06:07:18.879203 [ERR] (runner) watcher reported error: catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader)
Consul Template returned errors:
catalog.services(@dc2): Unexpected response code: 500 (rpc error: rpc error: No cluster leader)

Note:

Use this for setting up consul datacenters quickly

nohup consul agent -bind 127.0.1.1 -client=127.0.1.1 -node=consul-server_WAN_A_1 -data-dir=/tmp/consulA1 -retry-join=127.0.1.2 -retry-join=127.0.1.3 -retry-join-wan=127.0.2.1 -retry-join-wan=127.0.2.2 -retry-join-wan=127.0.2.3 -bootstrap-expect=3 -datacenter=dc1 -server &
nohup consul agent -bind 127.0.1.2 -client=127.0.1.2 -node=consul-server_WAN_A_2 -data-dir=/tmp/consulA2 -retry-join=127.0.1.1 -retry-join=127.0.1.3 -retry-join-wan=127.0.2.1 -retry-join-wan=127.0.2.2 -retry-join-wan=127.0.2.3 -bootstrap-expect=3 -datacenter=dc1 -server &
nohup consul agent -bind 127.0.1.3 -client=127.0.1.3 -node=consul-server_WAN_A_3 -data-dir=/tmp/consulA3 -retry-join=127.0.1.1 -retry-join=127.0.1.2 -retry-join-wan=127.0.2.1 -retry-join-wan=127.0.2.2 -retry-join-wan=127.0.2.3 -bootstrap-expect=3 -datacenter=dc1 -server &
nohup consul agent -bind 127.0.2.1 -client=127.0.2.1 -node=consul-server_WAN_B_1 -data-dir=/tmp/consulB1 -retry-join=127.0.2.2 -retry-join=127.0.2.3 -retry-join-wan=127.0.1.1 -retry-join-wan=127.0.1.2 -retry-join-wan=127.0.1.3 -bootstrap-expect=3 -datacenter=dc2 -server &
nohup consul agent -bind 127.0.2.2 -client=127.0.2.2 -node=consul-server_WAN_B_2 -data-dir=/tmp/consulB2 -retry-join=127.0.2.1 -retry-join=127.0.2.3 -retry-join-wan=127.0.1.1 -retry-join-wan=127.0.1.2 -retry-join-wan=127.0.1.3 -bootstrap-expect=3 -datacenter=dc2 -server &
nohup consul agent -bind 127.0.2.3 -client=127.0.2.3 -node=consul-server_WAN_B_3 -data-dir=/tmp/consulB3 -retry-join=127.0.2.1 -retry-join=127.0.2.2 -retry-join-wan=127.0.1.1 -retry-join-wan=127.0.1.2 -retry-join-wan=127.0.1.3 -bootstrap-expect=3 -datacenter=dc2 -server &
nohup consul agent -bind 127.0.1.4 -client=127.0.1.4 -node=consul-agent -data-dir=/tmp/consulA4 -retry-join=127.0.1.1 -retry-join=127.0.1.2 -retry-join=127.0.1.3 -datacenter=dc1 &
nohup consul agent -bind 127.0.2.4 -client=127.0.2.4 -node=consul-agent -data-dir=/tmp/consulB4 -retry-join=127.0.2.1 -retry-join=127.0.2.2 -retry-join=127.0.2.3 -datacenter=dc2 &

[sethvargo]

Hi @sukujgrg

There is no way to "skip" a data center, but you can permit an infinite number of retries in the configuration.

[sukujgrg]

@sethvargo it will be good to have a feature to proceed with config generation in case one datacenter is not available. At present, if one datacenter is not available, the config generation won't happen based on currently running other datacenters

[sethvargo]

Hi @sukujgrg

Unfortunately that's not really possible. There's no way to tell the difference between "a datacenter is not available" and any other transient error that might occur.

[Loushu]

+1 on this request. If one of datacenters lost leadership, for example, the request for its services would return a 500 code. It would be very useful to be able to tell template generation to just skip this datacenter and continue enumerating, treating any services from such datacenter as unavailable.

[blalor]

Yeah, I ran into this today. The problem is that consul-template exits when a (remote) dc doesn't have a leader.

(view) kv.get(datacenter-details@test0_aws_gen_us-east-1): Unexpected response code: 500 (retry attempt 5 after "4s")
(view) kv.get(datacenter-details@test0_aws_gen_us-east-1): Unexpected response code: 500 (exceeded maximum retries)
(runner) watcher reported error: kv.get(datacenter-details@test0_aws_gen_us-east-1): Unexpected response code: 500
Consul Template returned errors:
kv.get(datacenter-details@test0_aws_gen_us-east-1): Unexpected response code: 500

This wreaked all kinds of havoc until I got the remote dc recovered, due to the services ct was (re)starting.

Wouldn't an infinite retry block the template from rendering anything? In that case, it unnecessarily couples the datacenters together, causing an outage of one to pause config generation in another.

[anthcourtney]

At present the way we're working around this is to use a plugin which checks whether the datacenter has consensus, and if it doesn't, skips the datacenter and continues enumeration.

{{ range $dc := datacenters -}}
{{ if plugin "bin/dc-has-leader" $dc | parseBool -}}
dc: {{ $dc }}
{{- end -}}
{{- end -}}

See https://github.com/anthcourtney/golang-consul-dc-has-leader for an example plugin + usage.

[sethvargo]

Hi all,

Thank you for participating on this issue. Unfortunately this is not something that is efficient given Consul's current API. Allow me to explain.

First, it's possible to list datacenters, but as you can see, the response from that API call is a JSON list of strings corresponding to the datacenter names. The datacenters endpoint does not provide additional health information, so there's already a requirement to make an additional API call to determine if a datacenter is healthy. This introduces an N+1 operation. Given the relatively limited number of datacenters most clusters will have, I'm not _too_ concerned with this.

The second challenge is that there is no API which enables us to query for the status of a datacenter. Consul exposes a status API, but that API can only be run from an agent _within_ a datacenter; it's not possible to query the raft/leader/health information from an agent in a different datacenter through the HTTP API.

The only possibility is to use catalog services, but there are still performance implications to consider. As such, this needs to be an optional, off-by-default, setting. I've submitted #968.

[sukujgrg]

Thanks for adding #968