Consul-template: Make finding missing key easier

[yellowmegaman]

Hi there!
I have a template with 4 keys, one is missing, since i didn't create it in consul (metrics/influx_test).
Problem is, when i have template with 30+ keys and some are missing, thats really hard to find out what key it is.

Configuration

Consul-template 0.19

template {
  source = "/opt/consul-template/app.conf"
  destination = "/etc/app/app.conf"
}

backend.influxdb.host = "{{key "metrics/influx_host"}}"
backend.influxdb.name = "{{key "metrics/influx_db"}}"
backend.influxdb.port = "{{key "metrics/influx_port"}}"
backend.influxdb.test = "{{key "metrics/influx_test"}}"

Command

./consul-template --log-level=debug -once -template test:test.result

Debug output

https://gist.github.com/yellowmegaman/6ce8905643617aee445eebbe0564b10e

Expected behavior

See message is still needed only for missing keys at the end.

Actual behavior

All keys are missing, have to check one by one.

Steps to reproduce

Command, config and template from above.

Thanks a lot in advance!

[folex]

Experienced very same problem yesterday, had to hot-refactor all configs to use
{{ keyOrDefault "keyName" "keyName_MISSING" }}
for ALL keys so I can detect a typo in key name.
It was really _a pain_.

[sethvargo]

Hi @yellowmegaman @folex

This information is exposed in trace, not debug mode. If you run with trace, you'll see information like:

# ...
2017/07/05 14:31:29.601752 [INFO] (runner) initiating run
2017/07/05 14:31:29.601767 [DEBUG] (runner) checking template f0ea867d0c7c950ea6925c608bd20c82
2017/07/05 14:31:29.602111 [DEBUG] (runner) was not watching 1 dependencies
2017/07/05 14:31:29.602129 [DEBUG] (watcher) adding kv.block(foo)
2017/07/05 14:31:29.602134 [TRACE] (watcher) kv.block(foo) starting
2017/07/05 14:31:29.602141 [DEBUG] (runner) diffing and updating dependencies
2017/07/05 14:31:29.602151 [DEBUG] (runner) watching 1 dependencies
2017/07/05 14:31:29.602202 [TRACE] (view) kv.block(foo) starting fetch
2017/07/05 14:31:29.602232 [TRACE] kv.block(foo): GET /v1/kv/foo?stale=true&wait=1m0s
2017/07/05 14:31:29.603202 [TRACE] kv.block(foo): returned nil <-- **RIGHT HERE**
2017/07/05 14:31:29.603212 [TRACE] (view) kv.block(foo) marking successful data response
2017/07/05 14:31:29.603218 [TRACE] (view) kv.block(foo) asked for blocking query
2017/07/05 14:31:29.603233 [TRACE] kv.block(foo): GET /v1/kv/foo?index=1&stale=true&wait=1m0s
2017/07/05 14:31:29.603238 [TRACE] view kv.block(foo) successful contact, resetting retries
^C2017/07/05 14:31:33.213183 [DEBUG] (cli) receiving signal "interrupt"
Cleaning up...
2017/07/05 14:31:33.213204 [INFO] (runner) stopping
2017/07/05 14:31:33.213209 [DEBUG] (runner) stopping watcher
2017/07/05 14:31:33.213214 [DEBUG] (watcher) stopping all views
2017/07/05 14:31:33.213222 [TRACE] (watcher) stopping kv.block(foo)

There's no easy way to surface these because you might want to watch a key that does not exist, perhaps to trigger a feature flag. You can see in this issue, users wanted the opposite behavior you describe here, which is why we moved the warning to TRACE.

[yellowmegaman]

@sethvargo Thanks a lot!
Forgot to post quick and dirty solution written in bash, accepts only one argument - path to template that needs to be checked.

#!/bin/bash
template=$1
time=$(date +%s)
consul-template -log-level=trace -once -template "$template:/tmp/output$time" &> "/tmp/ebalog$time" &
ctpid="$!"
sleep 3

if kill -0 "$ctpid" &> /dev/null; then
        kill -9 "$ctpid"
        wait $! 2>/dev/null
fi

cat "/tmp/ebalog$time" | grep 'TRACE' | grep -w 'returned nil' | awk '{print $4}' | tr -d ':'