Consul-template: Issue with supplying only content(s) in the template {} block of config.hcl(json)

[caleyg]

Consul Template version

consul-template v0.24.1 (58aa6c6)

Configuration

 {"vault": [{"ssl": [{"verify": true}]}, {"renew_token": true}], "consul": {"ssl": [{"verify": true}]}, "log_level": "debug", "template": [{"content": "{{- /* dummy */ -}}"}, {"command_timeout": "300s"}], "exec": {"env": [{"custom": "PATH=$PATH:$PWD"}]}}

Command

 consul-template -config consul-config.json -template common.ctmpl -once -dry
2020/02/28 14:30:18 [ERR] (cli) from file: consul-config.json: mapstructure decode failed: 1 error(s) decoding:

* 'template[0]' has invalid keys: content

Debug output

https://gist.github.com/caleyg/c756759c39576f77f2f8d32de1bbbdcb

Expected behavior

I expect to be able to supply either source or content in the template {} map of the config.

Actual behavior

I am trying to specify either source or content(s) and I'm getting an error from consul-template when specifying only content(s) and no source.

Steps to reproduce

1. dl consol-template 0.24.1
2. add the config.json to your path
3. run the command above.

[caleyg]

this is even happening when supplying only config files to the command line and using the config.json to drive the show

3/2/2020 5:39:26 PM{"consul": {"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, "vault": [{"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, {"renew_token": true}], "exec": {"env": [{"custom": "PATH=$PATH:$PWD"}]}, "template": [{"source": "/etc/some_other_file.yml.ctmpl"}, {"destination": "/etc/some_other_file.yml"}, {"error_on_missing_key": true}, {"command_timeout": "300s"}], "log_level": "debug"}
3/2/2020 5:39:26 PM{"consul": {"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, "vault": [{"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, {"renew_token": true}], "exec": {"env": [{"custom": "PATH=$PATH:$PWD"}]}, "template": [{"source": "/app/index.html.ctmpl"}, {"destination": "/app/index.html"}, {"error_on_missing_key": true}, {"command_timeout": "300s"}], "log_level": "debug"}
3/2/2020 5:39:26 PM['-config=/tmp/some_other_file.yml-consul-template-config.json', '-config=/tmp/index.html-consul-template-config.json']
3/2/2020 5:39:26 PMLaunching application with the following: consul-template -config=/tmp/some_other_file.yml-consul-template-config.json -config=/tmp/index.html-consul-template-config.json -exec='python3 test-server.py' .
3/2/2020 5:39:26 PM2020/03/02 23:39:26.987907 [INFO] consul-template v0.24.1 (c54d0abc)
3/2/2020 5:39:26 PM2020/03/02 23:39:26.987924 [INFO] (runner) creating new runner (dry: false, once: false)
3/2/2020 5:39:26 PM2020/03/02 23:39:26.988272 [DEBUG] (runner) final config: {"Consul":{"Address":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"python3 test-server.py","Enabled":true,"Env":{"Blacklist":[],"Custom":["PATH=$PATH:$PWD","PATH=$PATH:$PWD"],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"debug","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"/etc/some_other_file.yml.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/etc/some_other_file.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"","ErrMissingKey":true,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":300000000000,"Contents":"","CreateDestDirs":true,"Destination":"","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":300000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"/app/index.html.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/app/index.html","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"","ErrMissingKey":true,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""},{"Backup":false,"Command":"","CommandTimeout":300000000000,"Contents":"","CreateDestDirs":true,"Destination":"","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":300000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionBlacklist":null,"SandboxPath":""}],"Vault":{"Address":"https://vault","Enabled":true,"Namespace":"","RenewToken":true,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false}
3/2/2020 5:39:26 PM2020/03/02 23:39:26.988417 [INFO] (runner) creating watcher
3/2/2020 5:39:26 PM2020/03/02 23:39:26.988580 [DEBUG] (watcher) adding vault.token
3/2/2020 5:39:26 PM2020/03/02 23:39:26.988782 [ERR] (cli) template: must specify exactly one of 'source' or 'content'
3/2/2020 5:39:26 PM/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'vault'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
3/2/2020 5:39:26 PM  InsecureRequestWarning,
3/2/2020 5:39:26 PMTraceback (most recent call last):
3/2/2020 5:39:26 PM  File "docker_entrypoint.py", line 363, in <module>
3/2/2020 5:39:26 PM    main()
3/2/2020 5:39:26 PM  File "docker_entrypoint.py", line 357, in main
3/2/2020 5:39:26 PM    exec_app(args)
3/2/2020 5:39:26 PM  File "docker_entrypoint.py", line 216, in exec_app
3/2/2020 5:39:26 PM    subprocess.run(startup_command, shell=True, universal_newlines=True, check=True)
3/2/2020 5:39:26 PM  File "/usr/local/lib/python3.7/subprocess.py", line 512, in run
3/2/2020 5:39:26 PM    output=stdout, stderr=stderr)
Unknown Date
3/2/2020 5:39:26 PMsubprocess.CalledProcessError: Command 'consul-template -config=/tmp/some_other_file.yml-consul-template-config.json -config=/tmp/index.html-consul-template-config.json -exec='python3 test-server.py'' returned non-zero exit status 14.

[eikenb]

Hey @caleyg, thanks for the report and I think I found your issue.

In your initial example you use "content" in your config file to specify the template when the needed field is "contents" (plural). After making that change it ran without error for me.

Please let me know if that resolves things for you.

[caleyg]

Hi @eikenb thanks for looking, I did notice that but then I attempted to try just providing source see this paste here https://github.com/hashicorp/consul-template/issues/1349#issuecomment-593680697 and it seems to also complain if just providing source (no contents) thoughts?

{"consul": {"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, "vault": [{"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, {"renew_token": true}], "exec": {"env": [{"custom": "PATH=$PATH:$PWD"}]}, "template": [{"source": "/etc/some_other_file.yml.ctmpl"}, {"destination": "/etc/some_other_file.yml"}, {"error_on_missing_key": true}, {"command_timeout": "300s"}], "log_level": "debug"}
{"consul": {"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, "vault": [{"ssl": [{"ca_cert": "/etc/ssl/certs/ca-certificates.crt"}, {"verify": true}]}, {"renew_token": true}], "exec": {"env": [{"custom": "PATH=$PATH:$PWD"}]}, "template": [{"source": "/app/index.html.ctmpl"}, {"destination": "/app/index.html"}, {"error_on_missing_key": true}, {"command_timeout": "300s"}], "log_level": "debug"}

[eikenb]

Those examples are don't work as the template section is formatted incorrectly. You have..

   "template": [
        {
            "source": "/etc/some_other_file.yml.ctmpl"
        },
        {
            "destination": "/etc/some_other_file.yml"
        },
        {
            "error_on_missing_key": true
        },
        {
            "command_timeout": "300s"
        }
    ],

Where consul-template is looking for (in JSON)..

    "template": [
        {
            "source": "/etc/some_other_file.yml.ctmpl",
            "destination": "/etc/some_other_file.yml",
            "error_on_missing_key": true,
            "command_timeout": "300s"
        }
    ]

Or (in HCL)..

template {
  source = "/etc/some_other_file.yml.ctmpl"
  destination = "/etc/some_other_file.yml"
  error_on_missing_key = true
  command_timeout = "300s"
}

[eikenb]

You get the "source or contents" error with your version as it tries to process each of the {} sections as a separate template. So when it gets to the second one, with only "destination", it dies with the error as that section has no source or contents.

[caleyg]

thank you @eikenb totally a user error on my part! I corrected my objects and it started fine!

[eikenb]

Great to hear @caleyg. Glad I could help out.