Telegraf: Hide passwords in Telegraf config

[dpozhidaev]

Our security unhappy if we keep using passwords in Telegraf config as they are.
Could you please make Telegraf to store any passwords in config in more secure manner?

[danielnelson]

Any thoughts on what this would mean for your organization concretely? Are you currently using any secret storage? I found this link with some discussion https://news.ycombinator.com/item?id=10927043

[phemmer]

The problem is that as long as you want to be able to start the service automatically (such as on boot, or on failure), there is no feasible way to hide the passwords. Even with a secret storage service, the application has to be able to automatically pull down the secrets, which means its no more secure than putting them in a file.

The only way to do it is to require an admin to unlock the secrets at the time the service is started. This is generally how encrypted private SSL keys work.
And if this is acceptable, you could just have telegraf read the config from STDIN, and paste it in manually when you start it.

[mathieulongtin]

It's one thing to have the password readable by telegraf, it's another to have them in configuration files.
Maybe have a way for to put a replacement in strings. For example, I put my secrets in files, I would do this:

[agent]
secret_reader="cat ./secrets/{name}"

Later, in any string:

servers = [ "tcp://:{secret:serverpass1}@localhost:1234"]

{secret:serverpass1} will get replaced by whatever cat ./secrets/serverpass1 outputs. This would work trivially with any secret manager. If a connection fails, secrets are refreshed, otherwise, they can be cached.

[phemmer]

You can already do that using environment variables. Env vars get substituted within the config. See http://docs.influxdata.com/telegraf/v1.7/administration/configuration/#environment-variables
Just have whatever you use for an init system load your env vars from a file prior to launching telegraf.

[mathieulongtin]

Except in that case, you have to restart telegraf in case of password change. Plus, you may need a ton of them. It's still better than nothing.

[prydin]

We've been experiencing similar concerns about the vSphere plugin requiring plaintext passwords. I like the approach @mathieulongtin takes, since it's trivial to adapt to almost any password storing or secret management mechanism.

If @danielnelson and @glinton approve this or some other mechanism, I may be able to help implement it.

[danielnelson]

So there are two concerns so far above, one is a security concern and the other is a desire to reload configuration options without restarting Telegraf. I think there is one other issue that is about segregating the passwords and storing them in a single shared location.

From a security point of view, I don't think the above helps. Telegraf would need permission to run the command and so it would be no different from reading the config file.

When it comes to not wanting to reload, the way I'd really like to solve this is by improving the reload flow to keep the output buffers and not interrupt running plugins that are not removed. This is quite complicated to do right and involves some work on top of the [tgconfig] experiment I worked on last year.

If the desire is to segregate passwords from the main config I recommend @phemmer method, create a file with environment variables, and then source it when starting the service. If you are using init you can source the file in /etc/defaults/telegraf and with systemd use the existing or add an additional EnvironmentFile.

If we improve reload, it won't allow environment variables to change of course, but I have been thinking adding some additional syntax to the config file to allow reading values from a file without needing multiple options. The design is inspired by curl and some other commands that use a @ to indicate it should be read from the file:

# set option using static string
bearer_token = "abcdef"
# or reading from a file
bearer_token = "@/etc/telegraf/bearer"

But what if you want to integrate with another secret store? The syntax above wouldn't solve that, and is probably a good argument against adding it. I think we could provide a layer similar the loader layer in the [tgconfig] experiment, and we might be able to leverage this to do some discovery tasks as well, but I think it is a large amount of work which I don't have time right now to experiment with. I think I would rather not make this addition until the config experiment can be integrated into Telegraf with the functionality it has now.

One more thing about the vSphere plugin in particular, I think the best action we could take for security would be to document a way to setup a limited privilege account that can be used to monitor the server.

[puckpuck]

Merging my comments from #5192 into here to keep a single conversation. The idea of loading via environment variables makes sense. Can this be documented with examples? For Windows I suspect this is not as easy, and perhaps we need to look at other methods like wrapper scripts, and clearing of the environment variables either after Telegraf initializes config, or via a short sleep within a wrapper script. But documenting all this with examples goes a long way.

Longer term is something that can be better adopted by enterprise security, such as a plugin framework for secrets. tgconfig as a whole looks great for other reasons, especially the potential pre-cursor to dynamic configs 💯

[danielnelson]

I can improve the docs, current ones are here but they are lacking, I'll update this section. There is also a small example in the default config:

[global_tags]
  ## Environment variables can be used as tags, and throughout the config file
  # user = "$USER"

Environment variables can't be cleared AFAIK, they are available in /proc/$pid/environ and are cannot be changed in process, this file is usually restricted to the telegraf user. Any wrapper script could just exit or start telegraf using exec.

I think the tgconfig method could be a complete solution if you store the entire config or config section in secret storage. This frees us from cobbling together the config from several sources, which may be a better way to go about this due to how much it simplifies the config loading. The downside is that you might have to update many configs when a password is changed. I wonder if any secret stores have a method to deal with dependent values?

[phemmer]

Environment variables can't be cleared AFAIK, they are available in /proc/$pid/environ and are cannot be changed in process, this file is usually restricted to the telegraf user.

This isn't true, but it does raise a concern. Any variables not cleared will become visible to programs (scripts) executed by telegraf.

But on the clearing thing, yeah variables can be cleared from the running process. Variables are just stored as a contiguous block of memory (null delimited) which you can overwrite just like any other block of memory owned by the process. Go exposes this through os.Unsetenv().

[danielnelson]

Yeah its definitely true that you can unset a variable to remove it from the processes view and "unexport" it, and you may or may not want this behavior, but this got me wondering about the value in/proc, so I slapped some code together:

func main() {
    pidstr := strconv.FormatInt(int64(os.Getpid()), 10)
    octets, _ := ioutil.ReadFile("/proc/"+pidstr+"/environ")
    fmt.Println(string(octets))
    os.Unsetenv("FOO")
    octets, _ = ioutil.ReadFile("/proc/"+pidstr+"/environ")
    fmt.Println(string(octets))
}

Then ran FOO=bar go run foo.go and the value is still set after both reads. Is there a way this file would be modified?

[phemmer]

I think somewhat recall there being some caveat about the way go manipulates its environment, in that it just works on a local copy.

That file is just a virtual map of the address space holding the env vars. I don't recall the way to do it from go, but if you can do the equivalent of C's getenv() or environ, then you should be able to modify it.

[jaroslavdextems]

Also, Encoding of .config fileds is one of possible ways. But we need to define which tags you want to encode on first launch.

For example if you want to encode special field add prefix (for example *) before its name: *password=JGVuY3J5cHRlZHBhc3N3b3Jk instead of password = password

Or simplest to implement: *encoded-field-name = base64-value

[endersonmaia]

I see other projects that can be configured via environment variables for running as a container using _FILE suffix for its variables.

I think it could be done detecting these suffixes and working accordingly reading the file's content instead of the variables content, or adding additional configurations for the plugins like password and password_file.

I you use docker secrets within docker swarm, you could pass the secrets to the container and use the contents of /run/secrets/<secret_name>.

[endersonmaia]

You could pass all the contents of telegraf.conf to a docker secret and load it with telegraf --config /run/secrets/telegaf.conf, I didn't tested it yet, but should work.

[luanhdeandrade]

Why not safeguard the secrets using a keystore as elasticsearch does? You can define the passwords safely into a key store and label them as a parameter name and in the configuration file you reference the keystore parameter name instead of the hardcoded password. the keystore is stored in the machine itself, but it gives you no way to retrieve the passwords you saved in it.
Another possibility is to make telegraf be compatible to vault applications or any other parameter store solution, such as AWS Systems Manager Parameter Store.
In this way, only the application will have access to the real password during its execution.