Alertmanager: Feature request: Make all config secrets templatable

[Wenzil]

Currently, we can't securely put alertmanager config in source control because secrets such as email smpt_auth_password must be in-lined in the config file.

Some configs like pagerduty service_key don't have the issue because they are templatable. Templates can easily be mounted individually as secrets without being part of source control, and then "referenced" by the config file.

My suggestion is to make all config secrets (documented here) tmpl_secrets instead. This would allow us to put the config file in source control without in-lining any secrets.

[brian-brazil]

This is not recommended on security grounds: https://prometheus.io/docs/operating/security/#alertmanager

Secrets used for routing are templatable in spite of being secrets.

[Wenzil]

What if we have restrictions on who can deploy the config to the alertmanager server?

Then a user who can edit the config file in source control needs approval to get their changes through. So they can't send themselves the templated secrets without another user approving the changes.

Or am I understanding this wrong?

[roidelapluie]

I'd suggest to adopt the practice of being able to source secrets from other file, like we do in prometheus (password_file).

[brian-brazil]

Being able to read certain secrets from files isn't anything to do with avoiding putting secrets in files - they're still in files after all, and nothing prevents the usual pattern of checking in the config file with a placeholder and interpolation before laying it down.

Reading secrets also from separate files is supported only where it is common that the secret is updated/rotated by an automated process that's completely independent of the regular config file reload cycles (e.g. kubernetes bearer tokens), which I think is safe to say is not the case for any of the secrets that we're talking about in the alertmanager which are all very static with any rotation being rare and manual. In addition having only some templating fields come from separate files would be a bit weird and confusing, kinda similar to how users confuse some secrets being templatable for being a security feature rather than a risk.

[atomic77]

The generic http_config block uses the password_file convention @roidelapluie referred to which is a convenient way of providing the basic_auth credentials in a secret for http-based alerts. It would be nice to be able to provide SMTP credentials in the same way.

@brian-brazil would you have any objection to the addition of a parameter like smtp_auth_password_file that would be mutually exclusive with smtp_auth_password? I'd be open to digging into the code and providing a PR.

[brian-brazil]

What's your reason for wanting to add it? I've already covered how we approach these things in my above comment.

[atomic77]

For consistency with the way we use secrets to pass basic_auth credentials for http-based clients. It's not obvious to me why SMTP credentials would be treated differently?

[brian-brazil]

That was already answered in https://github.com/prometheus/alertmanager/issues/1704#issuecomment-679870168 above.

[atomic77]

Reading secrets also from separate files is supported only where it is common that the secret is updated/rotated by an automated process that's completely independent of the regular config file reload cycles (e.g. kubernetes bearer tokens), which I think is safe to say is not the case for any of the secrets that we're talking about in the alertmanager

If you're referring to this, as best as I can understand, you seem to be grouping credentials into "dynamic" and "static" buckets to justify treating them differently.

Even if it were appropriate to lump plain basic_auth credentials into the "dynamic" bucket, it's not clear to me why alertmanager should think about them differently. What if I am deploying to an environment where the password for the SMTP service account is rotated on a daily basis? This distinction imposes AM's idea of what are "static" credentials, and makes it more inconvenient to use.

[brian-brazil]

What if I am deploying to an environment where the password for the SMTP service account is rotated on a daily basis?

If you were, you would have provided that as your argument. However that is not the argument you provided, nor have I ever heard of such a setup. I prefer to base design decisions on users' actual use cases, not hypotheticals as those could be used to justify anything. I've already explained the reasoning, I don't feel the need to get involved in an further argument around this point.

[roidelapluie]

I do agree with Brian, and I would also add that you can safely use the tools provided by your config management to store secrets in git, e.g. I am using ansible-vault for this purpose. This seems a sufficient workable solution to me to address the original poster's concern.

[atomic77]

I prefer to base design decisions on users' actual use cases, not hypotheticals as those could be used to justify anything.

Do you think I would be spending my time posting on this issue, and even offering to even help code an enhancement, if this were just a hypothetical scenario that I dreamed up? Frequent rotation of credentials for production systems that you think of as "static" are a common practice, and it's a trend that will continue as secrets management in platforms like k8s makes it easier to do.

I hope you will reconsider this impediment to the use of AM in production scenarios at some point in the future, and I would be glad to help in whatever way I can. Thanks for your time.

[roidelapluie]

It seems that all sides have been heard here, but anyway, the scope is beyond alertmanager as we'd like to keep this consistent within all the projects.

Can you please contact the Developers mailing list about your proposal: https://groups.google.com/g/prometheus-developers where more people will be able to discuss this matter, and we can adopt the same point of view in all the Prometheus projets.

Thanks!