Prometheus: Support for environment variable substitution in configuration file

[dopuskh3]

I think that would be a good idea to substitute environment variables in the configuration file.

That could be done really easily using os.ExpandEnv on configuration string when loading configuration string.

That would be much easier to substitute environment variables only on configuration values. go -ini provides a valueMapper but yaml.v2 doesn't have such mechanism.

[brian-brazil]

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

[alexellis]

This could be a particularly powerful way to configure Prometheus in a cluster without the burdensome task of a separate config file for every configuration you can ever deploy.

[nutchalum]

It's a nice to have.
Now I've encountered a problem with external_label.
I've deploy prometheus Deployment and ConfigMap via kubernetes federation.
All of them have the same configuration, except external_label.
I can't think a way to injected this external_label (like source: clusterA)
So I have to deploy ConfigMap in the old way.
(I have around 7 clusters, that means 7 times instead of 1 time via Federation)

[Nick-Triller]

Environment variable substitution would allow to keep sensitive data like passwords used for remote read/write out of the config file, too.

InfluxData recently mentioned environment variable substitution in config files would be a useful feature in their eyes.

[ababushkin]

I don't think it would cause usability issues guys, its pretty common to use environment variable substitution in config files like these. Any chance to revise that decision?

Here are some benefits:

• Keeps sensitive configuration out of config files
• Simplifies build / deployment process, especially in containerised environments
• Inline with 12factor principles (https://12factor.net/)

Am I missing something here?

[boeboe]

Any update on this?

Related to https://github.com/prometheus/prometheus/issues/2664

Eventually we just ended up having separate configuration files per cluster, which was annoying since it was an exception among other services deployed within our clusters.

PS: another option is to provide small wrapper script doing this cluster specific substitions (_sed -i prometheus.yml_ like) before actually starting prometheus. You will need a proper signal handling mechanism like dump-init however (https://github.com/Yelp/dumb-init), since you're prometheus will not be running as PID 1 anymore in this case.

# Install dumb-init
RUN curl -L https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_amd64 >/usr/bin/dumb-init &&  chmod +x /usr/bin/dumb-init 
...
ENTRYPOINT ["/usr/bin/dumb-init", "/entrypoint"]
CMD ["/bin/prometheus", "-config.file=/etc/prometheus/prometheus.yml", \
         "-storage.local.path=/prometheus", \
         "-web.console.libraries=/etc/prometheus/console_libraries", \
         "-web.console.templates=/etc/prometheus/consoles"]

... and within your entrypoint script:

#!/usr/bin/env bash
export CLUSTER_NAME=`grep search /etc/resolv.conf | awk '{print $NF}'`
# assuming your ConfigMap with prometheus.yml has a jinja style {{CLUSTER_NAME}} string for substitution...
sed -i "s/{{CLUSTER_NAME}}/${CLUSTER_NAME}/g" /etc/prometheus/prometheus.yml
exec "$@"

The disadvantage of this approach is you're introducing an external component dumb-init, breaking single process per container best practices, which some might not be willing to do.

[bwplotka]

We have similar issues when we run Prometheus in 2-replica statefulset which reuses single configmap. To support HA we have external_label called replica that suppose to have unique replica name (e.g hostname). We end up doing this:

<stateful-set>
...
spec:
      containers:
      - args:
        - -c
        - sed s/'@@HOSTNAME@@'/${HOSTNAME}/g /etc/prometheus/prometheus.yaml.tmpl
          > /etc/prometheus/prometheus.yaml && /prometheus/prometheus --config.file=/etc/prometheus/prometheus.yaml
          --log.level=info --query.max-concurrency=100 --query.timeout=2m --storage.tsdb.path=/prometheus-data
          --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h
          --storage.tsdb.retention=12h --storage.tsdb.no-lockfile --alertmanager.timeout=1ms
        command:
        - /bin/sh
...

This is not fun and this solution _screams_ for better approach. But I feel like this might be rather Kubernetes issue not enabling that, not necessarily Prometheus one.

[bwplotka]

Maybe moving external_labels to flags would at least fix that case?

[brian-brazil]

&& /prometheus/prometheus

A small tip: When you are using the shell like this where application is the last thing you are running do && exec /prometheus/prometheus so that the application replaces the shell process. This tends to make signal handling easier.

I'm unsure of the wisdom of using a hostname as an external label, that's going to make any remote storage endpoint have labels vary as Prometheus moves around over time. Two separate statefulsets with different label values like mydc1 and mydc2 might be better.

[bwplotka]

Cool, thanks for the advice!

Our host name is actually pod name so I think this is fine (: Won't change easily.

[fabxc]

Yea, the name for stateful sets is deterministic in contrast to deployments, where pods get a random suffix. So it will always be prometheus-infra-0, prometheus-infra-1 etc.

[liyaka]

I need to configure azure_sd_configs with sensitive info like client_secret and etc.
Since I keep my configuration in git, there is no option this info can appear in configuration file as is...
Naturally I would think of using environment variables and set them while deploying with Ansible.
What would be the best way to do it?..

[brian-brazil]

Naturally I would think of using environment variables and set them while deploying with Ansible.

Environment variables are a bad idea for secrets. Ansible has its own secret and templating support, which is the standard way to approach this.

[abh]

FWIW, the prometheus-config-reloader for prometheus-operator looks to have implemented a solution for this for the statefulset use case recently:

https://github.com/coreos/prometheus-operator/pull/1132

https://github.com/coreos/prometheus-operator/commit/608e71422fa6c26e869d762dc0a269559430a0c8#diff-a2dba0afa9edde401541fc9054415bba

[bwplotka]

If you use Thanos you have that out-of-the-box as well: https://github.com/improbable-eng/thanos/blob/master/cmd/thanos/sidecar.go#L73

[orubel]

One reason to have this is so that you can have better devops for SWARMS

If you are doing lots of builds, you want to be able to template your SWARMS and this does NOT make that possible. I build 4-50 node swarms constantly based on customer demands and being able to dynamically configure them would be IDEAL and save me hours.

[dbendelman]

B"H

We solved this by creating an entrypoint script that preprocesses environment variables before calling the binary. Example showing how to set --log.level:

# Prometheus
docker run -d -p 9090:9090 -e PROM_LOG_LEVEL=error hiveco/prometheus:v2.2.1

# Alertmanager
docker run -d -p 9093:9093 -e ALERTMGR_LOG_LEVEL=error hiveco/alertmanager:v0.15.0-rc.1

Here are the Prometheus and Alertmanager images, with sources linked there. Multiple tags are available depending on the version you need. These are super bare-bones wrappers around the official images, simply working around this particular issue and then getting out of the way.

The Prometheus authors made a great product and are making sound arguments for not including this functionality natively. The above images, however, might help people whose infrastructures or policies allow for taking a more relaxed approach.

[djbingham]

Environment variables are a bad idea for secrets.

What about using Docker secrets? My understanding is the expected way of controlling secret values in containers is to add a Docker secret, which generates a file inside your container, then within the container read that file into environment variables at runtime.

[brian-brazil]

@djbingham That is still putting secrets in environment variables.

[djbingham]

@brian-brazil Yes it is, but I'm led to believe (from the official documentation for Docker) that this is the expected way to handle such things. If that is actually a bad idea, please could you elaborate on why?

[brian-brazil]

See http://movingfast.io/articles/environment-variables-considered-harmful/ and https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ for the reasoning. Environment variables were never designed with secrets in mind, nor are they traditionally a place to put secrets.

[djbingham]

Good explanations, thanks.

[dbsanfte]

Secrets are being used as an excuse to avoid adding some pretty basic and expected functionality, methinks.

http://docs.grafana.org/installation/configuration/#using-environment-variables

Justify what's wrong with that, as not just Grafana but NPM and others have the exact same regime of overrides.

[fljmayer]

So we have to put environment-specific proxy settings into a configuration file that can otherwise be kept environment-independent? That doesn't seem like good solution.

[sgeisbacher]

Environment variables are a bad idea for secrets.

@brian-brazil but storing them statically in config-files in your git-repo is better? 👎 👎 👎

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

@brian-brazil but running "config-pre-render-scripts" to set aws-secret-key in different docker-deployments increases simplicity? 👎 👎 👎

imho sometimes you have no clue what you are talking about, sorry.

[Jake-Smullin]

Env variables make so much sense here... Can't check in my AWS creds for an ec2_sd scrape job, and mounting them in as secrets to ENV vars and then leaving in the scrape config $ACCESS_KEY for example would be perfect.

[speedooo]

Is it possible use the same way as file_sd_configs reading details fromm external file for external_labels?just read key-value pair

[mohr-michael-adam]

Is it safe to launch Prometheus (or AlertManager) then erase the config file after it is running? This way the sensitive data only exists on disk in plaintext for a brief moment. Not claiming this to be secure but it helps.

[logemann]

I think its obvious for several reasons to have that feature. For me, i want to have a more dynamic way of definiting ports of targets depending on the docker-compose.yml file. But i could think of thousands more.

[zakkg3]

This should be a must-have, not a nice to have.
@brian-brazil what's the problem on having 2 different ways of configuring things? Like ELK, grafana, and.. probably most of the things :D

[logemann]

@zakkg3 you wont get any positive feedback from the author regarding that. As you see in my closed reminder case. Sad but true. He dont mind that any other framework/product has such a mechanism.

[zakkg3]

I see @logemann I ve made a custom image with confd to workaround, feel free to fork:
https://github.com/zakkg3/Prometheus-confd

[nmaupu]

I am also surprised not to be capable of doing that. I would rather prefer using http_proxy environment variable instead of having to put this in global/http_config/proxy_url while it's already available and configured elsewhere. Those configurations are environment dependent and should be easily replaced in the same base code...

config:
  global:
    http_config:
      proxy_url: ${http_proxy}

would be much much better.

By the way, _alertmanager_ doesn't seem to use http_proxy and no_proxy standard environment variables, am I missing something here ?

[Qix-]

@brian-brazil

We've previously discussed several times and decided against. For simplicity we keep to one way to configure a given thing, otherwise we'd end up with 4-5 ways.

You're encouraging poor security practices by keeping credentials on the filesystem during e.g. Dockerfile-based builds. Not being able to configure secrets via environment variables is going to cause vulnerabilities in production systems.

It's quite sad it's not available.

[brian-brazil]

See https://github.com/prometheus/prometheus/issues/2357#issuecomment-388753562 above, putting secrets in environment variables is not a good security practice.

[Qix-]

@brian-brazil You're basing that off one article that has a bunch of holes in it.

You can easily see environment variables using procfs, sure. They are also inherited to child processes, sure. But this is assuming you're running untrusted code in production in the first place. Most people that know what they're doing, or have proper QA processes in place, are not doing this.

Assuming a container has been compromised means that any form of secret storage is vulnerable. The only alternative is to somehow detect that the production application has consumed the credentials and to dispose of them - which means you have to trust the application to also dispose of them in memory.

That's a tall order for most software today, and highly unrealistic in terms of threat modeling - to require _that degree_ of secrets handling isn't necessary in what I'd consider the "common case" for most production cases. The threat model generally takes into account the method of secret handling.

Storing credentials in a repository is a non-starter. That violates federal regulations under which we operate. I assume the same applies to many people.

Storing credentials in k8s secrets doesn't work because we're stuck with volume mounts or environment variables, both of which you're staunchly against.

As you've mentioned elsewhere, you are now expecting us to write glue code that initializes the configuration _at runtime_ as an added step using a substitution method (e.g. templates) to pull in the secrets from volume mounts or environment variables _anyway_ in order to build up prometheus configuration.

That's insane.

It isn't your job to dictate _my system's_ threat model. It's my job. That's what I was hired to do.

You have an army of users asking you for this feature. You're declining for a dogmatic reason that contradicts many of the points made against both your reasoning as well as against the articles you yourself linked to back up your viewpoint.

Not having this feature is getting in the way of my work, and causing me _more security overhead_ than not having it as I have to write configuration scripts which implies error surface area and liability. Plus, it's another piece of code I have to have audited.

Please reconsider. Even if you put a warning label in big, red capital letters next to the documentation for environment configuration, whatever it takes.

---

EDIT: some more points:

But sometimes software dumps out its environment in times of debugging or failure.

No it doesn't. I've yet to see a production-ready piece of software recklessly dump the _entirety of its environment map_ to stdout/err/file/etc. when it dies. Not only is it needless, it doesn't do anything to help developers anyway. Rarely are problems _in production_ related to the environment, especially with containers.

But child processes can maliciously access the parent process's secrets if they're exposed via environment variables.

This is assuming you're running software that runs child processes, and those child processes aren't also trusted. Compartmentalized security, credential auditing, rotation, codebase auditing, firewall rules - all of these things that should be happening will combat this.

But writing to the filesystem prevents all of that for free.

Again, this is only a net benefit if you are worried about the process somehow transmitting the credentials - which, in most cases, is not something you would worry about. Plus, if it's running in the container, you should be modeling the security as inherently compromised by the child process anyway - therefore, the assumption is that the child process can still read the filesystem just fine.

But what about UNIX permissions?

Sure, I guess. It's still a weak argument against proper security modeling, and only increases the chance for mis-configuration from within the container itself.

There's still the problem of secrets storage and transportation into the container or machine itself, too. That's fixed by exactly none of this.

Also, it's better if the application disposes of the credentials from memory too, which means you can unlink the credentials from the filesystem.

Sounds great - except not even prometheus can realistically do this. It runs based on HTTP + REST, which means it has to keep any URIs or bearer tokens in memory, which also means that it can't feasibly dispose of them if it wants to make more than one read/write.

[nmaupu]

Using environment variables for configuration is not necessary to pass critical secrets to a container. My use case (using HTTP_PROXY env vars) is not really secret, it is pushed via a podpreset and it would be neat to be reused instead of repeating the code elsewhere ...

[alexellis]

On the OpenFaaS project we are also advising users not to store secrets in environmental variables, but to use secrets in files instead.

I saw that the configuration for HTTP alert endpoints now supports reading credentials from a file at runtime. This would be ideal for other secrets too.

https://github.com/prometheus/prometheus/issues/3639

[Qix-]

On the OpenFaaS project we are also advising users not to store secrets in environmental variables, but to use secrets in files instead.

Why? How is that any better?

[fvclaus]

Wow. This is pretty ignorant. I am running prometheus in a docker container. To force my environment variables in, I modified the entrypoint of the official image and run this script instead:

#!/bin/ash

set -e

cp /etc/prometheus/prometheus-template.yml /etc/prometheus/prometheus.yml

env |
while IFS='=' read -r name value
do
  # Convert to lowercase: FOO -> foo
  name=$(echo $name | tr '[:upper:]' '[:lower:]')
  # Replace sed seperator character / -> \/
  value=$(echo $value | sed 's;/;\\/;g')
  # Replace occurrences in file.
  sed --in-place=.bak "s/((${name}))/${value}/g" /etc/prometheus/prometheus.yml
done

/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
  --log.level=debug \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.console.templates=/etc/prometheus/consoles

[dbsanfte]

Users are already routing around the brain damage of @brian-brazil. Very satisfying to watch.

[Qix-]

@alexellis Please respond. How does that help? I'm genuinely curious.

[abh]

Wow. This is pretty ignorant.

It’s hard to be motivated to help when you start out like that.

I am running prometheus in a docker container. To force my environment variables in, I modified the entrypoint of the official image and run this script instead:

Pay attention to the file paths, use a temporary file in /tmp if necessary. Remove the -i option from sed if you don’t need the backup file created (or again, use a different path). Those are basic file permission / shell scripting things, unrelated to Prometheus.

[Qix-]

/tmp is not magic. What benefits does it provide in this case?

I don't see how a backup file makes any difference.

Neither of those things have anything to do with file permissions.

Please explain @abh.

[abh]

The message I replied to originally mentioned how the proposed shell script got various permission errors.

Having a “sidecar” to process configuration and reloads is a completely reasonable and easily implementable setup. The important part is for the primary software to have good facilities for managing configuration reloads, implementing every possible configuration source or templating doesn’t make any sense.

You want environment variables. I want consul. She wants a grpc api. He wants integration with an ansible system.

[civik]

The important part is for the primary software to have good facilities for managing configuration reloads, implementing every possible configuration source or templating doesn’t make any sense. You want environment variables. I want consul. She wants a grpc api. He wants integration with an ansible system.

I generally appreciate @brian-brazil management of the GitHub feature 'squirrels' but in this specific case I believe his stance is wrong. Storing config data in the environment has become best-practice. Its even one of the 'factors' of a 'twelve-factor-app.' https://12factor.net/config

[abh]

I generally appreciate @brian-brazil https://github.com/brian-brazil management of the GitHub feature 'squirrels' but in this specific case I believe his stance is wrong. Storing config data in the environment has become best-practice. Its even one of the 'factors' of a 'twelve-factor-app.' https://12factor.net/config https://12factor.net/config
Doesn’t that only work if you’re restarting the app? Can you change environment variables “underneath” an already running app? If not, that seems like a terrible configuration mechanism for something where you expect regular reconfiguration.

[dbsanfte]

Doesn’t that only work if you’re restarting the app? Can you change environment variables “underneath” an already running app? If not, that seems like a terrible configuration mechanism for something where you expect regular reconfiguration.

Most people I'd imagine are running these apps in stateless containers, so bouncing or recreating them with new variables is standard practice.

[civik]

Also, in case feature parity matters to anyone - TICK stack components are adding this functionality in recent commits - its not in a release yet, but it will be soon. https://github.com/influxdata/telegraf/pull/5648

[ShadowOfLies]

@brian-brazil I don't quite understand why your personal objection should make life difficult for everyone else. Environment variables are not used only for secrets.

Refusing an industry standard method of configuration seems quite unnatural in this industry.

Even closing this thread after 2 years won't make people stop asking for it.

[gannonex]

I'd like to leave a note here that might help others mired by the ignorance of some foolish crusade against environment variables. Our systems utilize stack and we were able to get some of our configuration of this project accomplished by adding an "entrypoint" subsection to the prometheus section of the configuration. Everything else we could do with a "configs" subsection using version 3.6 of compose that inserts the prometheus.yml file into place.

In discussions with my colleagues I found that we are unanimous in our belief that the stubbornness exhibited here doesn't bode well for the future of the Prometheus project. When someone offers up their work for others to save them time it is a truly wonderful and generous thing. In doing so you want your efforts to have mass appeal, be easy to use and versatile enough to wedge in just about anywhere.

Good folks of Prometheus please reconsider your desire to dictate the infrastructure of those who might make use of your project. You will save yourselves countless future threads full of angry developers who after hours of attempting to force a square peg in a round hole, unfortunately, decide to vent their frustrations upon you. This thread is a messed up situation that never had to happen, and I'm sure everyone involved would much rather be hailing you as heroes than reviling you as villains.

[Serrvosky]

I'm trying to implement Prometheus on several kubernetes clusters, and I just want to change remote write URL in each of them. Because Prometheus developers thinks that env variables are dangerous or only for sensitive data (all @brian-brazil replies are only about dangerous of putting sensitive data on them), I have to rethink about my implementation.
It doesn't take any sense, having several config maps to have almost the same information.
And it will be much more difficult to implement a automatic and simpler solution.

Please rethink about this feature. I read all the comment of this topic, and a lot of people ask for this feature. I think it will be good for Prometheus to have it.

The best part of this topic:

brian-brazil closed this on Jan 21, 2017

It's still in discuss for a lot of people.

[Qix-]

I don't see a point in even responding/using prometheus if @brian-brazil is just going to let this issue rot and refuse to discuss anything with users.

[BarakBD-Globality]

Seems that there are plenty of users with this requirement. Why not just put it up for a poll? A good product is built by requirements, not individual opinions.

[Qix-]

I appreciate the staunch requirement of security, actually. I think a poll could end disatrously if the wrong things were polled for. However in this case, there is little to no evidence environment variables in sandboxed environments pose much threat, if any. It's just a case of the BDFL not caring about what people say.

[kron4eg]

There was a solution posted in this issue already: https://github.com/prometheus/prometheus/issues/2357#issuecomment-381054216

[BarakBD-Globality]

@kron4eg - Thanks, but that is only for K8s. Not sure it will work for ECS.

[BarakBD-Globality]

envsubst (part of gettext) could be a solution, however I could not find a way easily install it in Prometheus Docker. This is also much less secure than simply having the templating for config files.

[dbsanfte]

Tbh the only solution to obstinate stubborness like this is a fork.

[proskehy]

Wow. This thread is, baffling.

[Markbnj]

I'll leave a vote in favor of allowing configuration from environment variables. It's a standard approach in cloud deployments, is well-supported by existing deployment tooling, and like others I am not very convinced by the arguments against using the environment to pass secrets. The ability to read secrets from mounted files would make it possible for people who are convinced by those arguments to avoid that vector. As it stands now the various proposed work-arounds add complexity and are worse than the alternatives imo.

I'd also like to put in a strong vote for civility toward the project maintainers. Prometheus is an amazing tool that we get to use for free, thanks entirely to the volunteer efforts of the maintainers. Ad hominem attacks on them are childish and unproductive imo.

[iordaniordanov]

+1 for env vars

[kinghuang]

I'm working on a federated deployment, and the inability to easily fill values on things like external_labels using environment variables is just maddening.

[rbkaspr]

@kinghuang Having read the response to your issue, the idea that configuring Prometheus is something that needs to be handled outside Prometheus is something I find utterly baffling.

On a personally relevant note, I've actually been migrating my organization away from Prometheus and over to Telegraf. The fact the environment variable substitution is supported out of the box is something I consider critical to a distributed monitoring solution. It's a shame that the current CNCF graduated monitoring solution considers something like environment variable substitution for configs outside the scope of the tool itself, when so many other tools both within and without CNCF consider such a thing a standard feature. Hopefully at some point this attitude will change.

[kron4eg]

https://www.robustperception.io/environment-substitution-with-docker

[slifty]

How has Grafana managed to provide support for environment config without becoming a full blown configuration management service?

I'll add my use case to the pile!

I'm using Docker compose + Amazon ECS to set up a Prometheus / Grafana / Alertsmanger.

For the Grafana portion of our setup we are able to deploy semi-configured images stored on dockerhub (the images and config files are fully are public), and configure secrets using environment variables passed into the container.

I'd like to add alertmanager to our stack but without support for env configuration I'm left unsure how I might configure alertmanager without baking the secret into the image (which is a bad practice and would mean I have to either use Amazon's proprietary container registry or paying for private dockerhub). I imagine I'll face the same problem when I get around to configuring the prometheus image to be more secure.

It sounds like I'm going to have to write a script to generate a config file within the container based on passed-in environment variables. I'm left unsure why the Prometheus team demands this of me.

Update: Good to know that Telegraf exists @rbkaspr, I will definitely check that out for future projects.

[roidelapluie]

For ayone following this issue: https://groups.google.com/forum/#!topic/prometheus-developers/tSCa4ukhtUw

Please stay polite and respectful in the discussions.

[beorn7]

Thanks @roidelapluie for beating me to it. It was my intention to backlink from here.

For context: The linked mail thread is a formal voting thread to get an explicit statement from the 22 members of the Prometheus team. (See the governance document to see who is a team member and what that means.)

[Qix-]

Doesn't matter, they're all voting no without giving any information.

Time to fork.

[rbkaspr]

I get the frustration surrounding this issue, but let's wait until we have a majority of nay votes before we start talking forks. So far we only have 2 nays and one who seems to be leaning hard into a nay vote. Even folks who have previously closed issues relating to this feature request are at least having a reasoned discussion about implications, and not just slapping it down.

That said, I do feel that some of the arguments against environment variable expansions are head-scratching, to say the least. The main argument I really take issue with is the idea that somehow adding environment variable expansion as a config option for Prometheus will somehow turn it into a CM tool. As someone who's day-to-day job involves working with actual CM tools, this feature is not magically going to turn Prometheus into Salt, Chef, Ansible, or Puppet. If someone were to request Prometheus suddenly start pushing configurations out to other tools and systems, I'm confident that no one in the user community would bat an eye at such a request being rejected. When you have developers closing issues about how to configure Prometheus as out of scope for the Prometheus team, that's where people start having issues. Especially when (as has been mentioned before), so many other projects both in and out of CNCF have managed to solve this issue, and Kubernetes makes it so simple to take advantage of.

With that out of the way, I appreciate @beorn7 and @roidelapluie linking the developer discussion here, it's good to see that the team is actually having a serious discussion about this request internally rather than just silently rejecting it without explanation. Hopefully we'll land on a positive outcome once January 6th rolls around.

[brian-brazil]

The vote is complete, with results tallied at https://groups.google.com/forum/#!msg/prometheus-developers/tSCa4ukhtUw/J-j0bSEYCQAJ

You can read the whole thread for full details and discussion, but the brief summary is that the vote was 11:2 in favour of keeping the status quo of not supporting environment variable substitution.

Our governance means that this outcome is final, barring a future vote overturning this. I'll now be locking this issue. You are always free to bring things up on the -users/-developers list.