Habitat: Envvar config overrides ignored unless sup started from stopped state

[jerryaldrichiii]

The following works outside of the studio but not inside

HAB_MYPKG='{"foo":"bar"}' hab start myorigin/foobar

I have also tried specifying the .hart file directly and it produces the same behavior.

I am running 0.38.0 according to hab --version

Let me know if you need any additional info.

[eeyun]

Hmm interesting. Can you provide a little more context for your issue? Is the service starting with the wrong configuration values?

[jerryaldrichiii]

Correct.

@stevendanna helped me in http://slack.habitat.sh. It appears that the supervisor reads the environment variables at startup and that was the cause of the issue. Killing the supervisor and then running HAB_MYPKG='{"foo":"bar"}' hab start myorigin/foobar works.

In my specific example I'm creating a Mosquitto MQTT broker package and am wanting to allow anonymous connections by default but disable them when actually using the artifact (via overriding the config value at startup).

In the studio I was testing this by running build then hab stop myorigin/mosquitto followed by HAB_MOSQUITTO='{"allow_anoymous":"false"}' hab start myorigin/mosquitto but this was not working.

It was working outside the studio because I did not have the supervisor running so my hab start was also starting the supervisor.

I believe that either a warning should be shown when using environment variables with a supervisor running or there should be some other method of starting a service with an override config.

For example, I would not want to start a service with default admin:admin credentials (even though I could theoretically override with hab config apply) because the service would be vulnerable for a short time.

[rsertelon]

FYI: There's a mosquitto package being added to core at this moment :) see https://github.com/habitat-sh/core-plans/pull/902 if interested

[jerryaldrichiii]

@rsertelon I may have used yours as a slight reference :wink:

[rsertelon]

@jerryaldrichiii :smile: the PR is about merging my package in core ;)

[eeyun]

I can repro this behavior but I need to do some more testing on it. Here's what I've tested and the output I've seen:

First attempt definitely doesn't get picked up.

[1][default:/src:0]# HAB_HTTPD='{"serveradmin":"[email protected]"}' hab svc start core/httpd
[2][default:/src:0]# cat /hab/svc/httpd/config/httpd.conf | grep ServerAdmin
# ServerAdmin: Your address, where problems with the server should be
ServerAdmin [email protected]

Second attempt (in a clean studio environment) also doesn't get picked up.

[1][default:/src:0]# export HAB_HTTPD='{"serveradmin":"[email protected]"}'
[2][default:/src:0]# hab start core/httpd
[3][default:/src:0]# cat /hab/svc/httpd/config/httpd.conf | grep ServerAdmin
# ServerAdmin: Your address, where problems with the server should be
ServerAdmin [email protected]

Testing outside the studio with a running supervisor.

➜  core-plans git:(master) ✗ hab sup status
No services loaded.
➜  core-plans git:(master) ✗ sudo HAB_HTTPD='{"serveradmin":"[email protected]"}' hab start core/httpd 
hab-sup(MN): Supervisor starting core/httpd. See the Supervisor output for more details.
➜  core-plans git:(master) ✗ sudo cat /hab/svc/httpd/config/httpd.conf | grep ServerAdmin
# ServerAdmin: Your address, where problems with the server should be
ServerAdmin [email protected]
# Set to "EMail" to also include a mailto: link to the ServerAdmin.

Now killing the sup forcefully and trying again.

➜  core-plans git:(master) ✗ sudo hab sup term  
➜  core-plans git:(master) ✗ sudo hab sup status
The Supervisor is not running.
➜  core-plans git:(master) ✗ sudo HAB_HTTPD='{"serveradmin":"[email protected]"}' hab start core/httpd
➜  core-plans git:(master) ✗ sudo cat /hab/svc/httpd/config/httpd.conf | grep ServerAdmin
# ServerAdmin: Your address, where problems with the server should be
ServerAdmin [email protected]

It appears that the supervisor isn't picking up envvars unless they're available before the supervisor starts. This is a fairly recent bug as this certainly used to work.

[christophermaier]

This behavior is expected, based on how the Supervisor is currently implemented, and should be better documented to prevent this clash of expectation-vs.-reality.

That being said, the soon-to-be-released refactored Supervisor has streamlined the interaction model between users starting a service with hab and interacting with a currently-running Supervisor. It's possible that we can augment the communication protocol to pass the appropriate HAB_$SERVICE environment variable, if present, along and incorporate that into the running of the service.

[christophermaier]

Thinking more on this, I'm not convinced that we should change the behavior of environment variables here. I've actually got doubts as to whether they're even something we want to publicize the use of.

Back when the only way you could run a service was to literally start a Supervisor (i.e., back before the Launcher, or even the ability to run multiple services on a Supervisor), an environment variable made a lot of sense. Now, however, they're very limited. You'd need to plan exactly what services you're going to run before you start your Supervisor, for instance, and you'd need to bring _all_ the services down in order to change the configuration of _one_ of them.

In modern Supervisors, I think that anything you'd want to do with environment variables, you can do much more flexibly with a user.toml file. You can use user.toml to still have machine-specific configuration, and you can easily update the configuration in a running Supervisor, since user.toml files are watched for changes.

You could make an argument for keeping environment variables for containerized workloads, but even there, you can just as easily mount in a user.toml file.

@fnichol @adamhjk @baumanj Do you all have any thoughts on this?

[baumanj]

100% agree @christophermaier. Environment variables are suboptimal here and creates unintended coupling to the details of our implementation that I don't think we want users to know about or depend upon.

[jerryaldrichiii]

Given the age of this ticket and recent arguments, I am 100% in favor of @christophermaier's argument.

[baumanj]

I doesn't look like there's anything more do do here, please re-open if necessary.