Supervisor: Supervisor won't start using %(ENV_env_var_here)s in environment

[crisbratu]

Hey folks I'm trying to run a node process via supervisord (supervisor-3.0-0.1.a8.fc13.noarch) and I've noticed some strange behaviour.

Supervisor won't start when %(ENV_VAR)s is set. The config, sensitive info redacted:

[program:node-studio] environment=PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s" numprocs=1 command=/usr/bin/node /data/web/studio/current/app.js directory=/data/web/studio/current

When running 'supervisorctl update':

supervisorctl update

error: <class 'xmlrpclib.Fault'>, <Fault 92: 'CANT_REREAD: Format string PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s" for \'environment\' contains names which cannot be expanded'>: file: /usr/lib/python2.6/site-packages/supervisor/supervisorctl.py line: 933

However, if I set the path manually based on 'echo $PATH':

environment=PATH="/data/shared/blender:/data/shared/bin:/data/rvm/gems/ruby-2.0.0-p576/bin:/data/rvm/gems/ruby-2.0.0-p576@global/bin:/data/rvm/rubies/ruby-2.0.0-p576/bin:/usr/pgsql-9.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/data/rvm/bin"

Supervisord starts successfully as expected. If I modify the config again with environment as:

environment=PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s"

The path is now correct for the process, as grabbed from /proc/$PID/environ. The correct $PATH variable is only present when the PATH= variable is set to $PATH manually, then to %(ENV_PATH)s.

[mnaberez]

error: <class 'xmlrpclib.Fault'>, <Fault 92: 'CANT_REREAD: Format string PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s" for \'environment\' contains names which cannot be expanded'>: file: /usr/lib/python2.6/site-packages/supervisor/supervisorctl.py line: 933

This error indicates that the %(ENV_PATH)s expansion is not supported in the environment= option for the version of Supervisor that you are using.

This expansion should be supported since 2d6ca34582a8a07a5dd96ae45ef62cd58a459f4f, which will be released in Supervisor 3.2. I added a test for this specific case in 3fe078a77f0a56903000b0af3283df843fbed598. Since it is passing, I am going to close this issue.

[mrcoles]

Hey @mnaberez, looking at the supervisord release log, the last release was 3.1.3 on 2014-10-28. Do you know when 3.2 might release?

Separately, do you know any workarounds in the short-term to pass non-hardcoded environment variables into the environment of a supervisord process?

_EDIT:_ in my particular scenario I discovered that by running service supervisor start, I was not getting access to environment variables that I needed, while running supervisord -n directly would make them available to supervisor processes. So, not quite as pressing of a problem as I thought :blush:

[mnaberez]

Hey @mnaberez, looking at the supervisord release log, the last release was 3.1.3 on 2014-10-28. Do you know when 3.2 might release?

Very soon, probably this week or next.

_Edit: 3.2 is pending fix of #677._

[citizen-stig]

I've got the same problem. Mainly I use supervisor in amazon elastic beanstalk

[jbank]

So is 3.2 still coming out?

[mnaberez]

I actually tried to release it a few days ago, but found another significant problem just before uploading the package. I'll probably release it next week if I don't find anything else broken.

[arjenveenhuizen]

mrcoles comment on Sep 24

EDIT: in my particular scenario I discovered that by running service supervisor start, I was not getting access to environment variables that I needed, while running supervisord -n directly would make them available to supervisor processes. So, not quite as pressing of a problem as I thought

The same holds for me. This is the only way I got (custom) environment variables to work in a supervisor program (3.0b2, 3.2). Using -n, I no longer have to specify the environment= variable in the .conf file.

[mnaberez]

Supervisor 3.2.0 (November 30, 2015) supports using %(ENV_x)s expansions in environment= and there is a unit test for it specifically.

[Sshuichi]

3.3.1 and same issue...

[sacheth003]

Seeing the same issue in 3.3.1

[mnaberez]

For the commenters above saying that %(ENV_x)s expansions in environment= do not work on 3.3.1, please see the following example.

Supervisor version:

$ supervisord --version
3.3.1

Minimal supervisord.conf config file:

[supervisord]
loglevel=debug 
nodaemon=true

[program:env]
environment = PETS="cats,%(ENV_PETS)s"
command = bash -c 'env | grep PETS'
startsecs = 0
autorestart = false

Create an environment variable PETS:

$ export PETS=dogs

Run supervisord in the foreground with that config:

$ supervisord -n -c supervisord.conf
2018-12-16 15:21:07,247 INFO Increased RLIMIT_NOFILE limit to 1024
2018-12-16 15:21:07,250 INFO supervisord started with pid 3748
2018-12-16 15:21:08,253 INFO spawned: 'env' with pid 3751
2018-12-16 15:21:08,269 DEBG 'env' stdout output:
PETS=cats,dogs

2018-12-16 15:21:08,269 INFO success: env entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-12-16 15:21:08,269 DEBG fd 9 closed, stopped monitoring <POutputDispatcher at 4424125328 for <Subprocess at 4424125112 with name env in state RUNNING> (stderr)>
2018-12-16 15:21:08,269 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 4424125184 for <Subprocess at 4424125112 with name env in state RUNNING> (stdout)>
2018-12-16 15:21:08,269 INFO exited: env (exit status 0; expected)
2018-12-16 15:21:08,270 DEBG received SIGCHLD indicating a child quit

As you can see above, the environment when supervisord is started contains PETS=dogs. When the subprocess is run, its environment contains PETS=cats,dogs. This shows that the environment expansion (%(ENV_PETS)s) in environment= is working, which is this issue report.

Please see Subprocess Environment for more information about how Supervisor handles environment variables.

[markroxor]

@mnaberez I am getting -

Jan 20 18:10:30 raspberrypi supervisord[3183]: Error: Format string 'PETS="cats,%(ENV_PETS)s"' for 'environment' contains names ('ENV_PETS') which cannot be expanded. Available names: ENV_INVOCATION_ID, E

and yes I have the same version and have exported the variable as well.

PS - I am using supervisor as a service. Can that be an issue?

[mnaberez]

and yes I have the same version and have exported the variable as well.

PS - I am using supervisor as a service. Can that be an issue?

The environment variable must exist in the environment from which the supervisord process is started. See Subprocess Environment for more information. The error message (Available names: ...) is showing the environment variables that are present in supervisord's environment.

[cocobuster]

I am running in to the same issue:

My supervisor version is: supervisor:all/bionic 3.3.1-1.1 uptodate