Supervisor: shutdown supervisord once one of the programs is killed

[EslamElHusseiny]

Hi,
I've two programs managed by supervisord.
Is there anyway to shutdown supervisord once of those programs is killed ?

[tarrychk]

You can achive this with an event listener:

supervisord-watchdog:

#!/usr/bin/python
import sys
import os
import logging
import subprocess
import time

from supervisor.childutils import listener

def main(args):
    logging.basicConfig(stream=sys.stderr, level=logging.DEBUG, format='%(asctime)s %(levelname)s %(filename)s: %(message)s')
    logger = logging.getLogger("supervisord-watchdog")
    debug_mode = True if 'DEBUG' in os.environ else False

    while True:
        logger.info("Listening for events...")
        headers, body = listener.wait(sys.stdin, sys.stdout)
        body = dict([pair.split(":") for pair in body.split(" ")])

        logger.debug("Headers: %r", repr(headers))
        logger.debug("Body: %r", repr(body))
        logger.debug("Args: %r", repr(args))

        if debug_mode: continue

        try:
            if headers["eventname"] == "PROCESS_STATE_FATAL":
                logger.info("Process entered FATAL state...")
                if not args or body["processname"] in args:
                    logger.error("Killing off supervisord instance ...")
                    res = subprocess.call(["/bin/kill", "-15", "1"], stdout=sys.stderr)
                    logger.info("Sent TERM signal to init process")
                    time.sleep( 5 )
                    logger.critical("Why am I still alive? Send KILL to all processes...")
                    res = subprocess.call(["/bin/kill", "-9", "-1"], stdout=sys.stderr)
        except Exception as e:
            logger.critical("Unexpected Exception: %s", str(e))
            listener.fail(sys.stdout)
            exit(1)
        else:
            listener.ok(sys.stdout)

if __name__ == '__main__':
    main(sys.argv[1:])

Then add this config snippet:

[eventlistener:supervisord-watchdog]
command=/path/to/supervisord-watchdog
events=PROCESS_STATE_FATAL

[tarrychk]

That eventlistener snippet above was written for running supervisord in a docker container where it would have PID 1. You'll have to replace the kill commands with something suitable for your use case, like "pkill", "-15", "supervisord".

[mnaberez]

Writing an eventlistener similar to the one above is the recommended way to achieve this. More examples of eventlisteners can be found in the Superlance package.

[tillahoffmann]

This is probably not the recommended way of shutting down supervisor but the following configuration has worked well for me.

[program:something]
command = bash -c "something && kill -s SIGINT `cat supervisord.pid`"
stopasgroup = true

; other bits

It will execute something and then interrupt the supervisord process.

[bdusell]

The watchdog script works well if none of the processes exit immediately -- I've been using something similar on my server for over a year. But if one of the processes exits before Supervisor starts the watchdog process (e.g. Nginx exits because its configuration file has a syntax error), then Supervisor continues to run. Setting priority=1 on the watchdog doesn't help.

@tillahoffmann 's solution avoids this problem, avoids running an extra process, and is much simpler.

[mnaberez]

Note for maintainers: See additional comments in #1340.

[mnaberez]

This issue was reopened in error. Usually when we need to watch processes and perform some configurable action based on their behavior, we implement those things in eventlisteners rather than supervisord itself. This was a design decision that was made a long time ago and why the eventlistener system came about.

For example, it kept coming up that a subset of users wanted to monitor when a process uses "too much" memory and perform some action as a result. It was decided to keep that functionality in eventlisteners. The same points were raised in those tickets, e.g. some users do not want to run an extra eventlistener process. However, eventlisteners were written to do it and users run them. Overall, it seems to have worked out.

Similarly here, some users want to monitor when a process does something and cause supervisord itself to exit as a result. Users have reported successfully using eventlisteners in those scenarios. At least for the time being, we are going to continue to recommend that approach since it's what we do for similar things.