Supervisor: Restart signals not supported

[ido50]

Hi there. Thanks a lot for supervisor, I am using it on quite a few servers and it is performing greatly.

I would like to suggest a feature that would be very advantageous (well, at least for me). There are some processes that support receiving a certain signal, such as HUP, causing them to perform a graceful restart.

Examples for such processes include the HTTP web server Cherokee (http://www.cherokee-project.com) and servers initiated via Perl's Server::Starter (http://search.cpan.org/dist/Server-Starter), which acts as a superdaemon.

If I could set on supervisord.conf that a certain process is restarted with a HUP signal, just like I can set the signal for stopping a process, then I could just do "restart process_name" and get the correct behaviour. That would be awesome.

Thanks a lot,
Ido Perlmuter.

[benjaminws]

1,000 times this

[hidde-jan]

+1

I'd really like to be able to use graceful restarts for uwsgi, celeryd, etc. that I have running under supervisord.

[JulesAU]

Big +1.

Also note that you can't use the supervisorctl pid [processname] | awk | xargs kill -SIGNAL hack for groups that spawn more than one process.

[aj-justo]

I am using supervisorctl pid appname | xargs kill -s HUP for gunicorn, but a proper reload method, instead of relying on hackings, would be very welcome.

[calvinwyoung]

Another +1 for this!

[NachoSoto]

Another +1

[dlo]

This would be great, as I could perform deploys under gunicorn without bringing my site down.

[mattrobenolt]

Isn't this already handled with stopsignal? I use it for gunicorn all the time.

[dlo]

@mattrobenolt I don't think so, since then running supervisorctl stop gunicorn would not actually stop gunicorn. It would just reload it.

[mattrobenolt]

Oh, I get what you're saying. But guincorn supports a graceful shut down with the QUIT signal. So I use stopsignal=QUIT in my conf file for reloads without cutting off a process.

[delfick]

Stopsignal doesn't solve our problem. Supervisor still expects the process
to stop and will eventually sigkill it.

Having a restartsignal option would be nice...

[dlo]

If there's interest from the maintainer I will write a patch for this...

[jumski]

definitely +1 on this!

[ghost]

+1

[blsmth]

any word on this?

[kamilgrymuza]

+1 this would be very useful, in fact that's the only thing important feature I'm missing from Supervisor which is otherwise greate piece of software

[kevinastone]

+2

[antonpirker]

Also +1 from me! (Without this feature I cannot watch my nginx with supervisor, and I would love to, because supervisor is just so good!)

Anyone from the supervisor team reading this? What is the status on this feature request?

[roncohen]

+1

[jasonthomas]

+1

[JannKleen]

+1

[caioariede]

:+1:

[sickill]

:+1:

[72squared]

:+1:

[trollfred]

haha me too. It would be the best thing the ability to describe custom commands in config per program

[AlexDisler]

:+1:

[xordoquy]

Any update on this issue ? There seems to be an unreviewed patch pending.

[deronnax]

+1

[mnaberez]

Related: #179

[aromanovich]

+1

[minaguib]

Another +1 for this, as I'd love to see unicorn zero-downtime deploys working nicely in supervisord

However, I believe the problem is bigger than simply having supervisord support a configuration option for a restart signal. For a thorough discussion see ( http://blog.nicolai86.eu/posts/2012-11-28/zero-downtime-deployments-with-unicorn-and-supervisord/ ) and the mentioned tool ( https://github.com/alphagov/unicornherder/blob/master/unicornherder/herder.py )

Long story short: Many of these zero-downtime flows involve the master (that supervisord is managing) renaming itself to master-old, forking a new master-new, and when all the init and children are ready, old ones commit suicide and new ones accept requests.

While the signalling mechanism (typically sigUSR2) is, I believe, not challenging to add in supervisord, the bigger issue is master-new which supervisord 1. Knows little about and 2. Is not its parent

I feel that for this to become turn-key in supervisord, without the addition of even more layers like the unicornherder glue, supervisord would need to be able to "watch" the given PID file and be able to learn that a given [program]'s PID has changed and start watching the new PID. I'm unsure whether the fact that master-new isn't a child of supervisord is a big deal or not.

[emanuil-tolev]

Right, but not ALL of the zero-downtime workflows require that. The most popular one, just using HUP to gracefully reload the gunicorn config and restart all the workers when they finish processing their current requests, would be great to have as a supervisord capability.

[mjallday]

:+1:

[rocketraman]

+1

[ivanvotti]

+1

[gregtap]

+1

[employ]

+1

[psihonavt]

+1

[ezarowny]

+1

[fdemmer]

+1 for simple HUP (gunicorn reload usecase)

[illagrenan]

+1

[brendanmaguire]

+1

[salqueng]

+1

[ElvisTheKing]

It would be really useful if supervisor will allow sending custom signals via supervisorctl and web control pannel

[Dowwie]

People, this is not 'Nam. This is open source. There are rules.

Quit feature requesting and start coding it your damn selves

http://bit.ly/1mlTKy8

[fschulze]

To send custom signals, you can use the mr.laforge plugin.

[emanuil-tolev]

This is open source. There are rules.

Quit feature requesting and start coding it your damn selves

Of course. However, I've always seen this as letting current regular contributors know which suggestions / enhancements are most wanted by their community. Furthermore there's a pull req open which seems to implement this (though I assume it'll need a minor update since it was made 11 months ago). Maybe the pull req was unsatisfactory, but there's no comment on it to indicate that.

The mr.laforge plugin seems nice, if it works on a few older versions of supervisor besides newest stable, it'll probably do.

[pkallos]

+1

[vahdani]

+1

[fprochazka]

:+1:

[castarco]

+1

[rodrigoolmo]

+1

[sontek]

+1

[bertjwregeer]

:+1:

[wooparadog]

+1 !

[tonyseek]

+1

[faisalp4p]

+1

[msabramo]

I took PR #228 and updated it to create a PR based on the latest master: #477. I hope that's useful.

[msabramo]

As @fschulze already mentioned, the mr.laforge package supplies a supervisor plugin that can be used to send signals to processes:

supervisorctl kill HUP nginx

That said, it would be nice to have this built into supervisor, since it's such a common use case and tons of people want it. One could examine what mr.laforge does and use it as inspiration for creating a PR to add built-in support to supervisor.

[msabramo]

I added frontend (supervisorctl) support to #477 just now. Looks like this:

supervisor> help signal
signal <signal name> <name>           Signal a process
signal <signal name> <gname>:*        Signal all processes in a group
signal <signal name> <name> <name>    Signal multiple processes or groups
supervisor> signal HUP dog:3 dog:4
dog:3: signalled
dog:4: signalled
supervisor> signal HUP dog:*
dog:1: signalled
dog:0: signalled
dog:3: signalled
dog:2: signalled
dog:4: signalled
supervisor> signal USR1 dog:1 dog:2
dog:1: signalled
dog:2: signalled

[adnam]

Hi, as I mentioned in a comment in another thread, I think the ability to signal supervisor processes is great, but I think there's still a good argument to provide a "graceful restart" command. Here's a common use case:

• An application which consists of a PHP app running on Apache, a flask/gunicorn web app and a few celery worker processes.
• I have grouped these into a supervisor process group. A configuration has changed and I want to restart them all.
• In this case I have to send SIGUSR1 to Apache, SIGHUP to the gunicorn process, and SIGTERM to the celery workers.

It would be nice to be able to issue supervisorctl graceful myapp:* and supervisor knows which signal to send in each instance, while defaulting to a normal "restart" if not available. This could be done by allowing an optional "gracefulsignal" to be included in the program configuration:

[program:gunicorn]
command=gunicorn wsgi:app -c conf/gunicorn.py
directory=/myapp
autostart=true
autorestart=true
gracefulsignal=HUP

[anentropic]

+1 on @adnam 's suggestion

[lonetwin]

+1 on this request, irrespective of implementation details. I am interested in using the USR2 signal facility for nginx ( http://nginx.org/en/docs/control.html#upgrade ) (yes, we run nginx under supervisord !) to do on the fly nginx upgrades. Unfortunately this currently doesn't work since sending a USR2 signal to nginx (ie: outside of supervisord context) creates a new nginx master process which is not a child of supervisord.

Update: just saw pull #477 ...that'll work, methinks

[Feng23]

+1 Hope to reload gracefully

[Feng23]

Finally I set the stopwaitsecs long enough to work around my problem.
The number of seconds to wait for the OS to return a SIGCHILD to supervisord after the program has been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHILD from the process, supervisord will attempt to kill it with a final SIGKILL.

[leandrosouza]

apt-get install supervisor
My version is:
$ dpkg -l |grep -i supervisor
ii supervisor 3.0b2-1

restart the program
$ supervisorctl restart program-name
i.e.

$ supervisorctl restart order-form
order-form: stopped
order-form: started

[zllak]

+1 for graceful reload, where we can actually choose the graceful signal to send :)

[andy-pham]

+1

[munzur]

+1

[cades]

+1

[hakanw]

+1

[bijenkins]

+1

[CVirus]

+1

[staab]

There are 390 forks for this. Does anyone know if there is one that implements this change?

Edit: https://github.com/fedosov/supervisor appears to have the change, though I haven't tested it for myself yet.

[amineck]

+1

[omarkhd]

+1

[fordnox]

+1

[ruitao]

+1

[mdelah]

+1

[danericryan]

+1

[fdemmer]

+1 ... we should have a party at nov 22.

[tito]

+1. I'll bring champagne

[RealHacker]

+1 for supporting signal and graceful shutdown/restart.

[gudata]

My first impression was that when supervisord receive some signal to act as a proxy and send the same signal to the controlling processes. But the signals sent to supervisord are not related on how supervisord communicate with the controlled processes.

[mnaberez]

Since Supervisor 3.2 you can send any signal to a subprocess as shown above.

[gudata]

Thanks, I saw it, I have also read all the comments under the pull request https://github.com/Supervisor/supervisor/pull/477 and I am curious to see how supervisor will choose (or not) to support rolling/graceful restarts.

[mnaberez]

If your program foo supports graceful restart by receiving signal SIGUSR2, then you would use the command supervisorctl signal sigusr2 foo instead of the command supervisorctl restart foo.

[calvinwyoung]

The signal implementation in #477 looks great — I think it should suffice for what most folks on this thread want. Looking forward to finally using it :+1:

[sodabrew]

@mnaberez I propose that issue be reopened, because the proposed fix in #477 is really not correct at all for the use cases that are requested here. I will try to recap and explain what I mean:

The idea is that a common alias with the name graceful or reload or something like that should be configurable to mean _"send arbitrary signal foo to the process under supervision"_. The reason is that different software uses different signals for these actions, typically one of HUP or USR1 or USR2, and with different specific meanings assigned to each signal. By allowing third party code (e.g. sysadmin scripts to manage config files) to specify the signal they want to send, you _leak implementation details to the third party script_. The third party does _does not care which signal to send_ it just wants to say to supervisor _"Hey, I changed this application config file, so do whatever you need to tell this process to reload its configs."_

I strongly suggest that a more useful config and command for this use case would be something like:

[program:foo]
stopsignal=TERM
restartsignal=HUP
reloadsignal=USR2

supervisor restart foo # this is a hard restart
supervisor reload foo # this is a graceful reload

Again, different tools implement their signals differently, and the idea is that supervisor should abstract these differences to a reasonable degree. For example, in nginx, graceful restart is SIGHUP: http://nginx.org/en/docs/control.html But in Apache, hard restart is SIGHUP and graceful restart is SIGUSR2: https://httpd.apache.org/docs/current/stopping.html Meanwhile, in PHP-FPM, there is no hard restart (you would SIGTERM and begin again) but graceful restart is also SIGUSR2.

What's really important is to provide a way to abstract these command-to-signal mappings and allow sysadmins to configure the application's supervisor configs once, and then reuse the commands elsewhere without having to remember that nginx needs a HUP but Apache needs a USR2 all over their other automation scripts.

[timestee]

@mnaberez I propose that issue be reopened too, and i do not think supervisorctl signal sigusr2 foo useful for graceful restart.
The process foo(pid: 60900) for example under supervisor control, we send sigusr2 signal to it for graceful restart, the process(pid: 60900) receive the signal sigusr2, it will fork or exec a child process(pid: 60901) with some extra parameters such as the the active listeners, then this child process will close the parent process(pid:60900) by send SIGTERM to it, then the parent process(pid: 60900) exit. BUT NOW SUPERVISIOR WILL RESTART THE PROCESS TOO, the new process(pid: 60902) started by SUPERVISIOR may start failed due to the net port address already in use by process 60901. But 60901 will not under supervisor control.

[sodabrew]

@zsounder the problem you are describing is not one that supervisor can solve right now - if your process does a fork and exec then it will escape from under supervisor, and that limitation is different and unrelated to this issue of abstracting the restart signals. Again the choice of restart signals is program specific and is based on the behavior of the program that the administrator is writing their supervisor configs for. You need to do your own reading on which signals to use.

My very specific argument here is that once you the administrator figure out which signal you want to send, supervisor should help abstract the decision rather than help leak the decision into other parts of your system.

For your use case with fork exec, you might look into cgroups on Linux, because it allows you to manage a subtree of processes together.

[timestee]

@sodabrew perhaps you are right, in fact, i think supervisor should not just monitor the process by process id, monitor by the process name could be helpful too。

def monitor_process(key_word, cmd):
    if check_process_by_name(key_word):
        return

    sys.stderr.write('process[%s] is lost, run [%s]\n' % (key_word, cmd))
    subprocess.call(cmd, shell=True) 

a simple implementation for this, we could run it under crontab.

If supervisor support Monitor The Process By Process Name, Even The Port Process Listened on, it will be useful i think.

[sodabrew]

@zsounder Supervisor does not monitor by pid, per se, it monitors by parent-child relationship. Please read http://supervisord.org/subprocess.html to learn more about its design. Please please please do not hijack this thread to propose new ideas on UNIX process management.

[timestee]

@sodabrew I see now, ths

[wsYdd]

mark